I think this is a solved problem but my google-fu wasn't good enough.
I have a table tracking the status of multiple things. Server can push changes to clients at will.
The problem is I don't want to push update if the last update is less than 5 seconds ago.
What is the cleanest way of achieving this without making another event manager thread?
My current stab looks like this:
pushEvent(){
Look up the last update time
if: last update less than 5 sec ago
then do nothing
else
pushToClients
}
It works good enough for most part, but obviously the last update could be left unpushed.
What is a good way of doing this?
Some ways I have thought of:
Add a 5 second delay to all push (eg Thread.sleep), that way I can
check if an update has already been scheduled. Not ideal but no one
would mind.
Do the push, then set doNotPush=true. Use a timer to
set it back to false.
Thanks,
On server, you can use HashMap which will contain pair {current event, last sent event} as values and event's recepients (clients) as keys. Then use scheduled periodic task which will iterate over this map, check if current event != last sent event and, if yes, will send current event to client (and put it intolast sent).
One drawback can be that if sending to client is slow, there can be jams of events in outgoing queues. You can work around this by sending asynchronously (e.g., by offloading events to be sent into separate Executor).
Related
I want to process multiple events in order of their timestamps coming into the system via multiple source systems like MQ, S3 ,KAFKA .
What approach should be taken to solve the problem?
As soon as an event comes in, the program can't know if another source will send events that should be processed before this one but have not arrived yet. Therefore, you need some waiting period, e.g. 5 minutes, in which events won't be processed so that late events have a chance to cut in front.
There is a trade-off here, making the waiting window larger will give late events a higher chance to be processed in the right order, but will also delay event processing.
For implementation, one way is to use a priority-queue that sorts by min-timestamp. All event sources write to this queue and events are consumed only from the top and only if they are at least x seconds old.
One possible optimisation for the processing lag: As long as all data sources provide at least one event that is ready for consumption, you can safely process events until one source is empty again. This only works if sources provide their own events in-order. You could implement this by having a counter for each data source of how many events exist in the priority-queue.
Another aspect is what happens to the priority-queue when a node crashes. Using acknowledgements should help here, so that on crash the queue can be rebuilt from unacknowledged events.
I have a task to implement a distributed Queuing System something like the Amazon SQS.
If there is GET Request, I have to deliver the message to the user from the main queue and put the message in the invisible queue. And immediately a DELETE Request should come and I should delete the message from the invisible queue.
In case there is no DELETE Request, I am supposed to increase the redelivery count and send the message back to the main queue. This will happen till the redelivery count becomes 5 after which I will delete the message permanently.
Now my doubt is, how do I know that there has been no DELETE request which means that I should send the message back to the main queue?
My program works for the case where the DELETE Request follows the GET Request. I am using java for this implementation.
First of all, at the design level, the get and delete should be done in one action. Notice that in the JDK, the pull() operation of Queue will do get and delete. if you insist on separate actions, at the very least you should support an optional get-and-delete request type.
now, there is a problem when you want to detect an action that did not happen because it can forever "maybe happen in the future". So you need to set a window of time after which you decide that the expected action did not happen.
what is usually done is that you attach a "received" timestamp to the request (and also re-deliver count) before putting it in the invisible queue (a better name would be "pending delete requests" queue) you can wrap the request in a custom java class that adds the properties.
actually, I don't think a queue is a good choice for a collection. when a delete request does come, you need random access to the request. so perhaps a hash map is a better choice.
you will need to implement a Timer that invokes tasks every x seconds. the tasks will scan the pendingDeleteRequests map for requests that did not recevie delete in the allowed window of time and remove from the map.
last note: some messaging systems have "dead letter" feature, which is a destination where notices of failed deliveries are sent. this will help in debugging of problems.
Here is the problem,
I have a network request which downloads some information. However, it is essential that this request is called only once during some period of time ( you will get the idea later on ) and all subscribers get the same result. My first thought was to use the share() operator, so it would multicast the result while keeping a single request source. But I am not sure what is going to happen if I try to subscribe to it again after the share operator already disposed the resources due to refCount dropping to 0.
The thing I am trying to accomplish here is that every request that I make, is dependent on the current state of information stored and those requests update this information. Once I make the first request, I need to keep a reference to it and inform every subscriber that subscribes until the time of request completion. After the request is finished, all subscribers gets their notification and unsubscribes... However, if there is a new subscription after the disposal I need it to repeat the request, thus resubscribing to the original source observable that was modified using share
Is something like this possible with simple share operator, or do I need to create a subject and control the emissions manually ?
There is a nice library RxReplayingShare, which I think makes exactly, what you are trying to achieve.
It passes the same result to all Subscriber's, when at least one is subscribed. When there are no subscribers anymore, the Observable completes. When subscribing again, the original Observable is called.
The RxMarble shows it better than the description.
I have an requirement where I have to send the alerts when the record in db is not updated/changed for specified intervals. For example, if the received purchase order doesn't processed within one hour, the reminder should be sent to the delivery manager.
The reminder/alert should sent exactly at the interval (including seconds). If the last modified time is 13:55:45 means, the alert should be triggered 14:55:45. There could be million rows needs to be tracked.
The simple approach could be implementing a custom scheduler and all the records will registered with it. But should poll the database to look for the change every second and it will lead to performance problem.
UPDATE:
Another basic approach would be a creating a thread for each record and put it on sleep for 1 hour (or) Use some queuing concept which has timeout. But still it has performance problems
Any thoughts on better approach to implement the same?
probably using internal JMS queue would be better solution - for example you may want to use scheduled message feature http://docs.jboss.org/hornetq/2.2.2.Final/user-manual/en/html/examples.html#examples.scheduled-message with hornetq.
You can ask broker to publish alert message after exactly 1h. From the other hand during processing of some trading activity you can manually delete this message meaning that the trade activity has been processed without errors.
Use Timer for each reminder.i.e. If the last modified time is 17:49:45 means, the alert should be triggered 18:49:45 simply you should create a dynamic timer scheduling for each task it'll call exact after one hour.
It is not possible in Java, if you really insist on the "Real-timeness". In Java you may encouter Garbage collector's stop-the-world phase and you can never guarantee the exact time.
If the approximate time is also permissible, than use some kind of scheduled queue as proposed in other answers, if not, than use real-time Java or some native call.
If we can assume that the orders are entered with increasing time then:
You can use a Queue with elements that have the properties time-of-order and order-id.
Each new entry that is added to the DB is also enqueued to this Queue.
You can check the element at the start of the Queue each minute.
When checking the element at the start of the Queue, if an hour has passed from the time-of-order, then search for the entry with order-id in the DB.
If found and was not updated then send a notification, else dequeue it from the Queue .
We are developing and web application which allows users to register for certain events. What application is supposed to do, is to send them few notifications to remind them that they have registered. There will be more than 1k user which can register for many events in very wide time range. We have to send notofications like 3 months, 1 month, 1 week and one day before event.
The first thing is that I have to determine if I need to send notification to a specific user.
I'm thinking about thread which will iterate over registrations and determine whether sending notification is required or not. If notification is required, should I do it right away or maybe put all of the objects that need it in some kind of cache and then send them (by another thread)?
Second thing is: if I made that thread - is better to put it and run next to application or embed that thread into application and, for example, start it in the context listener?
How You would solve this? Maybe there are better approaches?
I would not spawn my own threads for that, I would use a scheduler like Quartz and run daily or hourly jobs (I don't know what granularity you need) that would:
find upcoming events in 1 day, 1 week, 1 month, 3 months and users that should get notified about them.
create the notifications and send them
I would probably implement that using separate jobs (sending notifications is a different concern) and thus queue the results of the first part, this will give you more flexibility. And the first part could be done by a unique job scheduled with different time frame parameters (1 day, 1 week, 1 month, 3 months).
Tabling the question about how to schedule the notifications once they're identified, I'd recommend looping over upcoming events, instead of over all users. It seems very likely that you'll have many more users than events (especially if you limit your scan to events that happen exactly 1 week, 1 month and 3 months in the future).
As far as the notifications, I think marking notifications to be sent first, then processing all the marked notifications will allow for more optimization than sending out notifications as part of your scan. If you have a queue of notifications to be sent out, you could then send each affected user one email including multiple events in the same time.