We are developing and web application which allows users to register for certain events. What application is supposed to do, is to send them few notifications to remind them that they have registered. There will be more than 1k user which can register for many events in very wide time range. We have to send notofications like 3 months, 1 month, 1 week and one day before event.
The first thing is that I have to determine if I need to send notification to a specific user.
I'm thinking about thread which will iterate over registrations and determine whether sending notification is required or not. If notification is required, should I do it right away or maybe put all of the objects that need it in some kind of cache and then send them (by another thread)?
Second thing is: if I made that thread - is better to put it and run next to application or embed that thread into application and, for example, start it in the context listener?
How You would solve this? Maybe there are better approaches?
I would not spawn my own threads for that, I would use a scheduler like Quartz and run daily or hourly jobs (I don't know what granularity you need) that would:
find upcoming events in 1 day, 1 week, 1 month, 3 months and users that should get notified about them.
create the notifications and send them
I would probably implement that using separate jobs (sending notifications is a different concern) and thus queue the results of the first part, this will give you more flexibility. And the first part could be done by a unique job scheduled with different time frame parameters (1 day, 1 week, 1 month, 3 months).
Tabling the question about how to schedule the notifications once they're identified, I'd recommend looping over upcoming events, instead of over all users. It seems very likely that you'll have many more users than events (especially if you limit your scan to events that happen exactly 1 week, 1 month and 3 months in the future).
As far as the notifications, I think marking notifications to be sent first, then processing all the marked notifications will allow for more optimization than sending out notifications as part of your scan. If you have a queue of notifications to be sent out, you could then send each affected user one email including multiple events in the same time.
Related
I have a requirement to update a field if a payment date is 30 days late.
Is it possible to trigger an action to occur 30 days after the payment date?
The process is likely to restart in-between those times so it can't be in-memory and it can't be a relative date.
I can create an endpoint like /api/paymentdates so that it could be called from something else like cron.
However, there are likely to be a over a million items at some point in the future, each with its own date.
Is there an effective way to trigger a task like this or is the only option to run a task every morning and query the database?
You can make use of a Queue Triggered Function (Storage Queue or Service Bus Queue). Basically what you will do is put a message in a queue and keep it invisible for 30 days (it is called initial visibility timeout in storage queue and scheduled message in service bus queue).
The message will only appear in the queue after its invisibility expires and the Function will be triggered at that time. Once the Function is triggered, you can do whatever processing you want to do on that message.
You may find following links useful:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-queue
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus
For the last few years we have used our own RM Application to process events related to our applications. This works by polling a database table every few minutes, looking for any rows that have a due date before now, and have not been processed yet.
We are currently making the transition to SNS, with SQS Worker tiers processing them. The problem with this approach is that we can't future date our messages. Our applications sometimes have events that we don't want to process until a week later.
Are there any design approaches, alternative services, clever tricks we could employ that would allow us to do achieve this?
One solution would be to keep our existing application running, at a simplified level, so all it does is send the SNS notifications when they are due, but the aim of this project is to try and do away with our existing app.
The database approach would be the wisest, being careful that each row is only processed once.
Amazon Simple Notification Service (SNS) is designed to send notifications immediately. There is no functionality for a delayed send (although some notification types are retried if they fail).
Amazon Simple Queue Service (SQS) does have a delay feature, but only up to 15 minutes -- this is useful if you need to do some work before the message is processed, such as copying related data to Amazon S3.
Given that your requirement is to wait until some future arbitrary time (effectively like a scheduling system), you could either start a process and tell it to sleep for a certain amount of time (a bad idea in case systems are restarted), or continue your approach of polling from a database.
If all jobs are scheduled for a distant future (eg at least one hour away), you theoretically only need to poll the database once an hour to retrieve the earliest scheduled time.
A week might be too long as SQS message retention itself is only 15 days. If you are okay with maximum retention of 15days, one idea is to keep the changing the visibility of a message every time you receive until it is ready for processing. The maximum allowed visibility timeout is 12 hours. More on visibility timeout and APIs for changing them,
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html
I found this approach: https://github.com/alestic/aws-sns-delayed. Basically, you can use a step function with a wait step in there
I use appengine (java) as a backend for a mobile app (android). The user of my app can create public events consisting of a title and a date/time. Those events are stored on my appengine backend. Any user can subscribe for events and will receive a push notification at that time the events starts.
So I want to schedule a job / task on appengine to run at the events date/time to send a push notification to all subscribers.
Example:
User A creates an event that will start on Saturday next week on 8 pm. User B and User B subscribe for this event.
On Saturday 8 pm a job/task should start to send the push notifications to User B and User C to inform that the event has started.
My question is:
How do I implement something like this in an efficient way on appengine? I want to say, start a Task for the Event on Saturday at 8 pm. There are Cronjobs and TaskQueues. Cronjobs can not be created programmatically. TaskQueue needs to be pulled and can not be scheduled to pull at a given date / time, right?
So the only solutions I see is to create a cronjob that will run every minute to check if there is a Event that starts right now.
An event can be created at any time and any day in the week. However the most events are created for the weekend and There are days when no event has been created for. So running a cronjob periodically ever minute is very inefficient. Im looking for a smarter solution, any ideas?
Indeed, based on your description, the solution is to create a cron job that checks for new events that start right now.
However, you should be careful not to exceed the 60 second window you have for each cron job. If you have a lot of events, you probably should move the actual processing from the cron job to background tasks using Push Task Queues.
You may take a look to this post for a combination of cron jobs and Task Queues.
I have an requirement where I have to send the alerts when the record in db is not updated/changed for specified intervals. For example, if the received purchase order doesn't processed within one hour, the reminder should be sent to the delivery manager.
The reminder/alert should sent exactly at the interval (including seconds). If the last modified time is 13:55:45 means, the alert should be triggered 14:55:45. There could be million rows needs to be tracked.
The simple approach could be implementing a custom scheduler and all the records will registered with it. But should poll the database to look for the change every second and it will lead to performance problem.
UPDATE:
Another basic approach would be a creating a thread for each record and put it on sleep for 1 hour (or) Use some queuing concept which has timeout. But still it has performance problems
Any thoughts on better approach to implement the same?
probably using internal JMS queue would be better solution - for example you may want to use scheduled message feature http://docs.jboss.org/hornetq/2.2.2.Final/user-manual/en/html/examples.html#examples.scheduled-message with hornetq.
You can ask broker to publish alert message after exactly 1h. From the other hand during processing of some trading activity you can manually delete this message meaning that the trade activity has been processed without errors.
Use Timer for each reminder.i.e. If the last modified time is 17:49:45 means, the alert should be triggered 18:49:45 simply you should create a dynamic timer scheduling for each task it'll call exact after one hour.
It is not possible in Java, if you really insist on the "Real-timeness". In Java you may encouter Garbage collector's stop-the-world phase and you can never guarantee the exact time.
If the approximate time is also permissible, than use some kind of scheduled queue as proposed in other answers, if not, than use real-time Java or some native call.
If we can assume that the orders are entered with increasing time then:
You can use a Queue with elements that have the properties time-of-order and order-id.
Each new entry that is added to the DB is also enqueued to this Queue.
You can check the element at the start of the Queue each minute.
When checking the element at the start of the Queue, if an hour has passed from the time-of-order, then search for the entry with order-id in the DB.
If found and was not updated then send a notification, else dequeue it from the Queue .
I think this is a solved problem but my google-fu wasn't good enough.
I have a table tracking the status of multiple things. Server can push changes to clients at will.
The problem is I don't want to push update if the last update is less than 5 seconds ago.
What is the cleanest way of achieving this without making another event manager thread?
My current stab looks like this:
pushEvent(){
Look up the last update time
if: last update less than 5 sec ago
then do nothing
else
pushToClients
}
It works good enough for most part, but obviously the last update could be left unpushed.
What is a good way of doing this?
Some ways I have thought of:
Add a 5 second delay to all push (eg Thread.sleep), that way I can
check if an update has already been scheduled. Not ideal but no one
would mind.
Do the push, then set doNotPush=true. Use a timer to
set it back to false.
Thanks,
On server, you can use HashMap which will contain pair {current event, last sent event} as values and event's recepients (clients) as keys. Then use scheduled periodic task which will iterate over this map, check if current event != last sent event and, if yes, will send current event to client (and put it intolast sent).
One drawback can be that if sending to client is slow, there can be jams of events in outgoing queues. You can work around this by sending asynchronously (e.g., by offloading events to be sent into separate Executor).