I have implemented the code from the below link for receiving events from the event hub. But say there are 10 events and checkpoint is done for every 5 events. Now the program exits abnormally when reading the 7th event and if i restart the event processor host again then events (1,2,3,4,6) are re-read. Please suggest how will I again avoid re-read and read from the 7th event? Any example would be appreciated. Thanks.
https://github.com/Azure/azure-event-hubs/blob/master/samples/Java/src/main/java/com/microsoft/azure/eventhubs/samples/Basic/EventProcessorSample.java
Well, in the context of the event hub it is your job to handle possible message duplication as the event hub guarantees at least once delivery, see
Does Azure Event Hub guarantees at least once delivery?
We do not know your scenario so we cannot suggest something other than for you to build a duplicate message detection mechanism. For example, adding a unique id to the data prior to sending it to the event hub and have the processor check that against a list of processed messages.
Related
I want to process multiple events in order of their timestamps coming into the system via multiple source systems like MQ, S3 ,KAFKA .
What approach should be taken to solve the problem?
As soon as an event comes in, the program can't know if another source will send events that should be processed before this one but have not arrived yet. Therefore, you need some waiting period, e.g. 5 minutes, in which events won't be processed so that late events have a chance to cut in front.
There is a trade-off here, making the waiting window larger will give late events a higher chance to be processed in the right order, but will also delay event processing.
For implementation, one way is to use a priority-queue that sorts by min-timestamp. All event sources write to this queue and events are consumed only from the top and only if they are at least x seconds old.
One possible optimisation for the processing lag: As long as all data sources provide at least one event that is ready for consumption, you can safely process events until one source is empty again. This only works if sources provide their own events in-order. You could implement this by having a counter for each data source of how many events exist in the priority-queue.
Another aspect is what happens to the priority-queue when a node crashes. Using acknowledgements should help here, so that on crash the queue can be rebuilt from unacknowledged events.
Through Hazelcast's Java client I am able to successfully add an EntryListener and the listener is called as expected when entries are added or updated. So far so good.
But sometimes the client application becomes disconnected from the cluster. When this happens the client often reconnects automatically and the client has to once again add the listener. But during this brief period (between disconnection and adding the listener again), there is a chance that events are missed.
Another time that events maybe missed is if the client needs to be restart.
Is there a way to guarantee that an EntryListener receives all events?
Event listeners work in a fire&forget manner, so there is no way to receive past events.
However; there's ReliableTopic, which stores the last N events in a ringbuffer. And if you listen it with a ReliableMessageListener, it can store the event id's locally and resume from the last received event in case of a disconnection etc. Please check out the interfaces of those two.
I want to implement push notification in java so please help me out
1-Each time a new record(Message) pushed into data base(due to event created by some other user), a push notification should be sent to specific Logged in user automatically.
2-Content of the push notification should be the message present in the db.
3-If there are multiple messages, then the user should receive them one by one in a queue fashion.
4-Most important thing is the logged in user need not have to trigger any event to get notification, user should receive it automatically throughout the session.
You could use Server Sent Events. Java provides SseEmitter to send timely notifications.
You can use EventSource API in JavaScript to trigger the SSE event stream and in the server-side, loop the database query code which is wrapped by an ExecutorService - which can spin of separate thread based on the initialization.
Put SSE timeout to -1 for listening for an infinite amount of time.
Please note this answer is only a hint. Use these to explore more from the internet.
I'm trying to achieve some kind of event processing in Kafka. I've got some producers which post events to Kafka queue. I've also consumers which get the event, process it, and save processed data in DB. However, I need to be sure that EVERY event had been processed and finished. What if something crash unexpectedly during processing of event after taking it from a queue? How can I inform Kafka that this particular event is still not processed? Are there any known patterns?
Kafka streams Version 0.10.* by design has "At least once" semantics. Once you are using DB if every event has its own key you will also get "Exactly once semantic " since there is no duplications if you write to the same key.
If you want to make sure that this is correct.
Start kafka,
Generate Data,
Start DB,
Start your stream,
Make sure data is getting there,
Now stop your DB,
Kill stream while it gets some errors,
Start DB again,
And you will see that Kafka reproduces the data into your DB again.
For further reading you can go here
Could anyone please elaborate a bit on watermark and its use with respect to recreating subscription using push notification in EWS application?
I read the Microsoft provided information regarding it. But I did not get to understand accurately its usage.
It is explained as:
"The Watermark element represents an event bookmark in the mailbox event queue."
Does it mean that for every event we get new or different watermark in the notification?
Also:
"If a Subscribe request contains a watermark, the subscription is created from the watermark forward."
Does it mean that if we subscribe using a watermark previously sent to us, we can get or identify all the events occurred after it?
Does it mean that for every event we get new or different watermark in the notification?
Yes as the Events are delivered to you client you will get the Water-Mark associated for that event in the Queue.
Does it mean that if we subscribe using a watermark previously sent to us, we can get or identify all the events occurred after it?
Yes if you use a previous watermark your client is telling the server to send the events that occurred after that watermark. Watermarks are valid for 30 days but there are events that can trigger them to become invalid eg
https://blogs.msdn.microsoft.com/exchangedev/2008/07/24/transitioning-to-exchange-web-services-notifications/ . So you need to consider that if you using them for synchronisation.