I am working on a multithreaded JMS receiver and publisher code.
XML message is received from a Queue, stored procedures(takes 70 sec to execute) are called and response is send to Topic within 90 sec.
I need to handle a condition when broker is down.
i.e. a condition in which messages are received from Queue and are being processed in java, in the mean time both Queue and Topic will be down. Then how to handle those messages which are not on queue and not send to topic but are in java memory?
Different options available:
1.To use CLIENT_ACKNOWLEDGE
2.To separate publisher code from receiver code.
3.To have error utility which will take messages from log and process them and send to Topic(least preferred)
Please suggest me the right option
Use a transacted session. Consume the message and send the response under a single unit of work and explicitly call COMMIT after sending the response. Then if the broker dies while the transaction is outstanding the input message will be rolled back. If you include the DB update in a two-phase coordinated transaction then it too can be rolled back of the broker goes down. This requires the consumer and responder to be within the same thread because JMS scopes sessions per thread, but you can have several threads running sessions in parallel.
Be aware that keeping many transactions open for 90 seconds might require some tuning at the broker side.
One solution is to use db for intermediate stored procedure. So the steps would be:
1. Consume Message from Queue and store into db and use Client_Acknowledge.
2. Run stored procedures on the consumed message.
3. Once the procedure is over, send message from db to the topic.
4. Delete message once acknowledgement is recieved.
If queue and topic goes down in between, you just need to send message again when acknowledgement is not recieved from topic.
Well I am not sure whether this is the best alternative and want to see how community responds on this question.
Related
How to prevent duplicated msg from happening in Google Cloud PubSub?
Say, I have a code that handles the msg that it is subscribed for.
Say, I have 2 nodes with the same Service that has this code.
Once one has received the msg but not yet acknowledged it, another node will receive the same message. And this is where there's the problem that we have two duplicated msgs.
void messageReceiver(PubsubMessage pubsubMessage, AckReplyConsumer ackReply) {
submitHandler.handle(toMessage(pubsubMessage))
.doOnSuccess((response) -> {
log.info("Acknowledging the successfully processed message id: {}, response {}", pubsubMessage.getMessageId(), response);
ackReply.ack(); // <---- acknowledged
})
.doOnError((e) -> {
log.error("Not acknowledging due to an exception", e);
ackReply.nack();
})
.doOnTerminate(span::finish)
.subscribe();
}
What is the solution for this? Is it normal behaviour?
Google Cloud Pub/Sub uses "At-Least-Once" delivery. From the docs:
Typically, Cloud Pub/Sub delivers each message once and in the order in which it was published. However, messages may sometimes be delivered out of order or more than once. In general, accommodating more-than-once delivery requires your subscriber to be idempotent when processing messages.
This means it guarantees it will deliver the message 1:N times, so you can potentially get the message multiple times if you don't pipe it through something else that deduplicates it first. There isn't a setting you can define to guarantee exactly once delivery. The docs do reference you can get the behavior you desire using Cloud Dataflow's PubSubIO, but that solution appears to be deprecated:
You can achieve exactly once processing of Cloud Pub/Sub message streams using Cloud Dataflow PubsubIO. PubsubIO de-duplicates messages on custom message identifiers or those assigned by Cloud Pub/Sub.
Saying all of this, I've never actually seen Google Cloud Pub/Sub send a message twice. Are you sure that's really the problem you're having, or is the message being reissued because you are not acknowledging the message within the Acknowledgement Deadline (as you stated above, this defaults to 10 seconds). If you don't acknowledge it, it will get reissued. From the docs (emphasis mine):
A subscription is created for a single topic. It has several properties that can be set at creation time or updated later, including:
An acknowledgment deadline: If your code doesn't acknowledge the message before the deadline, the message is sent again. The default is 10 seconds. The maximum custom deadline you can specify is 600 seconds (10 minutes).
If that's the situation, just acknowledge your messages within the deadline and you won't see these duplicates as often.
You can use Redis from Memorystore in order to deduplicate messages. Your publisher should add trace iD to the message body just before publishing it to PubSub. On the other side client (subscriber) should check if the trace ID is in the cache - skip the message. If there is no such message - process the message and add trace ID to cache with 7-8 days expiry time (PubSub deadline is 7 days). In such a simple way You can grant the correct messages received.
All messages in a given topic have a unique messageID field:
ID of this message, assigned by the server when the message is published. Guaranteed to be unique within the topic. This value may be read by a subscriber that receives a PubsubMessage via a subscriptions.pull call or a push delivery. It must not be populated by the publisher in a topics.publish call.
You can use it to deduplicate incoming messages. No need to manually assigning ID.
It is a bit harder in distributed systems (e.g. multiple instances of consumers for a given subscription). You would need a global synchronization mechanism, the simplest would be to setup database (e.g. Redis) and use it to keep processed messages IDs.
You should take a look at Replaying and discarding messages which describes how to configure message retention.
There are two properties of subscription:
retain_acked_messages - keep acknowledge messages,
message_retention_duration - how long to keep messages.
If you do not plan to rewind your subscription to a past point in time, e.g. if you do not plan to reprocess messages or have bugs forcing you to reset your subscription you can set retain_acked_messages=false and message_retention_duration='3600s'. This will allow you to keep only last hour message IDs.
Bear in mind that PubSub message also have publish_time so you don't need to add it in your message's data. It can be used with message_id. Both of these are set by a PubSub server when it receives a message.
I am trying to implement the following scenario and I could really use and appreciate some help. I am using ActiveMQ 5.14 with camel 2.21.
In the queue, each message corresponds to a single machine. The machines connect to the queue through a single polling consumer and are indistinguishable to the consumer. The messages should be kept in the queue until one machine acknowledges that it has reached the correct machine via a separate request. After each fetch of a message said message should be locked for a certain time.
I could not find any ActiveMQ functionality that translates to my problem. My approach would be to send the message after each fetch to a second queue, which serves as a lock mechanism and send it back to the fetchable queue after the specified timeout.
Maybe a better approach would be to rollback the session after each fetch if the message has not been acknowledged by the machine.
Do you have any suggestions what a viable solution to this problem would look like?
edit: more details to clarify the situation
The application communicates to the clients via exposing a REST API to the web with two calls: GET and DELETE.
GET fetches the next message from the queue and DELETE deletes the message from the queue. I need to make sure that a message is only fetched once in a given time period and that it makes its way back to the queue if the client doesn't send a DELETE request. Currently I have a route from the rest service to a bean which fetches a message from the queue returns it to the GET request and sends it back to the queue after. On a DELETE request I dequeue the message from the queue with the given id.
I still need to find a way to ensure that the last fetched message cant be accessed for a specified time period.
I am a bit confused about the part with the indistinguishable machines, but I understood the following:
You have 1 queue with messages
You have 1 consumer
The consumer takes a message and calls a service or similar
If the call is successful the message can be deleted
If the call fails the message must be reprocessed
If these assumptions are correct, you can build a Camel route that consumes messages from the queue (transacted) and calls the service.
If the Camel route fails to complete (service returns error) and the error is not handled, the broker does a rollback and the message is redelivered (immediately)
If the route fails multiple times (when max redelivery value is reached), the message is sent to the dead letter queue (by the broker) to move it out of the way but save it
If the route succeeds the message consumption is committed to the broker and the message deleted
In such a setup you could also configure more consumers to process the messages in parallel (if the called service allows this)
This behaviour is more or less the default behaviour if
Your broker is configured as persistent (avoid message loss)
You consume messages transacted (a local transaction with the broker is enough)
Camel does not handle the errors (when they are handled, the message is committed because the broker does not "see" any error)
You get an error from the service or you can at least decide if there was a problem and throw the error yourself. The broker must get an Exception so that a rollback is done
EDIT
Based on your clarification I understand that it is the other way round than I assumed.
Well then I would probably see the two request types as "workflow steps" since they are triggered from the clients.
GET
Consume a message, send it to requestor
Add a timestamp to the message header
Send the message to another queue (let's call it delievered)
DELETE
Dequeue the message from the delievered queue
Not deleted messages
Use the timestamp header and message selectors to consume not deleted messages after a certain amount of time
Move them back to the source queue
With a second queue you have various advantages
Messages in processing cannot be consumed again and therefore need no "lock"
The source queue contains only waiting messages, the delievered queue only messages in processing
You could increase message priority when sending not deleted messages back to the source queue so they are re-consumed fast
You could also add a counter header when sending not deleted messages back to the source queue to identify messages that are failed multiple times and process them in another way.
I have Tibco EMS server, some topics and number of durable subscriptions to this topics(more than one to every topic).
My task is to delete(by receiving them with appropriate acknowledge mode) messages for specific durable subscriber.
My question: is it possible to manage subscriber's pending messages by "substitute" it with my own subscriber(with the same name, id)? And it's important not to affect topic's pending messages, in other words, delete some messages from one topic subscription, but remain those messages in other topic(the same topic) subscription.
Well, I've found the answer, just forgot to post it before.
As mentioned above, under question itself, there is no way to delete messages from topic. But I had little different task: to delete messages under specific durable subscription. And this is real(with some conditions).
Lets say, you have to delete messages from durable subscription "MySubscr". To do so you should create connection and create Durable Subscriber with same name "MySubscr". But it's not enough. If you just do so, then another durable subscriber will be created with the same name, but under connection with different ClientID . And it will operate as standalone durable connection without any impact to required "MySubscr" durable(actually, they will look like MySubscr:123 and MySubscr:567 durable subscription, where 123 and 567 are the ClientIDs, at least for TibcoEMS). To fix it, you should set ClientID explicitly to your connection by connection.setClientID() method, but you can do it only if initial connection is not connected(that's why I noted about durable subscriber, it can accumulate messages without subscriber connected).
So you should wait until subscriber will disconnect by itself(isConnected() method for TibcoEMS, I didn't see similar method in JMS API, but suppose most of implementations have something like this) or to destroy connection(with certain ClientID) manually(TibjmsAdmin.destroyConnection() method from TibcoEMS). And after this set the ClientID to your connection and get access to the messages of this subscriber. You can read messages by consuming them with Acknowledge mode Client(then they will remain in the topic), or with the mode Auto(then they will be deleted).
Important note: you can't consume some certain message, all the messages are consumed like in queue, so you can do it only one-by-one. If you found some unwanted message and wish to delete it(by consuming with autoacknowledge mode or by calling acknowledge() method on the message) then you'll lost all prior messages with it. AFAIK, there is no way to delete message without deleting prior ones.
Another important note: while you doing your messages magic it is important for the initial client not to connect again till your connection isn't closed, because it will get DublicateClientIDException(if it is using certain ClientID) or it will create another Durable Subscription which will have no access to the prior messages from the subscription.
I am going to use a Session to commit the read of a JMS message after it (and any corresponding write) has successfully completed.
However, if I have an error, and have to do a rollback, I would like to process new messages first, rather than the one that had (caused???) the error that had to be rolled back. I want to eventually reprocess the failed message, but not to fail over and over while other yet-unseen messages stall behind it, waiting for action to remove the offending message or fix the environment that made it fail.
Is this automatic? (will be using Sonic MQ, if that matters). If so, the rest of this question is moot.
Do I need to, or can I even, reset the priority of the failed message to push it further back in the queue (behind other pending messages, if any)? If I need to reset the priority, how do I make that "stick", given that I would have rolled back the transaction that initially read the message in question.
I am not aware of feature in Sonic MQ which supports your requirements out-of-the-box, but there are other options:
Use a second queue for failed messages. So failed messages are sent again on another queue. Processing could start, if the first queue is empty, for example.
Resend the message on the same queue (with same or even with a lower priority)
In both cases, after the message has been sent, there is a normal commit on the main queue.
Related: Message processing with priorities. A quote from James Shek's answer:
JMS Spec states: "JMS does not require that a provider strictly implement priority ordering of messages; however, it should do its best to deliver expedited messages ahead of normal messages."
I am trying to find an answer on of how to notify an EMS Publisher in case of a Subscriber failure.
In a case of Publisher->EMS server->Subscriber, if a Subscriber fails, I need to inform Publisher to take a corrective action.I am not bothered about durabilty/PERSIETENCE, my significance is of time. In Trading systems, If I send an market order to a Subscriber who in turn sends it to an exchange, if it fails, I need to make my Publisher publish the messages on a different topic to another Sunscriber(another exchange).
Any ideas is appreciated.
The tibjmsadmin.jar library contains methods to detect when subscribers disconnect. Easier than writing code, you can:
if you have Hawk, use the
tibjmsadmin.hma to write a Hawk rule
in the event a subscriber
disconnects, or
listen on the monitor
topic
$sys.monitor.connection.disconnect -
the body of the message tells you
which subscriber disconnected.
However these "monitoring" approaches to failing over the publisher have a significant problem - in the time it takes for you to detect the subscriber failure and redirect the publisher, some messages may get through and get stuck in the defunct queue. You don't wat this happening to any $10M trades!
EMS knows when subscribers are connected or not and you should take advantage of this.
Use a "distributed queue" and there shouldn't be any need to code logic into your application to switch to a new subscriber when it fails. This happens without message loss and maintains the order of messages. It is also good architectural practice to keep load balancing and failover logic out of your code and in the administration setup of your JMS provider.
Basically you setup multiple subscribers to a queue (each exchange represented by a subscriber). The default action will be for EMS to load-balance messages across your subscribers in a round-robin fashion. But you can set the queue to "exclusive" so that messages go to only one subscriber at a time. Then if that active subscriber fails, the messages are forwarded to another subscriber.
See the EMS manual for more details on all these topics.
Not sure if you have access, you could try looking at the ReceiverCount or ConsumerCount in either QueueInfo or TopicInfo - I believe you need the tibjms.admin package. May be you can query this before you publish and then selectively publish? Not sure what the overhead is.
Because of the nature of JMS, AFAIK no transaction states (unless you use a XA transaction - with all of that overhead) or acknowledgements will propagate through the EMS broker. I.e.acks are always between publisher and broker and consumer and broker.
Failing the above, you could try a separate ack topic for which the roles are reversed, but then the failure case is a timeout - I'm not sure this is sensible.
If you don't really care which exchange the order goes to - why not make the topic/queue exclusive and make both consumers attempt to consume - the first one to succeed will process all of the messages - if it dies, then the second one (which could be periodically retrying - may successfully connect).. Alternatively allow both to process orders off the queue - remember a message will only be ever processed by a single consumer...
I really cannot see the advantage of a decoupled messaging bus in your order flow... makes no sense to me..