Develop Reference

How to handle exceptions with spring-jms

Hi I am using a jmslistener annotation to recieve messages from tibco queue. I am DefaultJmsListenerContainer factory with sessionTransacted = true. What I want to do is
When we get a RunTimeException I want to retry the specific message specific no of times(lets say x)
When we get cannot get jdbc connection I want to shutdown the system and want to make sure that this message is sent back to the queue to be redelivered the next time system is brought up.
What I am facing is
When I am setting sessionTransacted as true and I am throwing a RunTimeException the message is redelivered indefinitely . How can I set this configuration to redeliver the message only x times.( I have tried using message header property JMSXDeliveryCount but that does not give me the correct no of times a specific message is redelivered.)
I tried shutting down the system using System.exit(1) but this leads to deadlock and application hangs. I added another piece of code where I am shutting down the application in a different thread and making sure if in between the shutting down of the container another message is read by the listener I throw a RunTimeException so that I am able to get that message again once my system is brought up. However what I want is the 1st message for which we did not get the jdbc connection to be redelivered and no other messages to be read when I stop the container.How can we achieve this.

Where does the message head after a Patterns.ask timeout?

Been recently experiencing some timeouts with scala.concurrent.Future objects created awaiting processing within an Akka actor and I was wondering how to handle those timeout'd events. Are they really lost? Are they retried and preserved in memory or how does it work?
To put a bit of context, the code goes the following.
List<Future<MyMessage>> futureMessageList = plainMessages.stream()
.map(this::toFuture)
.collect(Collectors.toList());
Futures.sequence(futureMessageList, ExecutionContexts.global())
.onComplete(new OnComplete<Iterable<MyMessage>>() {
#Override
public void onComplete(Throwable throwable, Iterable<MyMessage> messages) {
... // iterate futureMessageList list
Within the onComplete an iteration over futureMessageList takes place, which is basically composed of Future objects which encapsulate MyMessage.
However, the function toFuture does a Patterns.ask() with a given dispatcher and that seems to be taking more than the timeout I sent (60 seconds). Take into account that the response times depend on an underlying system which may be under high load or without the fastest network depending on the environment it runs.
Future<MyMessage> message = Patterns.ask(actorSystem.getSampleDispatcher(), msg, TIMEOUT_60_SECS)
So my question is, after the onComplete throws the following exception due to the Future not being processed in time...
java.lang.NullPointerException
at my.package.Clazz.onComplete(Clazz.java:4)
at my.package.Clazz$1.onComplete(Clazz.java:5)
at akka.dispatch.OnComplete.internal(Future.scala:258)
at akka.dispatch.OnComplete.internal(Future.scala:256)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
at scala.concurrent.impl.CallbackRunnable.run$$$capture(Promise.scala:32)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala)
at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
Are those MyMessage objs saved within memory and retried afterwards? Should I somehow handle the exception and handle those timeout'd messages with an in-memory list or how should I workaround this?

When ask times out from not getting a reply it completes the Future (or CompletionStage) with a failure. The message may still be somewhere being processed and if there is a response it will end up in dead letters (https://doc.akka.io/docs/akka/current/general/message-delivery-reliability.html#dead-letters). Other scenarios where the timeout could hit are if the actor has stopped or crashed processing the message, the request or response got lost (not likely unless the responding actor is remote).
Future.sequence will either complete successfully when all futures passed to it has completed successfully or fail if any of them fails.
This means that if any of the asks time out you will get null as the messages parameter and the exception from the first failing future as the throwable parameter in your onComplete callback.
If you rather would like to get a partial list of results, each being either a successful value or an exception. You can do that with with the help of recover on each future before passing them to Future.sequence.

How to save message into database and send response into topic eventually consistent?

I have the following rabbitMq consumer:
Consumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, MQP.BasicProperties properties, byte[] body) throws IOException {
String message = new String(body, "UTF-8");
sendNotificationIntoTopic(message);
saveIntoDatabase(message);
}
};
Following situation can occur:
Message was send into topic successfully
Connection to database was lost so database insert was failed.
As a result we have data inconsistency.
Expected result either both action were successfully executed or both were not executed at all.
Any solutions how can I achieve it?
P.S.
Currently I have following idea(please comment upon)
We can suppose that broker doesn't lose any messages.
We have to be subscribed on topic we want to send.
Save entry into database and set field status with value 'pending'
Attempt to send data to topic. If send was successfull - update field status with value 'success'
We have to have a sheduled job which have to check rows with pending status. At the moment 2 cases are possible:
3.1 Notification wasn't send at all
3.2 Notification was send but save into database was failed(probability is very low but it is possible)
So we have to distinquish that 2 cases somehow: we may store messages from topic in the collection and job can check if the message was accepted or not. So if job found a message which corresponds the database row we have to update status to "success". Otherwise we have to remove entry from database.
I think my idea has some weaknesses(for example if we have multinode application we have to store messages in hazelcast(or analogs) but it is additional point of hypothetical failure)

Here is an example of Try Cancel Confirm pattern https://servicecomb.apache.org/docs/distributed_saga_3/ that should be capable of dealing with your problem. You should tolerate some chance of double submission of the data via the queue. Here is an example:
Define abstraction Operation and Assign ID to the operation plus a timestamp.
Write status Pending to the database (you can do this in the same step as 1)
Write a listener that polls the database for all operations with status pending and older than "timeout"
For each pending operation send the data via the queue with the assigned ID.
The recipient side should be aware of the ID and if the ID has been processed nothing should happen.
6A. If you need to be 100% that the operation has completed you need a second queue where the recipient side will post a message ID - DONE. If such consistency is not necessary skip this step. Alternatively it can post ID -Failed reason for failure.
6B. The submitting side either waits for a message from 6A of completes the operation by writing status DONE to the database.
Once a sertine timeout has passed or certain retry limit has passed. You write status to operation FAIL.
You can potentialy send a message to the recipient side opertaion with ID rollback.
Notice that all this steps do not involve a technical transactions. You can do this with a non transactional database.
What I have written is a variation of the Try Cancel Confirm Pattern where each recipient of message should be aware of how to manage its own data.

In the listener save database row with field staus='pending'
Another job(separated thread) will obtain all pending rows from DB and following for each row:
2.1 send data to topic
2.2 save into database
If we failured on the step 1 - everything is ok - data in consistent state because job won't know anything about that data
if we failured on the step 2.1 - no problem, next job invocation will attempt to handle it
if we failured on the step 2.2 - If we failured here - it means that next job invocation will handle the same data again. From the first glance you can think that it is a problem. But your consumer has to be idempotent - it means that it has to understand that message was already processed and skip the processing. This requirement is a consequence that all message brokers have guarantees that message will be delivered AT LEAST ONCE. So our consumers have to be ready for duplicated messages anyway. No problem again.

Here's the pseudocode for how i'd do it: (Assuming the dao layer has transactional capability and your messaging layer doesnt)
//Start a transaction
try {
String message = new String(body, "UTF-8");
// Ordering is important here as I'm assuming the database has commit and rollback capabilities, but the messaging system doesnt.
saveIntoDatabase(message);
sendNotificationIntoTopic(message);
} catch (MessageDeliveryException) {
// rollback the transaction
// Throw a domain specific exception
}
//commit the transaction
Scenarios:
1. If the database fails, the message wont be sent as the exception will break the code flow .
2. If the database call succeeds and the messaging system fails to deliver, catch the exception and rollback the database changes
All the actions necessary for logging and replaying the failures can be outside this method

If there is enough time to modify the design, it is recommended to use JTA like APIs to manage 2phase commit. Even weblogic and WebSphere support XA resource for 2 phase commit.
If timeline is less, it is suggested perform as below to reduce the failure gap.
Send data topic (no commit) (incase topic is down, retry to be performed with an interval)
Write data into DB
Commit DB
Commit Topic
Here failure will happen only when step 4 fails. It will result in same message send again. So receiving system will receive duplicate message. Each message has unique messageID and CorrelationID in JMS2.0 structure. So finding duplicate is bit straight forward (but this is to be handled at receiving system)
Both case will work for clustered environment as well.
Strict to your case, thought below steps might help to overcome your issue
Subscribe a listener listener-1 to your topic.
Process-1
Add DB entry with status 'to be sent' for message msg-1
Send message msg-1 to topic. Retry sending incase of any topic failure
If step 2 failed after certain retry, process-1 has to resend the msg-1 before sending any new messages OR step-1 to be rolled back
Listener-1
Using subscribed listener, read reference(meesageID/correlationID) from Topic, and update DB status to SENT, and read/remove message from topic. Incase reference-read success and DB update failed, topic still have message. So next read will update DB. Incase DB update success and message removal failed. Listener will read again and tries to update message which is already done. So can be ignored after validation.
Incase listener itself down, topic will have messages until listener reading the messages. Until then SENT messages will be in status 'to be sent'.

How to get the response if returned after the time out occurs

I am consuming a REST web service from Java code using Apache commons HTTP client API. If no response returns within the socket timeout value configured in the connection manager parameters, socket time out exception occurs. In such cases as the thread returns the exception to the caller class, even if the REST service returns response few secs later, will be lost.
Is it possible to create a new thread which will still listen to the service even after the timeout and just logs the response, while the main thread returns the exception to the caller class?
Is there any better way to achieve this?
Thanks.

The pattern you are most likely looking for involves asynchronous requests. For every action you post you create a unique "job" id and with that a specific URL for the job status. After starting the job, you can then query on that specific job instance's status. For example:
POST to /actions
Returns 202 Accepted & include a Location header to /actions/results/1234
Immediately GET /actions/results/1234 to ascertain it's status.
If it returns a 2xx your job is done.
If it returns 404, wait 10 seconds (or whatever) and try again.
Once you are happy with the result, issue a DELETE to /actions/results/1234 to clean up after yourself.
Of course you don't have to return 404 if the job is not done, there are other strategies for checking on the status - the key thing is that it's a subsequent call.

Getting UID of deleted message

I have an IMAPFolder with MessageCountListener that listens to messages being added / removed from the folder. Inside my messageRemoved(MessageCountEvent ...) I need to get the UID of the message that was just removed so that I can reflect those changes in my local cache.
The issue is that if i try to execute IMAPFolder.getUID(Message ...) on a deleted message I get
javax.mail.MessageRemovedException
at com.sun.mail.imap.IMAPMessage.checkExpunged(IMAPMessage.java:220)
at com.sun.mail.imap.IMAPFolder.getUID(IMAPFolder.java:1949)
at (...).IdleWatcher$1.messagesRemoved(IdleWatcher.java:64)
at javax.mail.event.MessageCountEvent.dispatch(MessageCountEvent.java:152)
at javax.mail.EventQueue.run(EventQueue.java:134)
at java.lang.Thread.run(Thread.java:856)
How can I determine the UID of the deleted message? I could go through all cached messages and check which ones still exist, however this is too resource intensive to be doing each time a message is deleted.

You cannot get the UID of something after it has been deleted. Deleting is deleting.
The classic way to solve this is to use the UID as cache key and design your program so you can cache deleted messages for a while without ill effect. For instance, if you want to display unseen mail, ask the server what's unseen right now, then ask your cache for those messages.

If you prefetch the UIDs for all messages (using the Folder.fetch method) you should be able to get the UID of a message using the Folder.getUID(Message) after it's been deleted/expunged.

Here is IMAPFolder source code. You can see what is happening in getUID method.
API Doc says :
The exception thrown when an invalid method is invoked on an expunged Message. The only valid methods on an expunged Message are isExpunged() and getMessageNumber().
I think you should cache messages UID while deletion in your MessageCounterListener may be, after when you need , you will be able to chech and get UID.