I want to understand a java program and need to modify which was developed using jms spring framework. Typically it has JMS receiver & sender, it receives a message from request queue and will invoke a job (another java program) once the job is completed the sender will send response to response queue. Have couple of questions which are below,
The request message is not deleted until response posted into response queue successfully. How its been achieved what is the logic behind it.
I want to write a functionality of writing response into flat file when sender fails to send message (by catching JMS exception). Once the sender queue is up and running i will read flat file and will send responses. The reason i need is because its involved in job processing could be in hours if job failed then input message will be read again by receiver. I want to avoid duplicate processing. Please suggest your ideas here.
Without seeing the configuration it's hard to answer these questions, but best guess is that #1 is because the app is using a transactional session. This means all updates on that session are not completed until the transaction is committed.
Just catch the exception and write the data; as long as the transaction commits (because you caught the exception) the input message will be removed.
Related
I have the following rabbitMq consumer:
Consumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, MQP.BasicProperties properties, byte[] body) throws IOException {
String message = new String(body, "UTF-8");
sendNotificationIntoTopic(message);
saveIntoDatabase(message);
}
};
Following situation can occur:
Message was send into topic successfully
Connection to database was lost so database insert was failed.
As a result we have data inconsistency.
Expected result either both action were successfully executed or both were not executed at all.
Any solutions how can I achieve it?
P.S.
Currently I have following idea(please comment upon)
We can suppose that broker doesn't lose any messages.
We have to be subscribed on topic we want to send.
Save entry into database and set field status with value 'pending'
Attempt to send data to topic. If send was successfull - update field status with value 'success'
We have to have a sheduled job which have to check rows with pending status. At the moment 2 cases are possible:
3.1 Notification wasn't send at all
3.2 Notification was send but save into database was failed(probability is very low but it is possible)
So we have to distinquish that 2 cases somehow: we may store messages from topic in the collection and job can check if the message was accepted or not. So if job found a message which corresponds the database row we have to update status to "success". Otherwise we have to remove entry from database.
I think my idea has some weaknesses(for example if we have multinode application we have to store messages in hazelcast(or analogs) but it is additional point of hypothetical failure)
Here is an example of Try Cancel Confirm pattern https://servicecomb.apache.org/docs/distributed_saga_3/ that should be capable of dealing with your problem. You should tolerate some chance of double submission of the data via the queue. Here is an example:
Define abstraction Operation and Assign ID to the operation plus a timestamp.
Write status Pending to the database (you can do this in the same step as 1)
Write a listener that polls the database for all operations with status pending and older than "timeout"
For each pending operation send the data via the queue with the assigned ID.
The recipient side should be aware of the ID and if the ID has been processed nothing should happen.
6A. If you need to be 100% that the operation has completed you need a second queue where the recipient side will post a message ID - DONE. If such consistency is not necessary skip this step. Alternatively it can post ID -Failed reason for failure.
6B. The submitting side either waits for a message from 6A of completes the operation by writing status DONE to the database.
Once a sertine timeout has passed or certain retry limit has passed. You write status to operation FAIL.
You can potentialy send a message to the recipient side opertaion with ID rollback.
Notice that all this steps do not involve a technical transactions. You can do this with a non transactional database.
What I have written is a variation of the Try Cancel Confirm Pattern where each recipient of message should be aware of how to manage its own data.
In the listener save database row with field staus='pending'
Another job(separated thread) will obtain all pending rows from DB and following for each row:
2.1 send data to topic
2.2 save into database
If we failured on the step 1 - everything is ok - data in consistent state because job won't know anything about that data
if we failured on the step 2.1 - no problem, next job invocation will attempt to handle it
if we failured on the step 2.2 - If we failured here - it means that next job invocation will handle the same data again. From the first glance you can think that it is a problem. But your consumer has to be idempotent - it means that it has to understand that message was already processed and skip the processing. This requirement is a consequence that all message brokers have guarantees that message will be delivered AT LEAST ONCE. So our consumers have to be ready for duplicated messages anyway. No problem again.
Here's the pseudocode for how i'd do it: (Assuming the dao layer has transactional capability and your messaging layer doesnt)
//Start a transaction
try {
String message = new String(body, "UTF-8");
// Ordering is important here as I'm assuming the database has commit and rollback capabilities, but the messaging system doesnt.
saveIntoDatabase(message);
sendNotificationIntoTopic(message);
} catch (MessageDeliveryException) {
// rollback the transaction
// Throw a domain specific exception
}
//commit the transaction
Scenarios:
1. If the database fails, the message wont be sent as the exception will break the code flow .
2. If the database call succeeds and the messaging system fails to deliver, catch the exception and rollback the database changes
All the actions necessary for logging and replaying the failures can be outside this method
If there is enough time to modify the design, it is recommended to use JTA like APIs to manage 2phase commit. Even weblogic and WebSphere support XA resource for 2 phase commit.
If timeline is less, it is suggested perform as below to reduce the failure gap.
Send data topic (no commit) (incase topic is down, retry to be performed with an interval)
Write data into DB
Commit DB
Commit Topic
Here failure will happen only when step 4 fails. It will result in same message send again. So receiving system will receive duplicate message. Each message has unique messageID and CorrelationID in JMS2.0 structure. So finding duplicate is bit straight forward (but this is to be handled at receiving system)
Both case will work for clustered environment as well.
Strict to your case, thought below steps might help to overcome your issue
Subscribe a listener listener-1 to your topic.
Process-1
Add DB entry with status 'to be sent' for message msg-1
Send message msg-1 to topic. Retry sending incase of any topic failure
If step 2 failed after certain retry, process-1 has to resend the msg-1 before sending any new messages OR step-1 to be rolled back
Listener-1
Using subscribed listener, read reference(meesageID/correlationID) from Topic, and update DB status to SENT, and read/remove message from topic. Incase reference-read success and DB update failed, topic still have message. So next read will update DB. Incase DB update success and message removal failed. Listener will read again and tries to update message which is already done. So can be ignored after validation.
Incase listener itself down, topic will have messages until listener reading the messages. Until then SENT messages will be in status 'to be sent'.
I'm new to RabbitMQ and want to implement asynchronous messaging of SAGA with RabbitMQ.So I used RPC example of RabbitMQ to do the task. I've one orchestrator ( RPCClient) and multiple microservices ( RPCServer). Orchestrator uses unique queues to command microservices.And each microservice uses a common queue ( Reply_ Queue) to reply orchestrator. To keep log I want to get notifications in orchestrator side, when any microservice is down for any configurable time.
I read about consumer cancellation,but it only works when I delete the queue.How to get notifications in JAVA with keeping queue messages? And is it correct way to implement saga asynchronous messaging?
To implement a reliable RPC is hard, I can't give a detail guide about how to do this. If we ignore same special failure situation, I can give a simple workaround:
First, we assume that RPCClient never fail, RPCServer may fail anytime.
RPCClient need to know which request is timeout, so it can send request message with a TTL. After RPCServer receive request message and send response message, it should ACK the request message.
If RPCServer:
has failed before consume request message
OR
has failed before send response message
The request message will be republish to Dead Letter Exchange, so RPCClient can consume to some queue binded with that exchange, it can know which request is timeout.
I have an application using MQTT implemented with the paho-mqtt-1.0.2 and I am using ActiveMQ as the broker. I have a class implementing the MqttCallback, what I am wondering is why does the client hang
#Override
messageArrived(...)
do work
mqtt.publish(TOPIC,PAYLOAD,2,false) <- here
I want to send a "response" message to the broker for the next step of the work to be done. Similar to this, I read in the docs for that callback function
It is possible to send a new message within an implementation of this callback (for example, a response to this message), but the implementation must not disconnect the client, as it will be impossible to send an acknowledgment for the message being processed, and a deadlock will occur.
Has anyone out there tried doing the above and get it to work?
I also tried using the MqttAsyncClient and that ended up with
"Error too many publishes in progress" leading to undelivered messages.
I know how to get around this issue, I'm not looking for workaround; I'm looking for receiving and publishing on the thread where messageArrived() gets executed.
Happy Hunting!
I'm running a 0.8 Kafka, and build a producer using the provided Java API.
The API functions of sending a message (or messages) return void.
Is there a way to get the status of the sent message? If it sent or failed?
This is extremely important to us since we are reading the messages from a file and we want to delete the file after all messages were sent. But if there were errors and some messages weren't sent and I delete the file it will cause a loss of a very important data.
You can configure your producer to wait until it gets n acks from the Kafka cluster (request.required.acks) so that you have some kind of guarantee that the data has been committed properly before deleting your source file.
If really you need to be sure that the message sent succeeded, you might want to consider the alternative of making the producer to be synchronous (producer.type=sync). This way, you would be able to catch any exception thrown by the blocking invocation and act accordingly. The exception thrown by send() is kafka.common.FailedToSendMessageException.
Kafka's Java API is not ideal, hope this helps you.
I would like to have an advice for this issue:
I am using Jbos 5.1.0, EJB3.0
I have system, which sending requests via UDP'S to remote modems, and suppose to wait for an answer from the target modem.
the remote modems support only UDP calls, therefor I o design asynchronous mechanism. (also coz I want to request X modems parallel)
this is what I try to do:
all calls are retrieved from Data Base, then each call will be added as a message to JMS QUE.
let's say i will set X MDB'S on that que, so I can work asynchronous. now each MDB will send UDP request to the IP-address(remote modem) which will be parsed from the que message.
so basicly each MDB, which takes a message is sending a udp request to the remote modem and [b]waiting [/b]for an answer from that modem.
[u]now here is the BUG:[/u]
could happen a scenario where MDB will get an answer, but not from the right modem( which it requested in first place).
that bad scenario cause two wrong things:
a. the sender which sent the message will wait forever since the message never returned to him(it got accepted by another MDB).
b. the MDB which received the message is not the right one, and probablly if it was on a "listener" mode, then it supposed to wait for an answer from diffrent sender.(else it wouldnt get any messages)
so ofcourse I can handle everything with a RETRY mechanisem. so both mdb's(the one who got message from the wrong sender, and the one who never got the answer) will try again, to do thire operation with a hope that next time it will success.
This is the mechanism, mybe you could tell me if there is any design pattren, or any other effective solution for this problem?
Thanks,
ray.
It's tough to define an exacting solution without knowing the details, but I will assume that when a response is received from a modem (either the correct one or not), it is possible to determine which exact modem the request came from.
If this is the case, I would separate out the request handler from the response handler:
RequestMDB receives a message from the [existing] queue, dispatches the request and returns.
A new component (call it the ResponseHandler) handles all incoming responses from the modems. The response sender is identified (a modem ID ?) and packages the response into a JMS message which is sent to a JMS Response Queue.
A new MDB (ResponseMDB) listens on the JMS Response Queue and processes the response for which the modem ID is now known.
In short, by separating concerns, you remove the need for the response processing MDB to only be able to process responses from a specific modem and can now process any response that is queued by the ResponseHandler.
The ResponseHandler (listening for responses from the modems) would need to be a multithreaded service. You could implement this as a JBoss ServiceMBean with some sort of ThreadPool support. It will need a reference to the JMS QueueConnectionFactory and the JMS response queue.
In order to handle request timeouts, I propose you create a scheduled task, one for each modem, named after the modem ID. When a request is sent, the task is scheduled for execution after a delay of the timeout period. When a response is received by the ResponseHandler, the ResponseHandler queues the response and then cancels the named task. If the timeout period elapsed without a cancellation, the scheduled task executes and queues another request (an reschedules the timeout task).
Easier said than done, I suppose, but I hope this helps.
//Nicholas