I have a requirement that a single JMS message sent by a client must be delivered reliably (exactly-once) to two systems. These 2 systems are not HA-enabled, so the best suggestion that I came up with is to:
create single queue where client posts to
set up two "intermediate" queues
use a custom "DuplicatorMDB" that will read messages from the client queue and post them to two queues within the same transaction.
client->JMSDQ->DuplicatorMDB->Q1->MDB->System1
\->Q2->MDB->System2
Is there any existing functionality like that? What would be the proper way to balance the system to keep it stable if one or both of the backend systems are down?
The application server is WebLogic 10.
I can't use topics for this because in a cluster topics will cause too much message duplication. If we have 2 instances, then with topics it'll go like this:
client->Topic-->MDB1#server1->System1
| \->MDB2#server1->System2
\---->MDB1#server2->System1
\--->MDB2#server2->System2
Thus every message will be delivered twice to System1 and twice to System2 and if there'll be 8 servers in a cluster, each message will be delivered 8 times. This is what I'd really like to avoid...
Finally I got some time to test it and here is what I observed:
2 nodes in a cluster. 2 JMS servers: jms1 on node1, jms2 on node2.
Distributed topic dt. MDB with durable subscription and jms-client-id=durableSubscriber. Started the system: 0 messages, mdb#node1 is up, mdb#node2 trying to connect periodically, but it can't because "Client id, durableSubscriber, is in use". As expected.
Sent in 100 messages:
jms1#dt messages current = 0, messages total = 100, consumers current = 1
I can see that node1 processed 100 messages.
jms2#dt messages current = 100, messages total = 100 , consumers current = 1
i.e. "duplicate" messages are pending in the topic.
Sent in another 100 messages, 100 were processed on the node1, 200 pending on node2.
Rebooted node1, mdb#node2 reconnected to dt and started processing "pending" messages. 200 messages were processed on node2.
After node1 is up, mdb#node1 can't connect to the dt, while mdb#node2 is connected.
jms1#dt messages current=0, messages total = 0, consumers current = 0
jms2#dt messages current=0, messages total = 200, consumers current = 1
Send in 100 more messages, I see that all 100 messages are processed on node2 and discarded on node1.
jms1#dt messages current=0, messages total = 100, consumers current = 0
jms2#dt messages current=0, messages total = 300, consumers current = 1
Now I reboot node2, mdb#node1 reconnects to dt. After reboot mdb#node2 reconnects to dt and mdb#node1 gets disconnected from dt.
jms1#dt messages current=0, messages total = 100, consumers current = 1
jms2#dt messages current=0, messages total = 0, consumers current = 1
I send in 100 messages, all are processed on node2 and stored in the topic on node1:
jms1#dt messages current=100, messages total = 200, consumers current = 1
jms2#dt messages current=0, messages total = 0, consumers current = 1
Then I shut down node2 and I see 100 "pending messages" being processed on node1 after mdb#node1 reconnects to the topic.
So the result is:
I sent 400 messages, 700 were processed by MDB out of which 300 were duplicates.
It looks like the MDB reconnection works good as expected, but the messages may be duplicated if a node hosting the "active" MDB goes down.
This might be a bug or a feature of weblogic JMS implementation.
I haven't used Weblogic, but most JMS solutions have the concept of Queues and Topics. You want a JMS Topic. Subscribers register and the topic ensures that the message is delivered to each subscriber once.
Configuration details.
Update: If you are running into issues in a clustered environment, I would make sure that everything is configured properly (here is a guide for JMS Topic Clustering). It definitely sounds strange that Weblogic would fail so miserably when clustering. If that does not work, you could look into a 3rd party Messaging Queue, such as RabbitMQ, which support JMS and will definitely not have this issue.
This is the kind of behaviour that an ESB implementation should enbale. In terms of processing overhead there would be no great difference, but it can be useful to have separation of concerns between "plumbing" and application code.
As it happens, the WebSphere JMS implementaion has support for installing mediations that address such requirements. I don't know whether WebLogic has something similar, or whether their associated ESB products are an option for you, but I would recommend investigating those capabilities. You currently have a simple requirement, and your code is surely sufficient, however it's quite easy to imagine how a few minor additional requirements (could we just convert this field from dollars to pounds before we transmit to that destination, could we not send messages with this content to that destination ...) and lo! you find yourself writing your own ESB.
[...] Thus every message will be delivered twice to System1 and twice to System2 and if there'll be 8 servers in a cluster, each message will be delivered 8 times. This is what I'd really like to avoid...
This is right for non-durable subscriptions, but not for durable. For durable, all MDBs share the same connection-id and subscription-id (based on the MDB name by default), so only one MDB will be able to attach and receive messages at a time. The first MDB to try will succeed in connecting, the others will detect a conflict and fail, but keep retrying. So using a durable topic subscription should do the trick.
Related
I am using Amazon SQS with Amazon SQS-JMS java library with Java EE 7. What I want to achieve is after receiving a message, depending on business logic of the application either confirm (consume) the message or resend it to the queue again and after 3 failed retries move it to DLQ.
I though about using CLIENT_Acknowledge mode in JMS and only acknowledging the messages that were successfully processed, but this is from their official documentation:
In this mode, when a message is acknowledged, all messages received before this message are implicitly acknowledged as well. For example, if 10 messages are received, and only the 10th message is acknowledged (in the order the messages are received), then all of the previous nine messages are also acknowledged.
This example also seems to confirm this: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/code-examples.html#example-synchronous-receiver-client-acknowledge-mode.
For me this is kind of a strange behavior and opposite what I would expect from a client_acknowledge. Is there a more elegant solution here than just manually sending message throughout the code to main SQS queue or DLQ depending on process status?
You can use:
UNORDERED_ACKNOWLEDGE
SQSSession.UNORDERED_ACKNOWLEDGE
Which comes from 'com.amazon.sqs.javamessaging;' and as it states in the documentation it is a variation of Client_Acknowledge which only acknowledges the message for which it is called.
/**
* Non standard acknowledge mode. This is a variation of CLIENT_ACKNOWLEDGE
* where Clients need to remember to call acknowledge on message. Difference
* is that calling acknowledge on a message only acknowledge the message
* being called.
*/
dependency example:
"com.amazonaws:amazon-sqs-java-messaging-lib:1.0.3"
To handle this case you can use RedrivePolicy attribute for the DLQ that you created. Solution for this case can be:
Create a 2 sqs Qs say my_q and my_q_dl (latter one is for DLQ)
Set DLQ my_q_dl as the DLQ of my_q by using RedrivePolicy.
Here, care should be taken to specify deadLetterTargetArn and maxReceiveCount. This maxReceiveCount is the number of times you want to process any message without acknowledging before sending it to the DLQ. If you set maxReceiveCount=3 then, the msg will remain in my_q up to 3rd pull by the consumer with no ack.
2 cases here:
Normal case: msg gets deleted as soon as ack is received.
If no ack (msg delete) for that msg upto third time then the msg gets deleted from my_q and pushed to
my_q_dl itself.
*RedrivePolicy - The string that includes the parameters for the deadletter queue functionality of the source queue.
deadLetterTargetArn - The Amazon Resource Name (ARN) of the dead-letter queue to which Amazon SQS moves messages after the value
of maxReceiveCount is exceeded.
maxReceiveCount - The number of times a message is delivered to the source queue before being moved to the dead-letter queue.
Note
The dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter queue of a standard queue must also be a
standard queue.*
I would like to fine tune the KafkaTemplate options for the Producer, to handle as optimally as possible, the various failover and recovery scenarios.
We have our KafkaProducerMessageHandler running in sync mode (i.e. waiting for the send operation results - see: acks below). Note: this is necessary in the current version of Kafka to enable ErrorChannel reporting.
Here are the options I have choosen:
acks = 1 (we are performing basic acknowlegement from the Kafka
broker leader)
retries = 10
max.in.flight.requests.per.connection = 1 (This will keep the messages in order, if an error state is reached)
linger.ms = 1 (not sure about this one or whether it is relevant?)
request.timeout.ms = 5000
(five seconds for timeout, this will work with the retries - so
total time of 50 seconds, before the message is deemed to have failed and will then appear on the error channel)
enable.idempotence = false (again, not sure about this option?)
retry.backoff.ms = 100 (this is the default - again is it worth playing with?)
How do these values sound?
Is there anything I am missing?
This is an old post about Kafka producer tuning: http://ingest.tips/2015/07/19/tips-for-improving-performance-of-kafka-producer/
TLDR version:
Pay attention on the 'batch.size' and 'linger.ms' parameters.
I have a rabbitmq queue and two spring cloud spring consumers.
I want that each consumers process messages in order.
I thought that when consumer1 send ack, consumer2 receive second message,
so I expected message1, message2 is processed in order in each consumers.
-------------------- time pass ------------------------>
consumer1: message1 message3
consumer2: message2 message4
But it wasn't. consumer1, consumer2 receive message1, message2, and process simultaneously.
-------------------- time pass ------------------------>
consumer1: message1 message3
consumer2: message2 message4
Is there a way for spring cloud stream to consume messages exclusively?
RabbitMQ (AMQP) doesn't support that; each consumer gets prefetch messages.
It does support exclusive consumers, but it means consumer1 would get all the messages and consumer2 would only get messages if consumer1 dies.
However, Spring Cloud Stream doesn't currently provide a property to set that option.
you would have to model your queues in a different way. E.g. by having an "incoming" queue which has exactly one consumer-coordinator. This consumer would relay messages to the "work" queue where consumer1+2 are both waiting and pick up work in a round robin way.
They would then signal completion to the coordinator on a third queue which would cause it to resume relaying a single message to the work queue.
I haven't been able to figure this one out from Google alone. I am connecting to a non-durable EMS topic, which publishes updates to a set of data. If I skip a few updates, it doesn't matter, as the following update will overwrite it anyway.
The number of messages being published on the EMS topic is quite high, and occasionally for whatever reason the consumer lags behind. Is there a way, on the client connection side, to determine a 'time to live' for messages? I know there is on other brokers, but specifically on Tibco I have been unable to figure out whether it's possible or not, only that this parameter can definitely be set on the server side for all clients (this is not an option for me).
I am creating my connection factory and then creating an Apache Camel jms endpoint with the following code:
TibjmsConnectionFactory connectionFactory = new TibjmsConnectionFactory();
connectionFactory.setServerUrl(properties.getProperty(endpoints.getServerUrl()));
connectionFactory.setUserName(properties.getProperty(endpoints.getUsername()));
connectionFactory.setUserPassword(properties.getProperty(endpoints.getPassword()));
JmsComponent emsComponent = JmsComponent.jmsComponent(connectionFactory);
emsComponent.setAsyncConsumer(true);
emsComponent.setConcurrentConsumers(Integer.parseInt(properties.getProperty("jms.concurrent.consumers")));
emsComponent.setDeliveryPersistent(false);
emsComponent.setClientId("MyClient." + ManagementFactory.getRuntimeMXBean().getName() + "." + emsConnectionNumber.getAndIncrement());
return emsComponent;
I am using tibjms-6.0.1, tibjmsufo-6.0.1, and various other tib***-6.0.1.
The JMSExpiration property can be set per message or, more globally, at the destination level (in which case the JMSExpiration of all messages received in this destination is overridden). It cannot be set per consumer.
One option would be to create a bridge from the topic to a custom queue that only your consumer application will listen to, and set the "expiration" property of this queue to 0 (unlimited). All messages published on the topic will then be copied to this queue and won't ever expire, whatever their JMSExpiration value.
I need a setup where messages can be transmitted to a queue or topic which is listened to by 2 or more servers.
The consumer is a specific client who will access one of those 2 servers and it is not known ahead of time which server the client will check from. The message will have an ID on it which correlates to the correct client.
There may be multiple messages at any time waiting to be consumed by various clients accessing these servers.
How can I accomplish this? Queue, topic? point-to-point or publish subscribe? What exact setup would do this trick?
Here's another way to look at the scenario: imagine multiple towns which have a community mail box. The residents of these towns do not have specific addresses rather they are constantly moving around between the towns. Someone needs to send a message to another person, so they create the mail and it gets copied and routed to each town's mailbox waiting to be received. When the right person checks and finds the message addressed to him, the message is consumed and destroyed on all the other mailboxes ensuring the same message is not read again.
So the JMS queue or topic is this mailbox, and the clients connecting to these servers (which specifically are web servers in a clustered environment) are the people. Multiple messages addressed to different people can exist at the same time.
Whats the best way to do this using JMS?
If you need to address messages to specific clients, you can use consumer-side selectors, here's a round trip example:
Server Sends
QueueSender queueSender = queueSession.createSender(queue);
queueSender.setDeliveryMode(DeliveryMode.PERSISTENT);
TextMessage message = queueSession.createTextMessage("Hello John!");
message.setObjectProperty("ToAddress", "John-123");
queueSender.send(message);
Consumer Receives
QueueConnection queueConn = connFactory.createQueueConnection();
QueueSession queueSession = queueConn.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
QueueReceiver queueReceiver =
queueSession.createReceiver(queue, "ToAddress = 'John-123'");
queueConn.start();
TextMessage message = (TextMessage) queueReceiver.receive();
The client creates a queueReceiver using the selector ToAddress=John-123, so only messages that match that selector are delivered to that client; other messages go to different consumers based on their selector.
If queuereceiver 'John-123' is not connected, any messages addressed to him simply accumulate in the queue. If you want to receive messages in real time, the receiver needs to be connected, always. To check for messages intermittently (sort of like checking email a few times a day), there's not too much overhead associated with creating a receiver, checking for messages, then disconnecting, however, avoid doing that repeatedly (1000's of times, or more); if that's the case, just keep the receiver connected all the time.
Hope that helps,