I need to process flux of message from kafka topic using reactor-kafka. What should I achieve are these steps:
Read messages from kafka topic as flux.
Try to send each message to external system.
In case of success log that event, in case of failure - send message to another topic.
Manually acknowledge processed messages (important: in order).
I've found in docs example of building reactive pipeline (just receiving message from one topic and sending it to another) with manual acknowledgement of messages:
sender.send(KafkaReceiver.create(receiverOptions)
.receive()
.map(m -> SenderRecord.create(transform(m.value()), m.receiverOffset())))
.doOnNext(m -> m.correlationMetadata().acknowledge());
But how can I send to another topic one part of processed messages and skip another part, but in the end make acknowledgements for all processed messages in order?
Related
I'm new to RabbitMQ and want to implement asynchronous messaging of SAGA with RabbitMQ.So I used RPC example of RabbitMQ to do the task. I've one orchestrator ( RPCClient) and multiple microservices ( RPCServer). Orchestrator uses unique queues to command microservices.And each microservice uses a common queue ( Reply_ Queue) to reply orchestrator. To keep log I want to get notifications in orchestrator side, when any microservice is down for any configurable time.
I read about consumer cancellation,but it only works when I delete the queue.How to get notifications in JAVA with keeping queue messages? And is it correct way to implement saga asynchronous messaging?
To implement a reliable RPC is hard, I can't give a detail guide about how to do this. If we ignore same special failure situation, I can give a simple workaround:
First, we assume that RPCClient never fail, RPCServer may fail anytime.
RPCClient need to know which request is timeout, so it can send request message with a TTL. After RPCServer receive request message and send response message, it should ACK the request message.
If RPCServer:
has failed before consume request message
OR
has failed before send response message
The request message will be republish to Dead Letter Exchange, so RPCClient can consume to some queue binded with that exchange, it can know which request is timeout.
I am a Reactor newbie. I am trying to develop the following application logic:
Read messages from a Kafka topic source.
Transform the massages.
Write a subset of the transformed messages to a new Kafka topic target.
Explicitly acknowledge the reading operation for all the messages originally read from topic source.
The only solution I found is to rewrite the above business logic as it follows.
Read messages from a Kafka topic source.
Transform the massages.
Immediately acknowledge the message not be written to topic target.
Filter all the above messages.
Write the rest of the transformed messages to the new Kafka topic target.
Explicitly acknowledge the reading operation for these messages
The code implementing the second logic is the following:
receiver.receive()
.flatMap(this::processMessage)
.map(this::acknowledgeMessagesNotToWriteInKafka)
.filter(this::isMessageToWriteInKafka)
.as(this::sendToKafka)
.doOnNext(r -> r.correlationMetadata().acknowledge());
Clearly, receiver type is KafkaReceiver, and method sendToKafka uses a KafkaSender. One of the things I don't like is that I am using a map to acknowledge some messages.
Is there any better solution to implement the original logic?
This is not exactly your four business logic steps, but I think it's a little bit closer to what you want.
You could acknowledge the "discarded" messages that won't be written in .doOnDiscard after .filter...
receiver.receive()
.flatMap(this::processMessage)
.filter(this::isMessageToWriteInKafka)
.doOnDiscard(ReceiverRecord.class, record -> record.receiverOffset().acknowledge())
.as(this::sendToKafka)
.doOnNext(r -> r.correlationMetadata().acknowledge());
Note: you'll need to use the proper object type that was discarded. I don't know what type of object the Publisher returned from processMessage emits, but I assume you can get the ReceiverRecord or ReceiverOffset from it in order to acknowledge it.
Alternatively, you could combine filter/doOnDiscard into a single .handle operator...
receiver.receive()
.flatMap(this::processMessage)
.handle((m, sink) -> {
if (isMessageToWriteInKafka(m)) {
sink.next(m);
} else {
m.getReceiverRecord().getReceiverOffset().acknowledge();
}
})
.as(this::sendToKafka)
.doOnNext(r -> r.correlationMetadata().acknowledge());
I am using Amazon SQS with Amazon SQS-JMS java library with Java EE 7. What I want to achieve is after receiving a message, depending on business logic of the application either confirm (consume) the message or resend it to the queue again and after 3 failed retries move it to DLQ.
I though about using CLIENT_Acknowledge mode in JMS and only acknowledging the messages that were successfully processed, but this is from their official documentation:
In this mode, when a message is acknowledged, all messages received before this message are implicitly acknowledged as well. For example, if 10 messages are received, and only the 10th message is acknowledged (in the order the messages are received), then all of the previous nine messages are also acknowledged.
This example also seems to confirm this: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/code-examples.html#example-synchronous-receiver-client-acknowledge-mode.
For me this is kind of a strange behavior and opposite what I would expect from a client_acknowledge. Is there a more elegant solution here than just manually sending message throughout the code to main SQS queue or DLQ depending on process status?
You can use:
UNORDERED_ACKNOWLEDGE
SQSSession.UNORDERED_ACKNOWLEDGE
Which comes from 'com.amazon.sqs.javamessaging;' and as it states in the documentation it is a variation of Client_Acknowledge which only acknowledges the message for which it is called.
/**
* Non standard acknowledge mode. This is a variation of CLIENT_ACKNOWLEDGE
* where Clients need to remember to call acknowledge on message. Difference
* is that calling acknowledge on a message only acknowledge the message
* being called.
*/
dependency example:
"com.amazonaws:amazon-sqs-java-messaging-lib:1.0.3"
To handle this case you can use RedrivePolicy attribute for the DLQ that you created. Solution for this case can be:
Create a 2 sqs Qs say my_q and my_q_dl (latter one is for DLQ)
Set DLQ my_q_dl as the DLQ of my_q by using RedrivePolicy.
Here, care should be taken to specify deadLetterTargetArn and maxReceiveCount. This maxReceiveCount is the number of times you want to process any message without acknowledging before sending it to the DLQ. If you set maxReceiveCount=3 then, the msg will remain in my_q up to 3rd pull by the consumer with no ack.
2 cases here:
Normal case: msg gets deleted as soon as ack is received.
If no ack (msg delete) for that msg upto third time then the msg gets deleted from my_q and pushed to
my_q_dl itself.
*RedrivePolicy - The string that includes the parameters for the deadletter queue functionality of the source queue.
deadLetterTargetArn - The Amazon Resource Name (ARN) of the dead-letter queue to which Amazon SQS moves messages after the value
of maxReceiveCount is exceeded.
maxReceiveCount - The number of times a message is delivered to the source queue before being moved to the dead-letter queue.
Note
The dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter queue of a standard queue must also be a
standard queue.*
Using Spring AMQP (Using RabbitMQ as message broker), I am preparing a message and I want my message to consume after sometimes. Till then it can wait in some queue like waiting queue and then moved to our main queue where we have the consumer which is waiting to process the message from main queue.
I got confused whether I should apply dead letter exchange in this scenario and how to apply the dead letter exchange in it is the big question for me.
Any Idea how can we make it work.
P.S > If it is possible without rabbitmq_delayed_message_exchange plugins.
If you don't want to use the delayed exchange plugin, you can send a message to a queue with a time to live (ttl set on the queue or message).
Configure the queue to route expired messages to a dead letter exchange which routes to the final queue.
someExchange -> ttlQueueWithDLX -> DLX -> liveQueue
I have a Queue and Topic with 2 messages in Activemq.If I restart Activemq.I am losing messages and also Topic.
Even If I restart Activemq,I don't want to lose any messages from any Topicand Queue.Is it possible.
I am using Activemq 5.8.0.
A producer produces the message and send it to the Topic, which ever
consumer is running at that point of time, will receive the message.
If you want consumer which is not up now, but might be running in
future to get this message, you will have to tell the Broker to
persist the message and store the information that this perticular
consumer has not received the message.
If you have working code with-out durable subscriber, you will have to do the following changes.
In the consumer,
1. set the clinetId. Because Topic should know which consumer is yet to receive the message. Or has received the message.
Connection.setClientID(String)
2. Should be creating a durable subscriber for your topic
Connection.createDurableSubscriber()
3. Add your listener to this subscriber.
subscriber.setMessageListener(yourlistener)
4. Once you receive the message, you will have to acknowledge it
This link shows how it is done: But its in c# i guess.
http://myadventuresincoding.wordpress.com/2011/08/16/jms-how-to-setup-a-durablesubscriber-with-a-messagelistener-using-activemq/
Read these links for more info :
http://activemq.apache.org/how-do-durable-queues-and-topics-work.html
http://activemq.apache.org/why-do-i-not-receive-messages-on-my-durable-topic-subscription.html
http://activemq.apache.org/manage-durable-subscribers.html