I am building a system that will receive messages via a Message broker (Currently, JMS) from different systems. All the messages from all the senders systems have a deviceId and there is no order in the reception of the message.
For instance, system A can send a message with deviceId=1 and system b be can send a message with deviceId=2.
My goal is not to start processing of the messages concerning the same deviceId unless I got all the message from all the senders with the same deviceId.
For example, if I have 3 systems A, B and C sending messages to my system :
System A sends messageA1 with deviceId=1
System B sends messageB1 with deviceId=1
System C sends messageC1 with deviceId=3
System C sends messageC2 with deviceId=1 <--- here I should start processing of messageA1, messageB1 and messageC2 because they are having the same deviceID 1.
Should this problem be resolved by using some sync mechanism in my system , by the message broker or an integration framework like spring-integration/apache camel ?
A similar solution with the Aggregator (what #Artem Bilan mentioned) can also be implemented in Camel with a custom AggregationStrategy and with controlling the Aggregator completion by using the Exchange.AGGREGATION_COMPLETE_CURRENT_GROUP property.
The following might be a good starting point. (You can find the sample project with tests here)
Route:
from("direct:start")
.log(LoggingLevel.INFO, "Received ${headers.system}${headers.deviceId}")
.aggregate(header("deviceId"), new SignalAggregationStrategy(3))
.log(LoggingLevel.INFO, "Signaled body: ${body}")
.to("direct:result");
SignalAggregationStrategy.java
public class SignalAggregationStrategy extends GroupedExchangeAggregationStrategy implements Predicate {
private int numberOfSystems;
public SignalAggregationStrategy(int numberOfSystems) {
this.numberOfSystems = numberOfSystems;
}
#Override
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
Exchange exchange = super.aggregate(oldExchange, newExchange);
List<Exchange> aggregatedExchanges = exchange.getProperty("CamelGroupedExchange", List.class);
// Complete aggregation if we have "numberOfSystems" (currently 3) different messages (where "system" headers are different)
// https://github.com/apache/camel/blob/master/camel-core/src/main/docs/eips/aggregate-eip.adoc#completing-current-group-decided-from-the-aggregationstrategy
if (numberOfSystems == aggregatedExchanges.stream().map(e -> e.getIn().getHeader("system", String.class)).distinct().count()) {
exchange.setProperty(Exchange.AGGREGATION_COMPLETE_CURRENT_GROUP, true);
}
return exchange;
}
#Override
public boolean matches(Exchange exchange) {
// make it infinite (4th bullet point # https://github.com/apache/camel/blob/master/camel-core/src/main/docs/eips/aggregate-eip.adoc#about-completion)
return false;
}
}
Hope it helps!
You can do this in Apache Camel using a caching component. I think there is the EHCache component.
Essentially:
You receive a message with a given deviceId say deviceId1.
You look up in your cache to see which messages have been received for deviceId1.
As long as you have not received all three you add the current system/message to the cache.
Once all messages are there you process and clear the cache.
You could then off course route each incoming message to a specific deviceId based queue for temporary storage. This can be JMS, ActiveMQ or something similar.
Spring Integration provides component for exactly this kind of tasks - do not emit until the whole group is collected. And it's name an Aggregator. Your deviceId is definitely a correlationKey. The releaseStrategy really may be based on the number of systems - how much deviceId1 messages you are waiting before proceed to the next step.
Related
I am currently implementing a Java messaging system with Apache Camel and ActiveMQ. My goal is to dynamically set the priority of a message based on a few attributes the message has.
I already configured my ActiveMQ as explained here. Then I created the following method that sends a TextMessage:
public void send(BaseMessage baseMessage, int jmsPriority) throws JsonProcessingException {
Map<String, Object> messageHeaders = new HashMap<>();
messageHeaders.put(MESSAGING_HEADER_JMS_PRIORITY, jmsPriority);
messageHeaders.put(MESSAGING_HEADER_TYPE, baseMessage.getClass().getSimpleName());
String payload = objectMapper.writeValueAsString(baseMessage);
producerTemplate.sendBodyAndHeaders(payload, messageHeaders);
}
Sending the message perfectly works, and the dynamic type of BaseMessage is properly set to the header of each message. The priority is set as well, but is ignored. The order for the outcoming messages is still FIFO, as queues usually do.
Until now I did not achieve to set the priority of the message dynamically. I do not want to use Apache Camel's Resequencer since I would have to create several new queues only for "sorting". From my point of view ActiveMQ must be able to prioritize and reorder the messages itself.
Any tip is appreciated. Ask me for further details if required.
By default, ActiveMQ disables message priority. This is normal. When doing distributed messaging-- sending messages across servers, prioritization does not practically work out, since the broker can only scan so many messages in the queue for messages of a higher priority before it stats to slow down all traffic for that queue.
Prioritized messages can work well when embedding a broker and using it for task dispatch-- where queue depth generally doesn't exceed the low-thousands.
Updated:
Reminder-- the QOS settings in JMS must be set on the MessageProducer object, and not the message per JMS-spec.
Enable Prioritized Messages
I'm writing an application with Spring Boot so to write to Kafka I do:
#Autowired
private KafkaTemplate<String, String> kafkaTemplate;
and then inside my method:
kafkaTemplate.send(topic, data)
But I feel like I'm just relying on this to work, how can I know if this has worked? If it's asynchronous, is it a good practice to return a 200 code and hoped it did work? I'm confused. If Kafka isn't available, won't this fail? Shouldn't I be prompted to catch an exception?
Along with what #mjuarez has mentioned you can try playing with two Kafka producer properties. One is ProducerConfig.ACKS_CONFIG, which lets you set the level of acknowledgement that you think is safe for your use case. This knob has three possible values. From Kafka doc
acks=0: Producer doesn't care about acknowledgement from server, and considers it as sent.
acks=1: This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers.
acks=all: This means the leader will wait for the full set of in-sync replicas to acknowledge the record.
The other property is ProducerConfig.RETRIES_CONFIG. Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error.
Yes, if Kafka is not available, that .send() call will fail, but if you send it async, no one will be notified. You can specify a callback that you want to be executed when the future finally finishes. Full interface spec here: https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/producer/Callback.html
From the official Kafka javadoc here: https://kafka.apache.org/20/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
Fully non-blocking usage can make use of the Callback parameter to
provide a callback that will be invoked when the request is complete.
ProducerRecord<byte[],byte[]> record = new ProducerRecord<byte[],byte[]>("the-topic", key, value);
producer.send(myRecord,
new Callback() {
public void onCompletion(RecordMetadata metadata, Exception e) {
if(e != null) {
e.printStackTrace();
} else {
System.out.println("The offset of the record we just sent is: " + metadata.offset());
}
}
});
you can use below command while sending messages to kafka:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-name
while above command is running you should run your code and if sending messages being successful then the message must be printed on the console.
Furthermore, likewise any other connection to any resources if the connection could not be established, then doing any kinds of operations would result some exception raises.
I am trying to send messages to Service bus using AMQP QPID java library
I am getting this error:
"SessionId needs to be set for all brokered messages to a Partitioned
Topic that supports Ordering"
My topic has "Enforce Message ordering" turned on (this is way i get this error i guess)
When using the Azure Service bus java library (and not AMQP) i have this function :
this.entity.setSessionId(...);
When using the AMQP library i do not see an option to set the session ID on the message i want to send
Note that if i un-check the option "Enforce Message ordering" the message will be sent successfully
This is my code
private boolean sendServiceBusMsg(MessageProducer sender,Session sendSession) {
try {
// generate message
BytesMessage createBytesMessage = (BytesMessage)sendSession.createBytesMessage();
createBytesMessage.setStringProperty(CAMPAIGN_ID, campaignKey);
createBytesMessage.setJMSMessageID("ID:" + bm.getMessageId());
createBytesMessage.setContentType(Symbol.getSymbol("application/octet-stream"));
/*message is the actual data i send / not seen here*/
createBytesMessage.writeBytes(message.toByteArray());
sender.send(createBytesMessage);
} catch (JMSException e) {
}
The SessionId property is mapped to AMQP message properties.group-id. The Qpid JMS client should map it to JMSXGroupID property, so try the following,
createBytesMessage.setStringProperty("JMSXGroupID", "session-1");
As you guessed, there is a similar SO thread Azure Service Bus topics partitioning verified that to disable the feature Enforce Message Ordering via set SupportOrdering with false can solve the issue, but it can't be done via Azure Service Bus Java library because the property supportsOrdering is privated now.
And you can try to set property Group as #XinChen said using AMQP, as the content below from here.
Service Bus Sessions, also called 'Groups' in the AMQP 1.0 protocol, are unbounded sequences of related messages. ServiceBus guarantees ordering of messages in a session.
Hope it helps.
I am trying to fetch a message with a particular correlation id like explained in rabbitmq docs. However I see that the irrelevant messages gets dequeued. I do not want it to happen. How can I tell rabbitmq to not dequeue after I get message and get to know that this is not the one I was looking for. Please help me.
`
.
.
replyQueueName = channel.queueDeclare().getQueue();
consumer = new QueueingConsumer(channel);
channel.basicConsume(replyQueueName, false, consumer);
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
System.out.println(delivery.getProperties().getCorrelationId());
if (delivery.getProperties().getCorrelationId().equals(corrId)) {
response = new String(delivery.getBody());
break;
}
}
`
You can't do what you want, the way you want. The "selective consumer" is an anti-pattern in RabbitMQ.
Instead, you should design your RabbitMQ setup so that your messages are routed to a queue that only contains messages for the intended consumer.
I wrote more about this, here: http://derickbailey.com/2015/07/22/airport-baggage-claims-selective-consumers-and-rabbitmq-anti-patterns/
If you can afford to lose the order of messages you can use the re-queueing mechanism.
Try turning off auto ack.
If not, you have to redesign your application to inject headers or routing keys to route to a particular queue.
On JConsole, We can see following route Statistics.
Minimum / Maximum / Mean Processing Time
First / last Message completion Time
Number of messages failed or re-delivered.
Total number of transaction processed
Requirement: I need to show above data on web page.
Below is my code:
public void process(Exchange exchange) throws Exception {
CamelContext context = exchange.getContext();
List<Route> routeObj = context.getRoutes();
for (Route routeId : routeObj) {
boolean started = context.getRouteStatus(strRouteId).isStarted();
boolean stopped = context.getRouteStatus(strRouteId).isStopped();
boolean suspended = context.getRouteStatus(strRouteId).isSuspended();
// TODO: find min/max/mean processing time, first/last message
// completion time, etc.
}
}
Thanks in advance.
Please suggest me how to get min/max/mean processing time, first/last message completion time, etc.
See for example the Camel Karaf commands that can dump statistics too. They use the JMX API to do that.
An example is the context-info command: https://github.com/apache/camel/blob/master/platforms/karaf/commands/src/main/java/org/apache/camel/karaf/commands/ContextInfo.java
Apache camel exposes these information using JMX.
A good starting point is the official JMX tutorial and the Apache Camel JMX Documentation
You can actually calculate the info you require, using org.apache.camel.management.PublishEventNotifier
One type of events will get notified of is concerning camel exchanges (like completion, failure...) of each route. The only piece of information you need after that is the processing time of a this exchange (last exchange) which is obtainable using JMX (LastProcessingTime).
Once you have the exchanges processing time for each route, all the information you require can be calculated in real-time.