Recently I wrote a demo program to launch reactive programming with a combination of Reactor and RabbitMQ. This is my demo code :
public class FluxWithRabbitMQDemo {
private static final String QUEUE = "demo_thong";
private final reactor.rabbitmq.Sender sender;
private final Receiver receiver;
public FluxWithRabbitMQDemo() {
this.sender = ReactorRabbitMq.createSender();
this.receiver = ReactorRabbitMq.createReceiver();
}
public void run(int count) {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.useNio();
SenderOptions senderOptions = new SenderOptions()
.connectionFactory(connectionFactory)
.resourceCreationScheduler(Schedulers.elastic());
reactor.rabbitmq.Sender sender = ReactorRabbitMq.createSender(senderOptions);
Mono<AMQP.Queue.DeclareOk> queueDeclaration = sender.declareQueue(QueueSpecification.queue(QUEUE));
Flux<Delivery> messages = receiver.consumeAutoAck(QUEUE);
queueDeclaration.thenMany(messages).subscribe(m->System.out.println("Get message "+ new String(m.getBody())));
Flux<OutboundMessageResult> dataStream = sender.sendWithPublishConfirms(Flux.range(1, count)
.filter(m -> !m.equals(10))
.parallel()
.runOn(Schedulers.parallel())
.doOnNext(i->System.out.println("Message " + i + " run on thread "+Thread.currentThread().getId()))
.map(i -> new OutboundMessage("", QUEUE, ("Message " + i).getBytes())));
sender.declareQueue(QueueSpecification.queue(QUEUE))
.thenMany(dataStream)
.doOnError(e -> System.out.println("Send failed"+ e))
.subscribe(m->{
if (m!= null){
System.out.println("Sent successfully message "+new String(m.getOutboundMessage().getBody()));
}
});
try {
Thread.sleep(20000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public static void main(String[] args) throws Exception {
int count = 20;
FluxWithRabbitMQDemo sender = new FluxWithRabbitMQDemo();
sender.run(count);
}
}
I expected that after Flux emit an item, Sender must send it to RabbitMQ and after receiving RabbitMQ the Receiver must receive it.
But everything happened sequentially and this is the result I got
Message 3 run on thread 25
Message 4 run on thread 26
Message 8 run on thread 26
Message 13 run on thread 26
Message 17 run on thread 26
Message 2 run on thread 24
Message 1 run on thread 23
Message 6 run on thread 24
Message 5 run on thread 23
Message 9 run on thread 23
Message 14 run on thread 23
Message 18 run on thread 23
Message 11 run on thread 24
Message 15 run on thread 24
Message 19 run on thread 24
Message 7 run on thread 25
Message 12 run on thread 25
Message 16 run on thread 25
Message 20 run on thread 25
Sent successfully message Message 3
Sent successfully message Message 1
Sent successfully message Message 2
Sent successfully message Message 4
Sent successfully message Message 5
Sent successfully message Message 6
Sent successfully message Message 8
Sent successfully message Message 9
Sent successfully message Message 11
Sent successfully message Message 13
Sent successfully message Message 14
Sent successfully message Message 15
Sent successfully message Message 17
Sent successfully message Message 18
Sent successfully message Message 19
Sent successfully message Message 7
Sent successfully message Message 12
Sent successfully message Message 16
Sent successfully message Message 20
Get message Message 3
Get message Message 1
Get message Message 2
Get message Message 4
Get message Message 5
Get message Message 6
Get message Message 8
Get message Message 9
Get message Message 11
Get message Message 13
Get message Message 14
Get message Message 15
Get message Message 17
Get message Message 18
Get message Message 19
Get message Message 7
Get message Message 12
Get message Message 16
Get message Message 20
I do not know what to do with my code to achieve the results as expected. Can someone help me? Thanks for advance!!!
Messages are generated too fast. To see the interleaving, in dataStream add
.doOnNext(i->Thread.sleep(10))
Related
I'm supposed to be listening on two queues and processing the messages in a concurrent manner. In a single moment I should not be processing more than 10 messages. To test this, I configured my DefaultJmsListenerContainerFactory 5-5 like below:
#Bean
public ActiveMQConnectionFactory activeMQConnectionFactory() {
ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(BROKER_URL);
return activeMQConnectionFactory;
}
#Bean
public DefaultJmsListenerContainerFactory jmsFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(activeMQConnectionFactory());
factory.setSessionAcknowledgeMode(ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE);
factory.setConcurrency("5-5");
return factory;
}
And the listeners as below:
#JmsListener(id = "queue1", destination = "QUEUE1", containerFactory = "jmsFactory")
#JmsListener(id = "queue2", destination = "QUEUE2", containerFactory = "jmsFactory")
public void test(ActiveMQTextMessage message) throws InterruptedException, JMSException {
log.info("Received Task: " + message.getText());
long randomLong = (long)(Math.random() * 500);
Thread.sleep(randomLong);
log.info("Slept for " + randomLong + "ms for "+ message.getText());
message.acknowledge();
}
Is each listener assigned 5 consumers or are the 5 consumers shared between the two listeners? If the former is true, is there any way to configure such that the 5 consumers are shared?
I sent 10 requests to both queues using two for loops:
for(int i = 0; i < 10; i++) {
Queue1Sender.sendMessage("Queue1 Request: " + (i+1));
}
for(int i = 0; i < 10; i++) {
Queue2Sender.sendMessage("Queue2 Request: " + (i+1));
}
This is what the logs printed:
Received Task: Queue1 Request: 2
Received Task: Queue1 Request: 3
Received Task: Queue1 Request: 1
Received Task: Queue1 Request: 4
Received Task: Queue1 Request: 5
Received Task: Queue2 Request: 1
Received Task: Queue2 Request: 2
Received Task: Queue2 Request: 3
Received Task: Queue2 Request: 4
Received Task: Queue2 Request: 5
Received Task: Queue2 Request: 6
Received Task: Queue1 Request: 6
Received Task: Queue2 Request: 7
Received Task: Queue2 Request: 8
Received Task: Queue1 Request: 7
Received Task: Queue1 Request: 8
Received Task: Queue1 Request: 9
Received Task: Queue1 Request: 10
Received Task: Queue2 Request: 9
Received Task: Queue2 Request: 10
I can't tell whether the consumers are being shared. Is there a better testing strategy?
You will get two complete listener containers with that configuration; each with 5 consumers.
I have set up a single customer Oracle AQ. I observe messages from this queue in a Java web application with CLIENT_ACKNOWLEDGE mode. But as soon as I receive the messages in the onMessage method, the messages seems to be getting deleted from the Oracle Queue. My assumption is, the message should not get deleted unless I acknowledge them in the client. How do I stop this?
Oracle Queue schema looks like this:
BEGIN DBMS_AQADM.CREATE_QUEUE_TABLE(
Queue_table => '"TESTUSER"."myqueuetable"',
Queue_payload_type => 'TESTUSER.messageobject',
multiple_consumers => false
);
END;
/
BEGIN DBMS_AQADM.CREATE_QUEUE(
Queue_name => 'TESTUSER.myqueue',
Queue_table => 'TESTUSER.myqueuetable',
Queue_type => 0, Max_retries => 5, Retry_delay => 0
);
END;
/
BEGIN dbms_aqadm.start_queue (
queue_name => 'testuser.myqueue'
);
END;
I observer for the messages in my Java application like this
//somewhere in my app
session = queueConnection.createQueueSession(false, Session.CLIENT_ACKNOWLEDGE);
queueReceiver = ((AQjmsSession) databaseConnectionManager.getSession())
.createReceiver(databaseConnectionManager.getQueue(), Messageobject.getORADataFactory());
queueReceiver.setMessageListener(this);
//in my listener class
#Override
public void onMessage(Message message) {
AdtMessage msg = (AdtMessage) message;
try {
Messageobject message = (Messageobject) msg.getAdtPayload();
if (isUserConnected(message.userId)) {
logger.debug("Message acknowledged");
msg.acknowledge();
//handle the message. the message should be deleted now.
} else {
//i don't want the message to be deleted
}
} catch (JMSException | IllegalArgumentException | SQLException e) {
logger.error("An error occurred while sending an outgoing blob", e);
}
}
You are creating a single consumer queue, for which message gets deleted as soon as it is consumed successfully.
To retain the message, set retention_time to a value greater than zero when creating a queue.
BEGIN
DBMS_AQADM.CREATE_QUEUE(
Queue_name => 'TESTUSER.myqueue',
Queue_table => 'TESTUSER.myqueuetable',
Queue_type => 0, Max_retries => 5, Retry_delay => 0,
retention_time => 300 -- retain for next 5 minutes
);
END;
/
Refer to https://docs.oracle.com/database/121/ARPLS/d_aqadm.htm#ARPLS109
Set up kafka consumer with this configuration
kafkaconfig:
acks: 1
autoCommit: true
bootstrapServers: example.com:9092
topic: item
groupId: EWok-group
keyDeserializer: org.apache.kafka.common.serialization.StringDeserializer
valueDeserializer: org.apache.kafka.common.serialization.StringDeserializer
maxPollRecords: 1
pollMillisTime: 15
retries: 5
heartBeatInterval: 300
sessionTimeout: 100000
maxPollInterval: 30000
code
while (true) {
try {
ConsumerRecords<String, String> consumerRecords = eWokIntegrationConsumer.poll(Duration.of(kafkaCommConfig.getPollMillisTime(), ChronoUnit.SECONDS));
if (!consumerRecords.isEmpty()) {
LOG.info("Consumed Record Count: {}", consumerRecords.count());
consumerRecords.forEach(record -> {
System.out.printf("offset = %d, key = %s, value = %s\n", record.offset(), record.key(), record.value());
eWokMessageProcessor.onMessage(record.value());
eWokIntegrationConsumer.commitSync();
});
} else {
LOG.info("Polling returned without any records.");
}
} catch (Exception exception) {
LOG.error("Consumer was interrupted. But still continue to poll. Exception:", exception);
eWokIntegrationConsumer.close();
}
}
10000 ms is taking for processing the data which we have received from kafka consumer.but getting
exception saying
java.lang.IllegalStateException: This consumer has already been closed.
Exception Logs
java.lang.IllegalStateException: This consumer has already been closed.
at org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:2202)
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1332)
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1298)
Kafka version : kafka-clients-2.0.1
Could you pls suggest any one how should configurations Kafka consume looks like.
I have put System.exit(0) in other place in the source code.That is why consumer has leave the group and mark as closed.
I have remove System.exit(0) from source code.Now it's working fine.
I'm trying to use resizer in akka routing with round-robin-pool. But it is not creating the instances. It is working on the instances which I mentioned in the lower-bound. I'm following the documents of akka version 2.5.3.
My configuration :
akka.actor.deployment {
/round-robin-resizer {
router = round-robin-pool
resizer {
lower-bound = 4
upper-bound = 30
pressure-threshold = 0
rampup-rate = 0.5
messages-per-resize = 1
}
}
Actor class :
return receiveBuilder()
.match(Integer.class, msg -> {
System.out.println("Message : " + msg + " Thread id : " + Thread.currentThread().getId());
Thread.sleep(5000);
})
.matchAny(msg -> {
System.out.println("Error Message : " + msg + " Thread id : " + Thread.currentThread().getId());
}).build();
}
Creation of actor :
ActorRef roundRobin = system.actorOf(FromConfig.getInstance().props(Props.create(RoutingActor.class)), "round-robin-resizer");
for (int i = 0; i < 15; i++) {
roundRobin.tell(i, ActorRef.noSender());
}
Output :
Message : 2 Thread id : 18
Message : 1 Thread id : 16
Message : 0 Thread id : 15
Message : 3 Thread id : 17
Message : 7 Thread id : 17
Message : 4 Thread id : 15
Message : 6 Thread id : 18
Message : 5 Thread id : 16
Message : 11 Thread id : 17
Message : 9 Thread id : 16
Message : 10 Thread id : 18
Message : 8 Thread id : 15
Message : 13 Thread id : 16
Message : 14 Thread id : 18
Message : 12 Thread id : 15
After every 4 result it is waiting for 5 seconds to complete the job of the previous instances.
See the thread IDs. For every creation of actor instance I'm letting my thread to sleep some time. At the time the new instance should be allocated on different thread. But this process in running till the first three instance. After that it is not creating the new instance as per the resizer. It is appending the message as per the normal flow of round robin pool.
You are getting confused with thread-id and actual actor instance. The number of actors instances does not match with the number of threads. Please refer to this answer in other similar question: Akka ConsistentHashingRoutingLogic not routing to the same dispatcher thread consistently
I'm trying to get list of all messages I have in activeMQ queue using java and delete one of the messages based on their ID. My code looks like the following:
Connection connection = connectionFactory.createConnection("username","password");
connection.start();
Session session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE);
Destination topicDestination = session.createQueue(queue_name);
QueueBrowser browser = session.createBrowser((Queue) topicDestination);
Enumeration<?> messages = browser.getEnumeration();
int count=0;
while( messages.hasMoreElements()){
count++;
TextMessage messageInTheQueue = (TextMessage)messages.nextElement();
System.out.println("Message "+count+" in the queue:" );
System.out.println(messageInTheQueue.getJMSMessageID());
System.out.println(messageInTheQueue.getText());
System.out.println("===============================================");
System.out.println(" ");
when I run it I get the following output:
Message 1 in the queue:
ID:message1-server-42764-1483561148119-0:0:1:1:1
Today is warm
===============================================
Message 2 in the queue:
ID:message1-server-42764-1483561148119-0:0:1:1:2
Today is dry
===============================================
I use the ID I get like for example the second ID message1-server-42764-1483561148119-0:0:1:1:2 to consume or delete the message like the following:
Connection connection = connectionFactory.createConnection("username","password");
Session session = connection.createSession(true,Session.AUTO_ACKNOWLEDGE);
Destination topicDestination = session.createQueue(queue_name);
MessageConsumer consumer = session.createConsumer(topicDestination, "JMSMessageID="+message_id);
connection.start();
consumer.receive();
consumer.close();
session.commit();
session.close();
connection.stop();
but I keep getting jms exception:
javax.jms.InvalidSelectorException: JMSMessageID=message1-server-42764-1483561148119-0:0:1:1:2
at org.apache.activemq.selector.SelectorParser.parse(SelectorParser.java:47)
at org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:186)
at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:840)
at activeMQ.DeleteSingleMessage.run(DeleteSingleMessage.java:30)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.activemq.selector.TokenMgrError: Lexical error at line 1, column 51. Encountered: ":" (58), after : ""
at org.apache.activemq.selector.SelectorParserTokenManager.getNextToken(SelectorParserTokenManager.java:1057)
at org.apache.activemq.selector.SelectorParser.jj_scan_token(SelectorParser.java:1133)
at org.apache.activemq.selector.SelectorParser.jj_3R_18(SelectorParser.java:849)
at org.apache.activemq.selector.SelectorParser.jj_3R_11(SelectorParser.java:857)
at org.apache.activemq.selector.SelectorParser.jj_3R_9(SelectorParser.java:883)
at org.apache.activemq.selector.SelectorParser.jj_3_5(SelectorParser.java:916)
at org.apache.activemq.selector.SelectorParser.jj_2_5(SelectorParser.java:563)
at org.apache.activemq.selector.SelectorParser.addExpression(SelectorParser.java:323)
at org.apache.activemq.selector.SelectorParser.comparisonExpression(SelectorParser.java:172)
at org.apache.activemq.selector.SelectorParser.equalityExpression(SelectorParser.java:132)
at org.apache.activemq.selector.SelectorParser.andExpression(SelectorParser.java:96)
at org.apache.activemq.selector.SelectorParser.orExpression(SelectorParser.java:75)
at org.apache.activemq.selector.SelectorParser.JmsSelector(SelectorParser.java:67)
at org.apache.activemq.selector.SelectorParser.parse(SelectorParser.java:44)
... 4 more
I tried following this post but I'm not sure what I'm missing?
javax.jms.InvalidSelectorException: JMSMessageID=message1-server-42764-1483561148119-0:0:1:1:2
you forgot ID: of JMSMessageID
add it and wrap selector value in single quotes
String message_id = "'ID:message1-server-42764-1483561148119-0:0:1:1:2'";
MessageConsumer consumer = session.createConsumer(topicDestination, "JMSMessageID=" + message_id);