Given the following publisher in node.js and the following subscriber in java (this setup is fully functional) I have the following two questions:
What should I use as the third argument in queueBind and why? Why does it works as is ("test" is a random pick)?
Is there a way to specify queue in addition to exchange in rabbit.js? If yes then how? If not then why and which module should I use instead (code example would be welcome)?
// node.js
var context = require("rabbit.js").createContext();
var pub = context.socket('PUB');
pub.connect(config.exchange);
server.post("/message/:msg", function(req, res) {
pub.write(req.params.msg, 'utf8');
res.end();
});
// java
ConnectionFactory factory = new ConnectionFactory();
factory.setHost(host);
try {
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.exchangeDeclare(exchange, "fanout");
String queueName = channel.queueDeclare().getQueue();
channel.queueBind(queueName, exchange, "test"); // Question1: what should I use as the third argument and why?
// Question2: is there a way to configure rabbit.js with a queue name instead?
//channel.queueDeclare(queueName, false, false, false, null);
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume(queueName, true, consumer);
try {
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
LOG.info("Received message: " + message);
}
} catch (InterruptedException e) {
LOG.catching(e);
} finally {
channel.close();
connection.close();
}
} catch (IOException e) {
LOG.catching(e);
}
Own answer, what I've digged up so far:
The third argument, the routing key, is what is known as topic in rabbit.js. By supplying test I am only subscribing to messages send to the test topic or without a topic set (default in rabbit.js). If I were to use topic in the publisher as well, I could use pub.publish(topic, message, encoding) instead of pub.write(message, encoding) or supply it to the connect method
Does not look so and still do not know why really. The argument goes that rabbit.js is a higher-level library and it, therefore, makes certain simplifications. Why exactly this simplification is made I do not know. However, I primarily wanted to use a single exchange for multiple communication threads, which I can also achieve by using topics/routing keys. So not a big deal.
Related
I am writing a little Kafka metrics exporter (Yes there are loads available like prometheus etc but I want a light weight custom one. Kindly excuse me on this).
As part of this I would like to know as soon as first message is received (or topic has messages) in a Kafka topic. I am using Spring Boot and Kafka.
I have the below code which gives the name of the topic and number of partitions. I want to know if the topic has messages? Kindly let me know how can I get this stat. Any lead is much appreciated!
#ReadOperation
public List<TopicManifest> kafkaTopic() throws ExecutionException, InterruptedException {
ListTopicsOptions listTopicsOptions = new ListTopicsOptions();
listTopicsOptions.listInternal(true);
ListTopicsResult listTopicsResult = adminClient.listTopics(listTopicsOptions);
Set<String> topics = listTopicsResult.names().get().stream().filter(topic -> !topic.startsWith("_")).collect(Collectors.toSet());
System.out.println(topics);
DescribeTopicsResult describeTopicsResult = adminClient.describeTopics(topics);
Map<String, KafkaFuture<TopicDescription>> topicNameValues = describeTopicsResult.topicNameValues();
List<TopicManifest> topicManifests = topicNameValues.entrySet().stream().map(entry -> {
try {
TopicDescription topicDescription = entry.getValue().get();
return TopicManifest.builder().name(entry.getKey())
.noOfPartitions(topicDescription.partitions().size())
.build();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
return null;
}).collect(Collectors.toList());
return topicManifests;
}
Create a KafkaConsumer and call endOffsets (the consumer does not need to be subscribed to the topic(s)).
#Bean
ApplicationRunner runner1(ConsumerFactory cf) {
return args -> {
try (Consumer consumer = cf.createConsumer()) {
System.out.println(consumer.endOffsets(List.of(new TopicPartition("ktest29", 0),
new TopicPartition("ktest29", 1),
new TopicPartition("ktest29", 2))));
}
};
}
Offsets stored on the topic never reduce. Getting the end offset doesn't guarantee you have a non-empty topic (start and end offsets for the topic partitions could be the same).
Instead, you will still create a consumer but set
auto.offset.reset=earliest
group.id=UUID.randomUUID()
Then subscribe and run
ConsumerRecords records = consumer.poll(Duration.ofSeconds(2));
boolean empty = records.count() == 0;
By setting auto.offset.earliest with a random group, you are guaranteed to start at the earliest offset and seek to the first available record, if it exists, at which point, you can try to poll any number of records, to see if any are returned within the specified timeout.
This should work for regular and compacted topics without needing to check committed offsets.
I would like to know how my Selenium framework can dequeue a message sitting in a message queue. I have built an application to send a JSON string containing k/v pairs to a message queue.
My architecture is as follows and separate apps:
A JSP Web Application exists accepting parameters resulting in a JSON string
A message sender exists and takes the JSON string and publishes it to a Queue
A message consumer exists and consumes the Messages. Its basically just sitting here
A Selenium Java Framework exists, but I would like to process the messages and for each message it will interpret the k/v pairs and kicks off the script.
I would like to use the messages already in the queue and process these messages within the selenium framework, how can I achieve this?
I will appreciate the help. I have edited the question with the code
This is the code snippet to send the JSON Message
public class MessageSender {
public static void main(String[] args) throws IOException {
SingleNumberLogin generateLogin = new SingleNumberLogin();
//function call to build the JSON object
String jsonQueue = generateLogin.buildJASONObject();
ConnectionFactory conFactory = new ConnectionFactory();
try {
Connection connInterface = conFactory.newConnection();
Channel mqChannel = connInterface.createChannel();
mqChannel.queueDeclare("MyQueue",false,false,false,null);
//Just assigning json to another string, then publish the message
String myMessage = jsonQueue;
mqChannel.basicPublish("","MyQueue",false ,false, null,myMessage.getBytes());
}catch (
IOException | TimeoutException e)
{
System.out.println(e.getStackTrace());
}
conFactory.setUsername("guest");
conFactory.setPassword("guest");
conFactory.setVirtualHost("/");
conFactory.setHost("localhost");
conFactory.setPort(5672);
}
}
code snippet for consumer code that I have inserted into the startup function of the automation script, so if a message arrives a single test case is executed
#BeforeTest
public static void initializeTestBaseSetup() throws Exception, IOException, TimeoutException {
ConnectionFactory conFactory = new ConnectionFactory();
Connection connInterface = conFactory.newConnection();
Channel mqChannel = connInterface.createChannel();
mqChannel.queueDeclare("MyQueue",false,false,false,null);
mqChannel.basicConsume("MyQueue", true, (consumerTag, message) -> {
//convert to byte array
String m = new String (message.getBody(), "UTF-8");
System.out.println("Message received" + m);
}, consumerTag -> {
});
}
Output JSON
JSON Message received 2020-08-28T20:39:30.845{
"NUMBER": "0000011111",
"Type": "BAU",
"User": "MyUser ",
"Email": "riidonesh#gmail.com",
}
When tested in isolation, it works perfectly fine, what I mean is that I send the message and check that the consumer receives it, adding the consumer code to my framework is where i am stuck.
I would suggest you don't think about what you have as a "selenium framework" - think of it as a "java framework".
Selenium is a set of libraries that allow you automate the web browser at a GUI level. The framework is the coded solution to facilitate creation and management of your test suite - it doesn't have to be limited to selenium and chances that's already just one of its components.
Trying to answer your question directly:
SELENIUM cannot read messages
JAVA can read messages
If your rabbitmq has a web front end then you may be able to use selenium for it, but this isn't a very efficient or a logical solution.
What you might want to consider, and what i would do, is extending your framework to use the rabbitmq libraries to process messages as you need. These libraries are designed for this task.
You say:
I would like to process the messages and for each message it will
interpret the k/v pairs and kicks off the script.
I understand this to mean that the messages are the pre-req data for the tests. If you want to read the values of a message before the test you can either:
Place the get/read in a generic #Before method
or if it's a specific message per test case, add it into the start of the test.
You're working in java so you can do whatever you want really.
To get you started, the rabbitmq tutorial starts here.
This is there hello world example for reading messages from the queue:
public class Recv {
private final static String QUEUE_NAME = "hello";
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
}
}
This is probably some silly mistake I'm missing, but here is the issue:
I am trying to insert a simple "hello" message into a Rabbit queue, with a predefined exchange and routing key.
This is the code that I am using:
private static void send_equity_task_to_rabbitmq(ConnectionFactory factory) throws IOException,TimeoutException{
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare("b", false, false, false, null);
channel.exchangeDeclare("b", "direct");
channel.basicPublish("b","b",null, "hello".getBytes());
channel.close();
connection.close();
}
public static void main(String[] argv) throws TimeoutException,IOException {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("127.0.0.1");
Date start_time= Calendar.getInstance().getTime();
Long start_time_timestamp=System.currentTimeMillis();
System.out.println("[INFO] Starting connection to queue at:"+start_time);
send_equity_task_to_rabbitmq(factory);
Long end_time_timestamp=System.currentTimeMillis();
System.out.println("[INFO] Message sent and processed successfully after:"+ (end_time_timestamp-start_time_timestamp)+" miliseconds");
}
}
The code runs without any error. However, when I check the amount of records inside the "b" queue, I get:
$ rabbitmqctl list_queues
Listing queues ...
b 0
...done.
I don't have consumers for this queue at the moment, so I assume since it has 0 records, that I am using basicPublish badly.
What could be wrong?
Thank you.
I think you need to bind the queue to the exchange. You've created a queue called "b" and an exchange called "b". The exchange will distribute messages to queues that are bound to it, using the "b" routingKey, but as the "b" queue isn't bound to the "b" exchange, the "b" exchange doesn't publish to that queue.
My aim is to put n number of messages in a for loop to a WebSphere MQ queue using WebSphere MQ java programming.
My java program will run as a standalone program.
If any exception in between , I need to rollback all the messages.
If no exception then I should commit all the messages .
The outside world should not see my messages in the queue until I complete fully.
How do I achieve this?
Updated with sample code as per reply from T.Rob:
Please check if sample code is fine ?
Does setting MQGMO_SYNCPOINT is only related to my program's invocation ?
(because similar programs running parallely will also be putting messages on the same queue and those messages should not gett affected by my program's SYNCPOINT.)
public void sendMsg() {
MQQueue queue = null;
MQQueueManager queueManager = null;
MQMessage mqMessage = null;
MQPutMessageOptions pmo = null;
System.out.println("Entering..");
try {
MQEnvironment.hostname = "x.x.x.x";
MQEnvironment.channel = "xxx.SVRCONN";
MQEnvironment.port = 9999;
queueManager = new MQQueueManager("XXXQMANAGER");
int openOptions = MQConstants.MQOO_OUTPUT;
queue = queueManager.accessQueue("XXX_QUEUENAME", openOptions, null, null, null);
pmo = new MQPutMessageOptions();
pmo.options = CMQC.MQGMO_SYNCPOINT;
String input = "testing";
System.out.println("sending messages....");
for (int i = 0; i < 10; i++) {
input = input + ": " + i;
mqMessage = new MQMessage();
mqMessage.writeString(input);
System.out.println("Putting message: " + i);
queue.put(mqMessage, pmo);
}
queueManager.commit();
System.out.println("Exiting..");
} catch (Exception e) {
e.printStackTrace();
try {
System.out.println("rolling back messages");
if (queueManager != null)
queueManager.backout();
} catch (MQException e1) {
e1.printStackTrace();
}
} finally {
try {
if (queue != null)
queue.close();
if (queueManager != null)
queueManager.close();
} catch (MQException e) {
e.printStackTrace();
}
}
}
WMQ supports both local and global (XA) units of work. The local units of work are available simply by specifying the option. Global XA transactions require a transaction manager, as mentioned by keithkreissl in another answer.
For what you described, a POJO doing messaging under syncpoint, specify MQC.MQGMO_SYNCPOINT in your MQGetMessageOptions. When you are ready to commit, issue the MQQManager.commit() or MQQManager.backout() call.
Note that the response and doc provided by ggrandes refers to the JMS and not Java classes. The Java classes use Java equivalents of the WMQ procedural API, can support many threads (doc) and even provide connection pooling (doc). Please refer to the Java documentation rather than the JMS documentation for the correct behavior. Also, I've linked to the WMQ V7.5 documentation which goes with the latest WMQ Java V7.5 client. The later clients have a lot more local functionality (tracing, flexible install path, MQClient.ini, etc.) and work with back-level QMgrs. It is highly recommended to be using the latest client and the download is free.
you only need to create a session with transaction enabled.
Session session;
// ...
boolean transacted = true;
session = connection.createSession(transacted, Session.AUTO_ACKNOWLEDGE);
try {
// ...do things...
session.commit();
} catch (Exception e) {
session.rollback();
}
// ...
WARN-NOTE: Sessions are not thread-safe ;-)
Doc Websphere MQ/JMS
If you have access to a transaction manager and more importantly an XATransaction wired up to your MQ access, you can start a transaction at the beginning of your message processing put all the messages on the queue then commit the transaction. Using the XATransactions it will not put any messages until the transaction commits. If you don't have access to that, you can do a little more plumbing by placing your messages in a local data object, wrap your code in a try/catch if no exceptions iterate through the local data object sending the messages. The issue with the later approach is that it will commit all your other processing but if a problem occurs in the sending of messages your other processing will not be rolled back.
Is there a way to remove a JMS message from an IBM MQ Queue using JMSMessageId ina Java application(not using tools)? Also are such operations vendor-specific?
Looked through the API for receive operations which are used to remove messages, but for removing specific messages, do we need to filter using MessageSelector and remove appropriately, or is there a more simple way? [checking for any available method which can be directly used]
Can you please provide tutorials/examples [can be links too] to show the API usage for such operations?
When you use JMSMessageID as the only message property in a selector, WMQ optimizes the lookup to be the same as a native WMQ API get by MQMD.MessageID which is an indexed field in the queue. Please see the JMS Message Selection topic for more details.
QueueReceiver rcvr = sess.createReceiver(inputQ, "JMSCorrelationID = '"+msgId+"'")
You can also do the same thing using native WMQ API calls using Java native code. You would do a normal GET operation but specify the message ID in the MQMD structure.
myMsg.messageId = someMsgID;
MQGetMessageOptions gmo = new MQGetMessageOptions();
myQueue.get(myMsg, gmo);
How to delete specific message form queue by using messageid?
I also have like your problem, I provide the resuable function. You just need to pass MessageId and Queue name. It is ok for me.
private void deleteMessage(String messageId, String queueName) {
try {
JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi");
JMXConnector jmxc = JMXConnectorFactory.connect(url);
MBeanServerConnection conn = jmxc.getMBeanServerConnection();
ObjectName name = new ObjectName("org.apache.activemq:type=Broker,brokerName=localhost");
BrokerViewMBean proxy = (BrokerViewMBean)MBeanServerInvocationHandler.newProxyInstance(conn, name, BrokerViewMBean.class, true);
for (ObjectName queue : proxy.getQueues()) {
QueueViewMBean queueBean = (QueueViewMBean) MBeanServerInvocationHandler.newProxyInstance(conn, queue, QueueViewMBean.class, true);
if(queueBean.getName().equals(queueName)) {
System.out.println("Deleted : " + messageId);
queueBean.removeMessage(messageId);
return;
}
}
} catch(Exception e) {
e.printStackTrace();
}
}
I use activemq-all-5.8.0.jar.