I have an application that queues and deques messages from Oracle AQ using the JMS interface. When the application is running items get queued and dequeued and I can see queued items in the queue table. However, one the application shuts down the queue table is cleared and the application cannot access the previously queued items. Any idea what might cause that behavior?
The Oracle AQ is created using this code:
BEGIN
dbms_aqadm.create_queue_table(
queue_table => 'schema.my_queuetable',
sort_list =>'priority,enq_time',
comment => 'Queue table to hold my data',
multiple_consumers => FALSE, -- THis is necessary so that a message is only processed by a single consumer
queue_payload_type => 'SYS.AQ$_JMS_OBJECT_MESSAGE',
compatible => '10.0.0',
storage_clause => 'TABLESPACE LGQUEUE_IRM01');
END;
/
BEGIN
dbms_aqadm.create_queue (
queue_name => 'schema.my_queue',
queue_table => 'schema.my_queuetable');
END;
/
BEGIN
dbms_aqadm.start_queue(queue_name=>'schema.my_queue');
END;
/
I also have a Java class for connecting to the queue, queueing items and processing dequeued items like this:
public class MyOperationsQueueImpl implements MyOperationsQueue {
private static final Log LOGGER = LogFactory.getLog(MyOperationsQueueImpl.class);
private final QueueConnection queueConnection;
private final QueueSession producerQueueSession;
private final QueueSession consumerQueueSession;
private final String queueName;
private final QueueSender queueSender;
private final QueueReceiver queueReceiver;
private MyOperationsQueue.MyOperationEventReceiver eventReceiver;
public MyOperationsQueueImpl(DBUtils dbUtils, String queueName) throws MyException {
this.eventReceiver = null;
this.queueName = queueName;
try {
DataSource ds = dbUtils.getDataSource();
QueueConnectionFactory connectionFactory = AQjmsFactory.getQueueConnectionFactory(ds);
this.queueConnection = connectionFactory.createQueueConnection();
// We create separate producer and consumer sessions because that is what is recommended by the docs
// See: https://docs.oracle.com/javaee/6/api/javax/jms/Session.html
this.producerQueueSession = this.queueConnection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
this.consumerQueueSession = this.queueConnection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
this.queueSender = this.producerQueueSession.createSender(this.producerQueueSession.createQueue(this.queueName));
this.queueReceiver = this.consumerQueueSession.createReceiver(this.consumerQueueSession.createQueue(this.queueName));
this.queueConnection.start();
} catch (JMSException| NamingException exception) {
throw new MyOperationException("Failed to create MyOperationsQueue", exception);
}
}
#Override
protected void finalize() throws Throwable {
this.queueReceiver.close();
this.queueSender.close();
this.consumerQueueSession.close();
this.producerQueueSession.close();
this.queueConnection.close();
super.finalize();
}
#Override
public void submitMyOperation(MyOperationParameters myParameters) throws MyOperationException {
try {
ObjectMessage message = this.producerQueueSession.createObjectMessage(myParameters);
this.queueSender.send(message);
synchronized (this) {
if(this.eventReceiver != null) {
this.eventReceiver.onOperationSubmitted(message.getJMSMessageID(), myParameters);
}
}
} catch (JMSException exc) {
throw new MyOperationException("Failed to submit my operation", exc);
}
}
#Override
public void setMyOperationEventReceiver(MyOperationEventReceiver operationReceiver) throws MyOperationException {
LOGGER.debug("Setting my operation event receiver");
synchronized (this) {
if(this.eventReceiver != null) {
throw new IllegalStateException("Cannot set an operation event receiver if it is already set");
}
this.eventReceiver = operationReceiver;
try {
this.queueReceiver.setMessageListener(message -> {
LOGGER.debug("New message received from queue receiver");
try {
ObjectMessage objectMessage = (ObjectMessage) message;
eventReceiver.onOperationReady(message.getJMSMessageID(), (MyOperationParameters) objectMessage.getObject());
} catch (Exception exception) {
try {
eventReceiver.onOperationRetrievalFailed(message.getJMSMessageID(), exception);
} catch (JMSException innerException) {
LOGGER.error("Failed to get message ID for JMS Message: "+message, innerException);
}
}
});
} catch (JMSException exc) {
throw new MyOperationException("Failed to set My message listener", exc);
}
}
}
}
Please, this is the first time that I write a flink job and I need help. The goal of the job is to calculate the average of different fields of an avro object. The avro schema that I use to parse json messages that come from an ActiveMQ queue is the following:
[
{
"type":"record",
"name":"SensorDataAnnotation",
"namespace":"zzz",
"fields":[
{"name":"meas","type":["null","string"]},
{"name":"prefix","type":["null","string"]}
]
},
{
"namespace":"zzz",
"name":"SensorDataList",
"type":"record",
"fields":[
{"name":"SensorDataListContainer",
"type":{"name":"SensorDataListContainer","type":"array","namespace":"zzz",
"items":{"type":"record","name":"SensorData","namespace":"zzz",
"fields":[
{"name":"prkey","type":"int"},
{"name":"prkeyannotation","type":["null","SensorDataAnnotation"]},
{"name":"value1","type":["null","double"]},
{"name":"value1annotation","type":["null","SensorDataAnnotation"]},
{"name":"value2","type":["null","double"]},
{"name":"value2annotation","type":["null","SensorDataAnnotation"]},
{"name":"value3","type":["null","int"]},
{"name":"value3annotation","type":["null","SensorDataAnnotation"]}
}
]
This is the flink job that I tried to write:
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<SensorData> messages = env.addSource(new StreamSource());
DataStream<Double> counts = messages
.map(new MapFunction<SensorData, Double>() {
#Override
public Double map(SensorData arg0) throws Exception {
return arg0.getValue1();
}
})
.timeWindowAll(Time.seconds(10), Time.seconds(5))
.apply(new Avg());
counts.print();
env.execute("ActiveMQ Streaming Job");
with the StreamSource and Avg classes:
StreamSource
class StreamSource extends RichSourceFunction<SensorData> {
private static final long serialVersionUID = 1L;
private static final Logger LOG = Logger.getLogger(StreamSource.class);
private transient volatile boolean running;
private transient MessageConsumer consumer;
private transient Connection connection;
private void init() throws JMSException {
// Create a ConnectionFactory
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61616");
// Create a Connection
connection = connectionFactory.createConnection();
connection.start();
// Create a Session
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// Create the destination (Topic or Queue)
Destination destination = session.createQueue("input");
// Create a MessageConsumer from the Session to the Topic or Queue
consumer = session.createConsumer(destination);
}
#Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
running = true;
init();
}
#Override
public void run(SourceContext<SensorData> ctx) {
// this source never completes
while (running) {
try {
// Wait for a message
Message message = consumer.receive(1000);
if (message instanceof TextMessage) {
TextMessage textMessage = (TextMessage) message;
String text = textMessage.getText();
try {
byte[] avroDesObj = jsonToAvro(text, SensorData.SCHEMA$.toString());
DatumReader<SensorData> reader = new SpecificDatumReader<SensorData>(SensorData.SCHEMA$);
Decoder decoder = DecoderFactory.get().binaryDecoder(avroDesObj, null);
SensorData data = reader.read(null, decoder);
ctx.collect(data);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} else {
LOG.error("Don't know what to do .. or no message");
}
} catch (JMSException e) {
LOG.error(e.getLocalizedMessage());
running = false;
}
}
try {
close();
} catch (Exception e) {
LOG.error(e.getMessage(), e);
}
}
#Override
public void cancel() {
running = false;
}
#Override
public void close() throws Exception {
LOG.info("Closing");
try {
connection.close();
} catch (JMSException e) {
throw new RuntimeException("Error while closing ActiveMQ connection ", e);
}
}
}
Avg
public static class Avg implements AllWindowFunction<Double,Double, TimeWindow> {
#Override
public void apply(TimeWindow window, Iterable<Double> values, Collector<Double> out) throws Exception {
double sum = 0.0;
int count = 0;
for(Double value : values) {
sum += value.doubleValue();
count++;
}
Double avg = values.iterator().next();
avg = sum / count;
out.collect(avg);
}
}
When I launch the exported jar of this job in the flink dashboard, it does not start and I don't know what I am doing wrong.
Thank You.
I have a publisher for a topic in activeMq that in case of an exception pushes the TextMessage to a vector. I want to spawn a thread when the exception occurs and this thread should keep retrying to push the messages to the same topic. But there should only be one thread and not multiple threads in case of an exception.
private Vector<TextMessage> pendingQueue = new Vector<TextMessage>();
#Override
public void sendMessage(Object myMessage)
{
try
{
factory = new ActiveMQConnectionFactory(activeMQServerURL);
connection = factory.createConnection();
connection.start();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
destination = session.createTopic(cmdTopicName);
producer = session.createProducer(destination);
message = session.createTextMessage();
message.setText(new Gson().toJson(myMessage));
producer.send(message);
}
catch (JMSException e)
{
pendingQueue.add(message);
runBackgroundDLQJobs();
}
}
public void runBackgroundDLQJobs()
{
bgThread.start();
}
And Run method of bgThread:
#Override
public void run()
{
while (pendingQueue.size() > 0)
{
try
{
int idx = 0;
while (idx < pendingQueue.size())
{
boolean sendDLQMessagesSuccess = sendDLQMessages(pendingQueue.get(idx));
if (sendDLQMessagesSuccess)
{
pendingQueue.remove(idx);
idx++;
}
else
{
Thread.sleep(5000);
}
}
}
catch (Exception e)
{
//DoSomething
}
}
}
In the jboss admin-console page I can view the current number of items in my queue.
However I'm getting empty enumeration from queueBrowser.getEnumeration().
Below is my code to browse the queue:
public class JMSQueueBrowser {
private final Log log = LogFactory.getLog(getClass());
private QueueConnectionFactory connectionFactory;
private Queue queue;
private QueueBrowser qBrowser;
private QueueSession qSession;
private QueueConnection qConn;
public JMSQueueBrowser() {
initialize();
}
private void initialize() {
try {
InitialContext initialContext = new InitialContext();
connectionFactory = (QueueConnectionFactory)initialContext.lookup("java:comp/env/jms/MyQCF");
queue = (Queue)initialContext.lookup("queue/sampleQueue");
qConn = (QueueConnection) connectionFactory.createConnection();
qConn.start();
qSession = qConn.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
qBrowser = qSession.createBrowser(queue);
initialContext.close();
} catch (NamingException e) {
log.error(e.getMessage());
} catch (JMSException e) {
log.error(e.getMessage());
}
}
public void browseQueue() {
try {
log.info("---------Queue Name: "+queue.getQueueName()+"-----------");
log.info("---------Queue Has Elements: "+qBrowser.getEnumeration().hasMoreElements()+"-----------");
} catch (JMSException e) {
log.error(e.getMessage());
}
}}
The log is always being the same as following:
INFO JMSQueueBrowser - ---------Queue Name: sampleQueue-----------
INFO JMSQueueBrowser - ---------Queue Has Elements: false----------
The library used for JMS Queue is jbossall-client.jar.
Any answer will be appreciated. Thanks in advance.
To receive Jboss JMS messsage, I have written the JMS Asynchronous Message Receiver
See below code
public class QueueReceive implements MessageListener
{
public final static String JNDI_FACTORY="org.jnp.interfaces.NamingContextFactory";
public final static String JMS_FACTORY="ConnectionFactory";
public final static String QUEUE="/queue/requestQueue";
private QueueConnectionFactory qconFactory;
private QueueConnection qcon;
private QueueSession qsession;
private QueueReceiver qreceiver;
private Queue queue;
private boolean quit = false;
public void onMessage(Message msg)
{
try {
System.out.println("onMessage");
String msgText;
if (msg instanceof TextMessage) {
msgText = ((TextMessage)msg).getText();
} else {
msgText = msg.toString();
}
System.out.println("Message Received: "+ msgText );
if (msgText.equalsIgnoreCase("quit")) {
synchronized(this) {
quit = true;
this.notifyAll(); // Notify main thread to quit
}
}
} catch (JMSException jmse) {
System.err.println("An exception occurred: "+jmse.getMessage());
}
}
public void init(Context ctx, String queueName)
throws NamingException, JMSException
{
qconFactory = (QueueConnectionFactory) ctx.lookup(JMS_FACTORY);
qcon = qconFactory.createQueueConnection();
qsession = qcon.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
queue = (Queue) ctx.lookup(queueName);
qreceiver = qsession.createReceiver(queue);
qreceiver.setMessageListener(this);
qcon.start();
}
public void close()throws JMSException
{
qreceiver.close();
qsession.close();
qcon.close();
}
public static void main(String[] args) throws Exception {
InitialContext ic = getInitialContext("jnp://host:port");
QueueReceive qr = new QueueReceive();
System.out.println("Ready to initialize");
qr.init(ic, QUEUE);
System.out.println("JMS Ready To Receive Messages (To quit, send a \"quit\" message).");
// qr.onMessage(qr.qreceiver.receive());
System.out.println("Message Receivd");
// Wait until a "quit" message has been received.
synchronized(qr) {
while (!qr.quit) {
try {
qr.wait();
} catch (InterruptedException ie) {}
}
}
qr.close();
}
private static InitialContext getInitialContext(String url)
throws NamingException
{
Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY, JNDI_FACTORY);
env.put(Context.PROVIDER_URL, url);
return new InitialContext(env);
}
}
When I run java program, it will not call the onMessage method directly, Please help me how to received the JMS message.
From this code I am able to ping the JMS messenger and unable to read the message?