I have set up a single customer Oracle AQ. I observe messages from this queue in a Java web application with CLIENT_ACKNOWLEDGE mode. But as soon as I receive the messages in the onMessage method, the messages seems to be getting deleted from the Oracle Queue. My assumption is, the message should not get deleted unless I acknowledge them in the client. How do I stop this?
Oracle Queue schema looks like this:
BEGIN DBMS_AQADM.CREATE_QUEUE_TABLE(
Queue_table => '"TESTUSER"."myqueuetable"',
Queue_payload_type => 'TESTUSER.messageobject',
multiple_consumers => false
);
END;
/
BEGIN DBMS_AQADM.CREATE_QUEUE(
Queue_name => 'TESTUSER.myqueue',
Queue_table => 'TESTUSER.myqueuetable',
Queue_type => 0, Max_retries => 5, Retry_delay => 0
);
END;
/
BEGIN dbms_aqadm.start_queue (
queue_name => 'testuser.myqueue'
);
END;
I observer for the messages in my Java application like this
//somewhere in my app
session = queueConnection.createQueueSession(false, Session.CLIENT_ACKNOWLEDGE);
queueReceiver = ((AQjmsSession) databaseConnectionManager.getSession())
.createReceiver(databaseConnectionManager.getQueue(), Messageobject.getORADataFactory());
queueReceiver.setMessageListener(this);
//in my listener class
#Override
public void onMessage(Message message) {
AdtMessage msg = (AdtMessage) message;
try {
Messageobject message = (Messageobject) msg.getAdtPayload();
if (isUserConnected(message.userId)) {
logger.debug("Message acknowledged");
msg.acknowledge();
//handle the message. the message should be deleted now.
} else {
//i don't want the message to be deleted
}
} catch (JMSException | IllegalArgumentException | SQLException e) {
logger.error("An error occurred while sending an outgoing blob", e);
}
}
You are creating a single consumer queue, for which message gets deleted as soon as it is consumed successfully.
To retain the message, set retention_time to a value greater than zero when creating a queue.
BEGIN
DBMS_AQADM.CREATE_QUEUE(
Queue_name => 'TESTUSER.myqueue',
Queue_table => 'TESTUSER.myqueuetable',
Queue_type => 0, Max_retries => 5, Retry_delay => 0,
retention_time => 300 -- retain for next 5 minutes
);
END;
/
Refer to https://docs.oracle.com/database/121/ARPLS/d_aqadm.htm#ARPLS109
Related
[EDIT]: the problem has been solved and solution provided by Artur has been added as an edit at the end.
The idea I am trying to implement is, a TCP server is allowed n connections, if I get a n+1 connection do not allow the connection.
So, I needed to somehow get to cancelling a connection and I went ahead by connecting that particular flow to a Sink.cancelled().
What I have is a IncomingConnection connected to a custom flow that partitions the IncomingConnection based on connection count. Once max connection count is breached the partition logic directs it to an outlet connected to Sink.cancelled.
The expectation was to immediately cancel the connection, but it allows the client to connect and then disconnects after some time.
Maybe I am running into the same issue as noted in the answers for Why does Akka TCP stream server disconnect client when there is no flow for the connection.handlewith? where a flow to handle is not found, and it lingers and disconnects.
I am looking for
A clean solution to not allow incoming connections when the max is breached.
What is Sink.cancelled() doing (if at all it's doing something).
package com.example;
import java.util.concurrent.CompletionStage;
import akka.Done;
import akka.NotUsed;
import akka.actor.typed.ActorSystem;
import akka.actor.typed.javadsl.Behaviors;
import akka.stream.FlowShape;
import akka.stream.SinkShape;
import akka.stream.UniformFanOutShape;
import akka.stream.javadsl.Flow;
import akka.stream.javadsl.GraphDSL;
import akka.stream.javadsl.Partition;
import akka.stream.javadsl.Sink;
import akka.stream.javadsl.Source;
import akka.stream.javadsl.Tcp;
import akka.stream.javadsl.Tcp.IncomingConnection;
import akka.stream.javadsl.Tcp.ServerBinding;
import akka.util.ByteString;
public class SimpleStream03 {
private static int connectionCount = 0;
private static int maxConnectioCount = 2;
public static void runServer() {
ActorSystem actorSystem = ActorSystem.create(Behaviors.empty(), "actorSystem");
Source<IncomingConnection, CompletionStage<ServerBinding>> source = Tcp.get(actorSystem).bind("127.0.0.1",
8888);
Sink<IncomingConnection, CompletionStage<Done>> handler = Sink.foreach(conn -> {
System.out.println("Handler Sink Connection Count " + connectionCount);
System.out.println("Handler Sink Client connected from: " + conn.remoteAddress());
conn.handleWith(Flow.of(ByteString.class), actorSystem);
});
Flow<IncomingConnection, IncomingConnection, NotUsed> connectioncountFlow = Flow
.fromGraph(GraphDSL.create(builder -> {
SinkShape<IncomingConnection> sinkCancelled = builder.add(Sink.cancelled());
FlowShape<IncomingConnection, IncomingConnection> inFlowShape = builder
.add(Flow.of(IncomingConnection.class).map(conn -> {
connectionCount++;
return conn;
}));
UniformFanOutShape<IncomingConnection, IncomingConnection> partition = builder
.add(Partition.create(IncomingConnection.class, 2, param -> {
if (connectionCount > maxConnectioCount) {
connectionCount = maxConnectioCount;
System.out.println("Outlet 0 -> Sink.cancelled");
return 0;
}
System.out.println("Outlet 1 -> forward to handler");
return 1;
}));
builder.from(inFlowShape).toFanOut(partition);
builder.from(partition.out(0)).to(sinkCancelled);
return new FlowShape<>(inFlowShape.in(), partition.out(1));
}));
CompletionStage<ServerBinding> bindingFuture = source.via(connectioncountFlow).to(handler).run(actorSystem);
bindingFuture.handle((binding, throwable) -> {
if (binding != null) {
System.out.println("Server started, listening on: " + binding.localAddress());
} else {
System.err.println("Server could not bind to : " + throwable.getMessage());
actorSystem.terminate();
}
return NotUsed.getInstance();
});
}
public static void main(String[] args) throws InterruptedException {
SimpleStream03.runServer();
}
}
The output confirms the partitioning is working and main sink handler is being reached by the 2 connections.
Server started, listening on: /127.0.0.1:8888
Outlet 1 -> forward to handler
Handler Sink Connection Count 1
Handler Sink Client connected from: /127.0.0.1:60327
Outlet 1 -> forward to handler
Handler Sink Connection Count 2
Handler Sink Client connected from: /127.0.0.1:60330
Outlet 0 -> Sink.cancelled
Edit :
Implementing the accepted answer, the following change prevents the incoming connection after threshold is breached. The client sees a Connection reset by peer
+---------------------------------------------------------+
| |
| Fail Flow |
| +-------------+ |
| +-->+Sink |Source| |
| | |cancel|fail | |
| | +-------------+ |
| +----------+ | |
| | | | |
| +----------+ | O0+--+ |
connections|FLOW | | | O0:count > threshold |
+-------+--> +------->+ Partition| |
| |count++ | | O1+----------------------------->
| +----------+ | | |
| | | O1:count <= threshold|
| +----------+ |
| |
+---------------------------------------------------------+
Replace
SinkShape<IncomingConnection> sinkCancelled = builder.add(Sink.cancelled());
With
Sink<IncomingConnection, CompletionStage<Done>> connectionCancellingSink = Sink.foreach(ic -> ic
.handleWith(Flow.fromSinkAndSource(Sink.cancelled(), Source.failed(new Throwable("killed"))),
actorSystem));// Sink.ignore and Sink.cancel give me the same expected result
SinkShape<IncomingConnection> sinkCancelledShape = builder.add(connectionCancellingSink);
The Sink.cancelled() cancels upstream immediately (https://doc.akka.io/docs/akka/current/stream/operators/Sink/cancelled.html).
However your Partition is created with eagerCancel set to false
/**
* Create a new `Partition` operator with the specified input type, `eagerCancel` is `false`.
*
* #param clazz a type hint for this method
* #param outputCount number of output ports
* #param partitioner function deciding which output each element will be targeted
*/
def create[T](
#unused clazz: Class[T],
outputCount: Int,
partitioner: function.Function[T, Integer]): Graph[UniformFanOutShape[T, T], NotUsed] =
new scaladsl.Partition(outputCount, partitioner.apply, eagerCancel = false)
which means that the Partition will only cancel when ALL of its downstream connections cancel. This is not what you want. But you don't want eagerCancel=true either, because that means that the first connection over the limit will zap the whole Partition and therefore all your connections.. basically trashing the whole server.
Perhaps it's useful to think about the situation here in terms of nested streams. The top level Source<IncomingConnection> represent a stream of accepted TCP connections. You don't want to cancel that stream. If you do, you just killed your server. Every IncomingConnection represents an individual TCP connection. The exchange of bytes happening on such connection is also represented as a stream. It is this stream that you want to cancel for every connection above the threshold.
To do that you can define a connection cancelling Sink like that:
Sink<IncomingConnection, CompletionStage<Done>>
connectionCancellingSink =
Sink.foreach(
ic ->
ic.handleWith(
Flow.fromSinkAndSource(Sink.cancelled(), Source.empty()),
actorSystem));
The IncomingConnection allows you to attach a handler using handleWith method. For that you need a Flow as you both consume bytes from client and potentially send bytes to client too (incoming bytes go into the Flow and whatever you want to send back to the client you need to produce on the output of the Flow). In our case we just want to cancel that stream immediately. You can use Flow.fromSinkAndSource to get a Flow out of... Sink and Source. You can leverage that to plugin Sink.cancelled and Source.empty. So Source.empty means we won't send any bytes to the connection and Sink.cancelled will immediately cancel the stream and hopefully the underlying TCP connection. Let's give it a shot.
Last thing to do is to plugin our new cancelling Sink to the Partition
SinkShape<IncomingConnection> sinkCancelled =
builder.add(connectionCancellingSink);
//...the rest stays the same
builder.from(partition.out(0))
.to(sinkCancelled);
If you do that, on the third connection you will see the following message:
Not aborting connection from 127.0.0.1:49874 because downstream cancelled stream without failure
So Sink.cancelled() is not really triggering what you want. Let's redefine our cancelling Flow:
Sink<IncomingConnection, CompletionStage<Done>>
connectionCancellingSink =
Sink.foreach(
ic ->
ic.handleWith(
Flow.fromSinkAndSource(Sink.ignore(),
Source.failed(new Throwable("killed"))),
actorSystem));
Now, this is using Sink.ignore() to just ignore the incoming bytes but fails the stream via Source.failed(...). This will cause the connection to be terminated immediately and the stracktrace will be printed on the server output. If you want to keep it quiet, you can create exception without stacktrace:
public static class TerminatedException extends Exception
{
public TerminatedException(String message)
{
super(message, null, false, false);
}
}
and then use that to fail your connection stream
Flow.fromSinkAndSource(Sink.ignore(),
Source.failed(
new TerminatedException(("killed"))))
that way you will get cleaner logs.
Set up kafka consumer with this configuration
kafkaconfig:
acks: 1
autoCommit: true
bootstrapServers: example.com:9092
topic: item
groupId: EWok-group
keyDeserializer: org.apache.kafka.common.serialization.StringDeserializer
valueDeserializer: org.apache.kafka.common.serialization.StringDeserializer
maxPollRecords: 1
pollMillisTime: 15
retries: 5
heartBeatInterval: 300
sessionTimeout: 100000
maxPollInterval: 30000
code
while (true) {
try {
ConsumerRecords<String, String> consumerRecords = eWokIntegrationConsumer.poll(Duration.of(kafkaCommConfig.getPollMillisTime(), ChronoUnit.SECONDS));
if (!consumerRecords.isEmpty()) {
LOG.info("Consumed Record Count: {}", consumerRecords.count());
consumerRecords.forEach(record -> {
System.out.printf("offset = %d, key = %s, value = %s\n", record.offset(), record.key(), record.value());
eWokMessageProcessor.onMessage(record.value());
eWokIntegrationConsumer.commitSync();
});
} else {
LOG.info("Polling returned without any records.");
}
} catch (Exception exception) {
LOG.error("Consumer was interrupted. But still continue to poll. Exception:", exception);
eWokIntegrationConsumer.close();
}
}
10000 ms is taking for processing the data which we have received from kafka consumer.but getting
exception saying
java.lang.IllegalStateException: This consumer has already been closed.
Exception Logs
java.lang.IllegalStateException: This consumer has already been closed.
at org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:2202)
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1332)
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1298)
Kafka version : kafka-clients-2.0.1
Could you pls suggest any one how should configurations Kafka consume looks like.
I have put System.exit(0) in other place in the source code.That is why consumer has leave the group and mark as closed.
I have remove System.exit(0) from source code.Now it's working fine.
I'm using the Kafka JDK client ver 0.10.2.1 . I am able to produce simple messages to Kafka for a "heartbeat" test, but I cannot consume a message from that same topic using the sdk. I am able to consume that message when I go into the Kafka CLI, so I have confirmed the message is there. Here's the function I'm using to consume from my Kafka server, with the props - I pass the message I produced to the topic only after I have indeed confirmed the produce() was succesful, I can post that function later if requested:
private def consumeFromKafka(topic: String, expectedMessage: String): Boolean = {
val props: Properties = initProps("consumer")
val consumer = new KafkaConsumer[String, String](props)
consumer.subscribe(List(topic).asJava)
var readExpectedRecord = false
try {
val records = {
val firstPollRecs = consumer.poll(MAX_POLLTIME_MS)
// increase timeout and try again if nothing comes back the first time in case system is busy
if (firstPollRecs.count() == 0) firstPollRecs else {
logger.info("KafkaHeartBeat: First poll had 0 records- trying again - doubling timeout to "
+ (MAX_POLLTIME_MS * 2)/1000 + " sec.")
consumer.poll(MAX_POLLTIME_MS * 2)
}
}
records.forEach(rec => {
if (rec.value() == expectedMessage) readExpectedRecord = true
})
} catch {
case e: Throwable => //log error
} finally {
consumer.close()
}
readExpectedRecord
}
private def initProps(propsType: String): Properties = {
val prop = new Properties()
prop.put("bootstrap.servers", kafkaServer + ":" + kafkaPort)
propsType match {
case "producer" => {
prop.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
prop.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")
prop.put("acks", "1")
prop.put("producer.type", "sync")
prop.put("retries", "3")
prop.put("linger.ms", "5")
}
case "consumer" => {
prop.put("group.id", groupId)
prop.put("enable.auto.commit", "false")
prop.put("auto.commit.interval.ms", "1000")
prop.put("session.timeout.ms", "30000")
prop.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
prop.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
prop.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
// poll just once, should only be one record for the heartbeat
prop.put("max.poll.records", "1")
}
}
prop
}
Now when I run the code, here's what it outputs in the console:
13:04:21 - Discovered coordinator serverName:9092 (id: 2147483647
rack: null) for group 0b8947e1-eb68-4af3-ac7b-be3f7c02e76e. 13:04:23
INFO o.a.k.c.c.i.ConsumerCoordinator - Revoking previously assigned
partitions [] for group 0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:24
INFO o.a.k.c.c.i.AbstractCoordinator - (Re-)joining group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:25 INFO
o.a.k.c.c.i.AbstractCoordinator - Successfully joined group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e with generation 1 13:04:26 INFO
o.a.k.c.c.i.ConsumerCoordinator - Setting newly assigned partitions
[HeartBeat_Topic.Service_5.2018-08-03.13_04_10.377-0] for group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:27 INFO
c.p.p.l.util.KafkaHeartBeatUtil - KafkaHeartBeat: First poll had 0
records- trying again - doubling timeout to 60 sec.
And then nothing else, no errors thrown -so no records are polled. Does anyone have any idea what's preventing the 'consume' from happening? The subscriber seems to be successful, as I'm able to successfully call the listTopics and list partions no problem.
Your code has a bug. It seems your line:
if (firstPollRecs.count() == 0)
Should say this instead
if (firstPollRecs.count() > 0)
Otherwise, you're passing in an empty firstPollRecs, and then iterating over that, which obviously returns nothing.
I have a java application with below properties
kafkaProperties = new Properties();
kafkaProperties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBrokersList);
kafkaProperties.put(ConsumerConfig.GROUP_ID_CONFIG, consumerGroupName);
kafkaProperties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
kafkaProperties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
kafkaProperties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, consumerSessionTimeoutMs);
kafkaProperties.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, maxPartitionFetchBytes);
kafkaProperties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
I've created 15 consumer threads and let them process the below runnable .I don't have any other consumer with this consumer group name consuming .
#Override
public void run() {
try {
logger.info("Starting ConsumerWorker, consumerId={}", consumerId);
consumer.subscribe(Arrays.asList(kafkaTopic), offsetLoggingCallback);
while (true) {
boolean isPollFirstRecord = true;
logger.debug("consumerId={}; about to call consumer.poll() ...", consumerId);
ConsumerRecords<String, String> records = consumer.poll(pollIntervalMs);
Map<Integer,Long> partitionOffsetMap = new HashMap<>();
for (ConsumerRecord<String, String> record : records) {
if (isPollFirstRecord) {
isPollFirstRecord = false;
logger.info("Start offset for partition {} in this poll : {}", record.partition(), record.offset());
}
messageProcessor.processMessage(record.value(), record.offset());
partitionOffsetMap.put(record.partition(),record.offset());
}
if (!records.isEmpty()) {
logger.info("Invoking commit for partition/offset : {}", partitionOffsetMap);
consumer.commitAsync(offsetLoggingCallback);
}
}
} catch (WakeupException e) {
logger.warn("ConsumerWorker [consumerId={}] got WakeupException - exiting ... Exception: {}",
consumerId, e.getMessage());
} catch (Exception e) {
logger.error("ConsumerWorker [consumerId={}] got Exception - exiting ... Exception: {}",
consumerId, e.getMessage());
} finally {
logger.warn("ConsumerWorker [consumerId={}] is shutting down ...", consumerId);
consumer.close();
}
}
I also have a OffsetCommitCallbackImpl like below . It basically maintains the partition's and their commited offset as map .It also logs whenever offset is committed .
#Override
public void onComplete(Map<TopicPartition, OffsetAndMetadata> offsets, Exception exception) {
if (exception == null) {
offsets.forEach((topicPartition, offsetAndMetadata) -> {
partitionOffsetMap.put(topicPartition, offsetAndMetadata);
logger.info("Offset position during the commit for consumerId : {}, partition : {}, offset : {}", Thread.currentThread().getName(), topicPartition.partition(), offsetAndMetadata.offset());
});
} else {
offsets.forEach((topicPartition, offsetAndMetadata) -> logger.error("Offset commit error, and partition offset info : {}, partition : {}, offset : {}", exception.getMessage(), topicPartition.partition(), offsetAndMetadata.offset()));
}
}
Problem/Issue :
I noticed that i miss events/messages whenever i (restart) bring the application down and bring it back up . So when i closely looked at the logging . by comparing the offsets that are committed(using offsetcommitcallback logging) before shutdown vs offsets that are picked up for processing after restart, i see that for certain partition we did not pickup the offset where we left before shutdown. sometimes the start offset for certain partition's are like 1000 more than the committed offset .
NOTE : This happens to like 8 out of 40 partitions
If you closely look at the logging in run method there is one log statement where i actually print the offset before invoking async commit . For example if that last log before shutdown shows that as 10 for partition 1 . After restart the first offset we are processing for partition 1 would be like 100 . And i validated that we are exactly missing 90 messages .
Can any one think of a reason why this would be happening ?
We've performed a performance test with Oracle Advanced Queue on our Oracle DB environment. We've created the queue and the queue table with the following script:
BEGIN
DBMS_AQADM.create_queue_table(
queue_table => 'verisoft.qt_test',
queue_payload_type => 'SYS.AQ$_JMS_MESSAGE',
sort_list => 'ENQ_TIME',
multiple_consumers => false,
message_grouping => 0,
comment => 'POC Authorizations Queue Table - KK',
compatible => '10.0',
secure => true);
DBMS_AQADM.create_queue(
queue_name => 'verisoft.q_test',
queue_table => 'verisoft.qt_test',
queue_type => dbms_aqadm.NORMAL_QUEUE,
max_retries => 10,
retry_delay => 0,
retention_time => 0,
comment => 'POC Authorizations Queue - KK');
DBMS_AQADM.start_queue('q_test');
END;
/
We've published 1000000 messages with 2380 TPS using a PL/SQL client. And we've consumed 1000000 messages with 292 TPS, using Oracle JMS API Client.
The consumer rate is almost 10 times slower than the publisher and that speed does not meet our requirements.
Below, is the piece of Java code that we use to consume messages:
if (q == null) initializeQueue();
System.out.println(listenerID + ": Listening on queue " + q.getQueueName() + "...");
MessageConsumer consumer = sess.createConsumer(q);
for (Message m; (m = consumer.receive()) != null;) {
new Timer().schedule(new QueueExample(m), 0);
}
sess.close();
con.close();
Do you have any suggestion on, how we can improve the performance at the consumer side?
Your use of Timer may be your primary issue. The Timer definition reads:
Corresponding to each Timer object is a single background thread that is used to execute all of the timer's tasks, sequentially. Timer tasks should complete quickly. If a timer task takes excessive time to complete, it "hogs" the timer's task execution thread. This can, in turn, delay the execution of subsequent tasks, which may "bunch up" and execute in rapid succession when (and if) the offending task finally completes.
I would suggest you use a ThreadPool.
// My executor.
ExecutorService executor = Executors.newCachedThreadPool();
public void test() throws InterruptedException {
for (int i = 0; i < 1000; i++) {
final int n = i;
// Instead of using Timer, create a Runnable and pass it to the Executor.
executor.submit(new Runnable() {
#Override
public void run() {
System.out.println("Run " + n);
}
});
}
executor.shutdown();
executor.awaitTermination(1, TimeUnit.DAYS);
}