I'm trying to use resizer in akka routing with round-robin-pool. But it is not creating the instances. It is working on the instances which I mentioned in the lower-bound. I'm following the documents of akka version 2.5.3.
My configuration :
akka.actor.deployment {
/round-robin-resizer {
router = round-robin-pool
resizer {
lower-bound = 4
upper-bound = 30
pressure-threshold = 0
rampup-rate = 0.5
messages-per-resize = 1
}
}
Actor class :
return receiveBuilder()
.match(Integer.class, msg -> {
System.out.println("Message : " + msg + " Thread id : " + Thread.currentThread().getId());
Thread.sleep(5000);
})
.matchAny(msg -> {
System.out.println("Error Message : " + msg + " Thread id : " + Thread.currentThread().getId());
}).build();
}
Creation of actor :
ActorRef roundRobin = system.actorOf(FromConfig.getInstance().props(Props.create(RoutingActor.class)), "round-robin-resizer");
for (int i = 0; i < 15; i++) {
roundRobin.tell(i, ActorRef.noSender());
}
Output :
Message : 2 Thread id : 18
Message : 1 Thread id : 16
Message : 0 Thread id : 15
Message : 3 Thread id : 17
Message : 7 Thread id : 17
Message : 4 Thread id : 15
Message : 6 Thread id : 18
Message : 5 Thread id : 16
Message : 11 Thread id : 17
Message : 9 Thread id : 16
Message : 10 Thread id : 18
Message : 8 Thread id : 15
Message : 13 Thread id : 16
Message : 14 Thread id : 18
Message : 12 Thread id : 15
After every 4 result it is waiting for 5 seconds to complete the job of the previous instances.
See the thread IDs. For every creation of actor instance I'm letting my thread to sleep some time. At the time the new instance should be allocated on different thread. But this process in running till the first three instance. After that it is not creating the new instance as per the resizer. It is appending the message as per the normal flow of round robin pool.
You are getting confused with thread-id and actual actor instance. The number of actors instances does not match with the number of threads. Please refer to this answer in other similar question: Akka ConsistentHashingRoutingLogic not routing to the same dispatcher thread consistently
Related
I have a use case where i am writing to a Kafka topic in batches using spark job (no streaming).Initially i pump-in suppose 10 records to Kafka topic and run the spark job which does some processing and finally write to another Kafka topic.
Next time when i push another 5 records and run the spark job, my requirement is to start processing these 5 records only not from starting offset. I need to maintain the committed offset so that spark job should run on next offset position and do the processing.
Here is code from kafka side to fetch the offset:
private static List<TopicPartition> getPartitions(KafkaConsumer consumer, String topic) {
List<PartitionInfo> partitionInfoList = consumer.partitionsFor(topic);
return partitionInfoList.stream().map(x -> new TopicPartition(topic, x.partition())).collect(Collectors.toList());
}
public static void getOffSet(KafkaConsumer consumer) {
List<TopicPartition> topicPartitions = getPartitions(consumer, topic);
consumer.assign(topicPartitions);
consumer.seekToBeginning(topicPartitions);
topicPartitions.forEach(x -> {
System.out.println("Partition-> " + x + " startingOffSet-> " + consumer.position(x));
});
consumer.assign(topicPartitions);
consumer.seekToEnd(topicPartitions);
topicPartitions.forEach(x -> {
System.out.println("Partition-> " + x + " endingOffSet-> " + consumer.position(x));
});
topicPartitions.forEach(x -> {
consumer.poll(1000) ;
OffsetAndMetadata offsetAndMetadata = consumer.committed(x);
long position = consumer.position(x);
System.out.printf("Committed: %s, current position %s%n", offsetAndMetadata == null ? null : offsetAndMetadata
.offset(), position);
});
}
Below code is for spark to load the messages from topic which is not working :
Dataset<Row> kafkaDataset = session.read().format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", topic)
.option("group.id", "test-consumer-group")
.option("startingOffsets","{\"Topic1\":{\"0\":2}}")
.option("endingOffsets", "{\"Topic1\":{\"0\":3}}")
.option("enable.auto.commit","true")
.load();
After above code executes i am again trying to get the offset by calling
getoffset(consumer)
from the topic which always reads from 0 offset and committed offset fetched initially keeps on increasing. I am new to kafka and still figuring out how to handle such scenarion.Please help here.
Initially i had 10 records in my topic, i published another 2 records and here is the o/p:
Output post getoffset method executes :
Partition-> Topic00-0 startingOffSet-> 0 Partition->
Topic00-0 endingOffSet-> 12 Committed: 12, current position
12
Output post spark code executes for loading messages.
Partition-> Topic00-0 startingOffSet-> 0 Partition->
Topic00-0 endingOffSet-> 12 Committed: 12, current position
12
I see no diff and . Please take a look and suggest resolution for this sceanario.
As per documentation, max.poll.interval.ms is the time before which processor thread must complete processing. Otherwise, rebalancing will trigger.
Time interval mentioned in consumer.poll() has a separate purpose i.e. waiting for data before time-out is triggered.
Now, I am having a weird problem. In my code, consumer.poll() is set with value of 5000ms. And max.poll.interval.ms has a value of 20000. As per above details, my processor thread should wait for 5 seconds during polling, but it is waiting for 20 seconds. Similary, when max.poll.interval.ms value is not set explicitly, I am getting delays of 5 minutes i.e. code gets blocked during poll for 5 minutes (verified from debugger and logging).
Code:
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(kafkaProps);
consumer.subscribe(Arrays.asList(topic));
LGR.info(consumer.partitionsFor(topic));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Integer.parseInt(kafkaProps.getProperty("consume.poll.timeout")));
if(records.count() == 0) {
LGR.info("No records found for kafka topic: " + kafkaProps.getProperty("topic") + ", and bootstrap servers: " + kafkaProps.getProperty("bootstrap.servers"));
break;
}
for (ConsumerRecord<String, String> record : records) {
LGR.info("Topic: " + record.topic() + ", Timestamp: " + record.timestamp() + ", Partition: " + record.partition() + ", Offset: " + record.offset() + ", Key: " + record.key() + ", Value: " + record.value());
}
}
Configurations:
topic=****
bootstrap.servers=****
group.id=orderstatus_61
enable.auto.commit=false
auto.commit.interval.ms=10000
auto.offset.reset=earliest
session.timeout.ms=30000
client.id=orderstatus_61
max.poll.interval.ms=20000
max.pool.records=500
consume.poll.timeout=5000
ssl=no
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
Kafka Version: kafka-clients-1.0.0
Set up kafka consumer with this configuration
kafkaconfig:
acks: 1
autoCommit: true
bootstrapServers: example.com:9092
topic: item
groupId: EWok-group
keyDeserializer: org.apache.kafka.common.serialization.StringDeserializer
valueDeserializer: org.apache.kafka.common.serialization.StringDeserializer
maxPollRecords: 1
pollMillisTime: 15
retries: 5
heartBeatInterval: 300
sessionTimeout: 100000
maxPollInterval: 30000
code
while (true) {
try {
ConsumerRecords<String, String> consumerRecords = eWokIntegrationConsumer.poll(Duration.of(kafkaCommConfig.getPollMillisTime(), ChronoUnit.SECONDS));
if (!consumerRecords.isEmpty()) {
LOG.info("Consumed Record Count: {}", consumerRecords.count());
consumerRecords.forEach(record -> {
System.out.printf("offset = %d, key = %s, value = %s\n", record.offset(), record.key(), record.value());
eWokMessageProcessor.onMessage(record.value());
eWokIntegrationConsumer.commitSync();
});
} else {
LOG.info("Polling returned without any records.");
}
} catch (Exception exception) {
LOG.error("Consumer was interrupted. But still continue to poll. Exception:", exception);
eWokIntegrationConsumer.close();
}
}
10000 ms is taking for processing the data which we have received from kafka consumer.but getting
exception saying
java.lang.IllegalStateException: This consumer has already been closed.
Exception Logs
java.lang.IllegalStateException: This consumer has already been closed.
at org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:2202)
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1332)
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1298)
Kafka version : kafka-clients-2.0.1
Could you pls suggest any one how should configurations Kafka consume looks like.
I have put System.exit(0) in other place in the source code.That is why consumer has leave the group and mark as closed.
I have remove System.exit(0) from source code.Now it's working fine.
Recently I wrote a demo program to launch reactive programming with a combination of Reactor and RabbitMQ. This is my demo code :
public class FluxWithRabbitMQDemo {
private static final String QUEUE = "demo_thong";
private final reactor.rabbitmq.Sender sender;
private final Receiver receiver;
public FluxWithRabbitMQDemo() {
this.sender = ReactorRabbitMq.createSender();
this.receiver = ReactorRabbitMq.createReceiver();
}
public void run(int count) {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.useNio();
SenderOptions senderOptions = new SenderOptions()
.connectionFactory(connectionFactory)
.resourceCreationScheduler(Schedulers.elastic());
reactor.rabbitmq.Sender sender = ReactorRabbitMq.createSender(senderOptions);
Mono<AMQP.Queue.DeclareOk> queueDeclaration = sender.declareQueue(QueueSpecification.queue(QUEUE));
Flux<Delivery> messages = receiver.consumeAutoAck(QUEUE);
queueDeclaration.thenMany(messages).subscribe(m->System.out.println("Get message "+ new String(m.getBody())));
Flux<OutboundMessageResult> dataStream = sender.sendWithPublishConfirms(Flux.range(1, count)
.filter(m -> !m.equals(10))
.parallel()
.runOn(Schedulers.parallel())
.doOnNext(i->System.out.println("Message " + i + " run on thread "+Thread.currentThread().getId()))
.map(i -> new OutboundMessage("", QUEUE, ("Message " + i).getBytes())));
sender.declareQueue(QueueSpecification.queue(QUEUE))
.thenMany(dataStream)
.doOnError(e -> System.out.println("Send failed"+ e))
.subscribe(m->{
if (m!= null){
System.out.println("Sent successfully message "+new String(m.getOutboundMessage().getBody()));
}
});
try {
Thread.sleep(20000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public static void main(String[] args) throws Exception {
int count = 20;
FluxWithRabbitMQDemo sender = new FluxWithRabbitMQDemo();
sender.run(count);
}
}
I expected that after Flux emit an item, Sender must send it to RabbitMQ and after receiving RabbitMQ the Receiver must receive it.
But everything happened sequentially and this is the result I got
Message 3 run on thread 25
Message 4 run on thread 26
Message 8 run on thread 26
Message 13 run on thread 26
Message 17 run on thread 26
Message 2 run on thread 24
Message 1 run on thread 23
Message 6 run on thread 24
Message 5 run on thread 23
Message 9 run on thread 23
Message 14 run on thread 23
Message 18 run on thread 23
Message 11 run on thread 24
Message 15 run on thread 24
Message 19 run on thread 24
Message 7 run on thread 25
Message 12 run on thread 25
Message 16 run on thread 25
Message 20 run on thread 25
Sent successfully message Message 3
Sent successfully message Message 1
Sent successfully message Message 2
Sent successfully message Message 4
Sent successfully message Message 5
Sent successfully message Message 6
Sent successfully message Message 8
Sent successfully message Message 9
Sent successfully message Message 11
Sent successfully message Message 13
Sent successfully message Message 14
Sent successfully message Message 15
Sent successfully message Message 17
Sent successfully message Message 18
Sent successfully message Message 19
Sent successfully message Message 7
Sent successfully message Message 12
Sent successfully message Message 16
Sent successfully message Message 20
Get message Message 3
Get message Message 1
Get message Message 2
Get message Message 4
Get message Message 5
Get message Message 6
Get message Message 8
Get message Message 9
Get message Message 11
Get message Message 13
Get message Message 14
Get message Message 15
Get message Message 17
Get message Message 18
Get message Message 19
Get message Message 7
Get message Message 12
Get message Message 16
Get message Message 20
I do not know what to do with my code to achieve the results as expected. Can someone help me? Thanks for advance!!!
Messages are generated too fast. To see the interleaving, in dataStream add
.doOnNext(i->Thread.sleep(10))
Why does my app throw a
java.lang.ArrayIndexOutOfBoundsException: -1
when I invoke future.get() on java.utils.concurrent.Future??
List<Future> tableLoadings = new LinkedList<>();
ExecutorService executor = Executors.newFixedThreadPool(8);
try{
for(Entry<Integer, String> entry: farmIds.entrySet())
{
int id = entry.getKey();
String username = entry.getValue();
psLog.println("START ELABORAZIONE FARMACIA ID : "+ id+" TPH_USERNAME : "+username );
/*SdajSdaj*/
tableLoadings.add(executor.submit(new StatusMultiThreading(id, username, psLog, connSTORY, connCF, mongoDatabase)));
}
for (Future<Void> future : tableLoadings) {
try{
future.get();
}catch(Exception e){
psLog.println("[EE] ERORE ELABORAZIONE THREAD FARMACIA [EE] "+e.getMessage());
}
}
}finally{
executor.shutdown();
psLog.println("END CONSOLIDA STATUS FARMACIE");
}
this is the log..
START ELABORAZIONE FARMACIA ID : 62 TPH_USERNAME : A0102987
START ELABORAZIONE FARMACIA ID : 63 TPH_USERNAME : A0103019
START SENDING DATA TO DB FARMID = 66
...
START SENDING DATA TO DB FARMID = 17
[EE] ERORE ELABORAZIONE THREAD FARMACIA [EE] java.lang.ArrayIndexOutOfBoundsException: -1
[EE] ERRORE ELABORAZIONE THREAD FARMACIA [EE] java.lang.ArrayIndexOutOfBoundsException: -1
END CONSOLIDA STATUS FARMACIE
I can't find anything wrong if I debug.
I cannot go inside .get() method, so I don't understand which line of code is invalid.
What can be said so far: you use the ExecutorService to pass in tasks:
new StatusMultiThreading(id, username, psLog, connSTORY, connCF, mongoDatabase)
Later on, when you call get() the corresponding task is triggered. So that exception takes place inside that class of yours.