Develop Reference

How to track committed offset with Spark job for kafka batch

I have a use case where i am writing to a Kafka topic in batches using spark job (no streaming).Initially i pump-in suppose 10 records to Kafka topic and run the spark job which does some processing and finally write to another Kafka topic.
Next time when i push another 5 records and run the spark job, my requirement is to start processing these 5 records only not from starting offset. I need to maintain the committed offset so that spark job should run on next offset position and do the processing.
Here is code from kafka side to fetch the offset:
private static List<TopicPartition> getPartitions(KafkaConsumer consumer, String topic) {
List<PartitionInfo> partitionInfoList = consumer.partitionsFor(topic);
return partitionInfoList.stream().map(x -> new TopicPartition(topic, x.partition())).collect(Collectors.toList());
}
public static void getOffSet(KafkaConsumer consumer) {
List<TopicPartition> topicPartitions = getPartitions(consumer, topic);
consumer.assign(topicPartitions);
consumer.seekToBeginning(topicPartitions);
topicPartitions.forEach(x -> {
System.out.println("Partition-> " + x + " startingOffSet-> " + consumer.position(x));
});
consumer.assign(topicPartitions);
consumer.seekToEnd(topicPartitions);
topicPartitions.forEach(x -> {
System.out.println("Partition-> " + x + " endingOffSet-> " + consumer.position(x));
});
topicPartitions.forEach(x -> {
consumer.poll(1000) ;
OffsetAndMetadata offsetAndMetadata = consumer.committed(x);
long position = consumer.position(x);
System.out.printf("Committed: %s, current position %s%n", offsetAndMetadata == null ? null : offsetAndMetadata
.offset(), position);
});
}
Below code is for spark to load the messages from topic which is not working :
Dataset<Row> kafkaDataset = session.read().format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", topic)
.option("group.id", "test-consumer-group")
.option("startingOffsets","{\"Topic1\":{\"0\":2}}")
.option("endingOffsets", "{\"Topic1\":{\"0\":3}}")
.option("enable.auto.commit","true")
.load();
After above code executes i am again trying to get the offset by calling
getoffset(consumer)
from the topic which always reads from 0 offset and committed offset fetched initially keeps on increasing. I am new to kafka and still figuring out how to handle such scenarion.Please help here.
Initially i had 10 records in my topic, i published another 2 records and here is the o/p:
Output post getoffset method executes :
Partition-> Topic00-0 startingOffSet-> 0 Partition->
Topic00-0 endingOffSet-> 12 Committed: 12, current position
12
Output post spark code executes for loading messages.
Partition-> Topic00-0 startingOffSet-> 0 Partition->
Topic00-0 endingOffSet-> 12 Committed: 12, current position
12
I see no diff and . Please take a look and suggest resolution for this sceanario.

How to correctly handle Hikari connection pool in SpringBoot and oracle with java?

I have a service that does 10 parallel tasks and when all of them finish, it returns a response.
In my opinion, I can implement it in 2 ways—the first one is creating in a usual way, and the second one is using the multi-thread concept.
I coded and compared both. But I had been having a problem with the connection pool in multithreaded.
When I send specific requests at a particular time (e.g. 1000 requests per 0.001 seconds) with JMeter in the usual implementation, all requests are done; but while running it with the multithread implement, some of the requests are failed.
The cause of this problem is related to the connection pool; Service is gotten the Hikari pool error.
I know one solution as increment the Hikari pool, but I can't do it, because the speed of the oracle database will decrease (I don't have access to it).
What is the best practice for handling multi-threads in this situation? increasing Hikari pool or dividing threads into smaller parts?
( Technologies used:Springboot v2.6.0, Java 14, Oracle database, for multithread used completeableFuture from java.util.concurrent, JMeter (last version) )
The code shows you, all my configurations in the application.properties :
#maximum number of milliseconds that a client will wait for a connection
spring.datasource.hikari.connection-timeout = 20000
#minimum number of idle connections maintained by HikariCP in a connection pool
spring.datasource.hikari.minimum-idle= 10000
#maximum pool size
spring.datasource.hikari.maximum-pool-size= 10000
#maximum idle time for connection
spring.datasource.hikari.idle-timeout=10000
# maximum lifetime in milliseconds of a connection in the pool after it is closed.
spring.datasource.hikari.max-lifetime= 1000
Implementation of multithreaded :
#Override
public AllInquiriesDTO getAllInquiries(Long cardRequestId, String outputPostCode, String outputTraceId) {
AllInquiriesDTO allInquiriesDTO = new AllInquiriesDTO();
CompletableFuture<Void> gomrokHoghooghiInquiry = this.getGomrokHoghooghiInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setGomrokHoghoghi);
CompletableFuture<Void> returnCheckInquiry = this.getReturnCheckInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setReturnCheck);
CompletableFuture<Void> behinyabInquiry = this.getBehinyabInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setBehinyab);
CompletableFuture<Void> graduationInquiry = this.getGraduationInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setGraduation);
CompletableFuture<Void> legalPersonalInfoInquiry = this.getLegalPersonalInfoInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setLegalPersonalInfo);
CompletableFuture<Void> tax186TaeedInquiry = this.getTax186TaeedInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setTax186Taeed);
CompletableFuture<Void> tax186SodoorInquiry = this.getTax186SodoorInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setTax186Sodoor);
CompletableFuture<Void> taxAnnualDetailListInquiry = this.getTaxAnnualDetailListInquiry(cardRequestId)
.thenAccept(res -> allInquiriesDTO.getTax186Sodoor().get(0).setTaxAnnualDetailList(res));
CompletableFuture<Void> postInquiry = this.getPostInquiry(cardRequestId).
thenAccept(allInquiriesDTO::setPost);
CompletableFuture<Void> gomrokHaghighiInquiry = this.getGomrokHaghighiInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setGomrokHaghighi);
CompletableFuture<Void> nezamVazifeInquiry = this.getNezamVazifeInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setNezamVazife);
CompletableFuture<Void> sahaInquiry = this.getSahaInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setSaha);
CompletableFuture<Void> amlakInquiry = this.getAmlakInquiry(cardRequestId, outputPostCode, outputTraceId)
.thenAccept(allInquiriesDTO::setAmlak);
CompletableFuture<Void> jahadInquiry = this.getJahadInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setJahad);
CompletableFuture<Void> employeeCheckInquiry = this.getEmployeeCheckInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setEmployeeCheck);
CompletableFuture<Void> taeminDebtInquiry = this.getTaeminDebtInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setTaminDebt);
CompletableFuture<Void> madanInquiry = this.getMadanInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setMadan);
CompletableFuture<Void> cCompanyInquiry = this.getCCompanyInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setCCompany);
CompletableFuture<Void> soePishineInquiry = this.getSoePishineInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setSoePishine);
CompletableFuture<Void> managerInfoInquiry = this.getManagerInfoInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setManagerInfo);
CompletableFuture<Void> tanzimMoghararatInquiry = this.getTanzimMoghararatInquiry(cardRequestId)
.thenAccept(allInquiriesDTO::setTanzimMoghararat);
CompletableFuture.allOf(gomrokHoghooghiInquiry, returnCheckInquiry, behinyabInquiry, graduationInquiry,
legalPersonalInfoInquiry, tax186TaeedInquiry,
tax186SodoorInquiry, taxAnnualDetailListInquiry, postInquiry, gomrokHaghighiInquiry,
nezamVazifeInquiry, sahaInquiry, amlakInquiry, jahadInquiry, employeeCheckInquiry,
taeminDebtInquiry, madanInquiry, cCompanyInquiry, soePishineInquiry, managerInfoInquiry,
tanzimMoghararatInquiry).join();
return allInquiriesDTO;
}
#Override
public CompletableFuture<List<ReturnCheckDTO>> getReturnCheckInquiry(Long cardRequestId) {
return CompletableFuture.supplyAsync(()
-> returnCheckRepository.findByCardRequestId(cardRequestId));
}
#Override
public CompletableFuture<List<GraduationDTO>> getGraduationInquiry(Long cardRequestId) {
return CompletableFuture.supplyAsync(()
-> graduationRepository.findByCardRequestId(cardRequestId));
}
#Override
public CompletableFuture<List<LegalPersonalInfoDTO>> getLegalPersonalInfoInquiry(Long cardRequestId) {
return CompletableFuture.supplyAsync(()
-> legalPersonalInfoMIMTWSRepository.findByCardRequestId(cardRequestId));
}
#Override
public CompletableFuture<List<Tax186WebServiceTaeed>> getTax186TaeedInquiry(Long cardRequestId) {
return CompletableFuture.supplyAsync(()
-> tax186DebtRepository.findByCardRequestId(cardRequestId));
}

Java : how to send x emails per minute

I need to send a customized email to 400 clients.
I am doing this :
for (Client c : clients){
setUpEmail(c);
sendMail(c);
}
My problem is that my email provider authorizes me to send only 10 emails per minute. How could I do that in the loop?
Thanks.

Use Guava's RateLimiter.
If you already have Guava in your library path, or if you're interested in adding it, you can use this solution:
RateLimiter rateLimiter = RateLimiter.create(10/60d); // 10 permits per 60 seconds.
for (Client c : clients){
setUpEmail(c);
rateLimiter.acquire(1);
sendMail(c);
}
Your kind of problem is exactly why RateLimiter was created.

Use a counter and wait for a minute when ten mails were sent:
int counter = 0;
for (Client c : clients){
counter++;
setUpEmail(c);
sendMail(c);
if(counter%10==0){
Thread.sleep(60*1000); // wait a minute
}
}
This is not ideal since you may lost some time, e.g. when sending ten mails needs 20 seconds, you only may wait 40 seconds before starting a new bulk.
Another option would be to wait between each mail so that the time for 10 mails is at least 60 seconds:
for (Client c : clients){
setUpEmail(c);
sendMail(c);
Thread.sleep(6*1000); // wait 6 seconds
}
And a more sophisticated one:
int counter = 0;
long start = System.currentTimeMillis();
for (Client c : clients){
counter++;
setUpEmail(c);
sendMail(c);
if(counter%10==0){
long needed = System.currentTimeMillis() - start; // ms needed for ten mails
Thread.sleep(60*1000 - needed); // wait rest of the minute
start = System.currentTimeMillis(); // start of the next bulk
}
}

Deque clientsDeque = new ArrayDeque(clients);
ScheduledExecutorService executor = Executors.newScheduledThreadPool(1);
Runnable task = () => {
for (int i=0; i<10; i++){
Client c = clientsDeque.poll();
setUpEmail(c);
sendMail(c);
}
}
executor.schedule(task, 60, TimeUnit.SECONDS);

How to stop getting messages deleted without acknowledgement in Oracle AQ?

I have set up a single customer Oracle AQ. I observe messages from this queue in a Java web application with CLIENT_ACKNOWLEDGE mode. But as soon as I receive the messages in the onMessage method, the messages seems to be getting deleted from the Oracle Queue. My assumption is, the message should not get deleted unless I acknowledge them in the client. How do I stop this?
Oracle Queue schema looks like this:
BEGIN DBMS_AQADM.CREATE_QUEUE_TABLE(
Queue_table => '"TESTUSER"."myqueuetable"',
Queue_payload_type => 'TESTUSER.messageobject',
multiple_consumers => false
);
END;
/
BEGIN DBMS_AQADM.CREATE_QUEUE(
Queue_name => 'TESTUSER.myqueue',
Queue_table => 'TESTUSER.myqueuetable',
Queue_type => 0, Max_retries => 5, Retry_delay => 0
);
END;
/
BEGIN dbms_aqadm.start_queue (
queue_name => 'testuser.myqueue'
);
END;
I observer for the messages in my Java application like this
//somewhere in my app
session = queueConnection.createQueueSession(false, Session.CLIENT_ACKNOWLEDGE);
queueReceiver = ((AQjmsSession) databaseConnectionManager.getSession())
.createReceiver(databaseConnectionManager.getQueue(), Messageobject.getORADataFactory());
queueReceiver.setMessageListener(this);
//in my listener class
#Override
public void onMessage(Message message) {
AdtMessage msg = (AdtMessage) message;
try {
Messageobject message = (Messageobject) msg.getAdtPayload();
if (isUserConnected(message.userId)) {
logger.debug("Message acknowledged");
msg.acknowledge();
//handle the message. the message should be deleted now.
} else {
//i don't want the message to be deleted
}
} catch (JMSException | IllegalArgumentException | SQLException e) {
logger.error("An error occurred while sending an outgoing blob", e);
}
}

You are creating a single consumer queue, for which message gets deleted as soon as it is consumed successfully.
To retain the message, set retention_time to a value greater than zero when creating a queue.
BEGIN
DBMS_AQADM.CREATE_QUEUE(
Queue_name => 'TESTUSER.myqueue',
Queue_table => 'TESTUSER.myqueuetable',
Queue_type => 0, Max_retries => 5, Retry_delay => 0,
retention_time => 300 -- retain for next 5 minutes
);
END;
/
Refer to https://docs.oracle.com/database/121/ARPLS/d_aqadm.htm#ARPLS109

Does the call of join in the following CompletableFuture example block the process

I am trying to understand CompletableFutures and chaining of calls that that return completed futures and I have created the bellow example which kind of simulates two calls to a database.
The first method is supposed to be giving a completable future with a list of userIds and then I need to make a call to another method providing a userId to get the user (a string in this case).
to summarise:
1. fetch the ids
2. fetch a list of the users coresponding to those ids.
I created simple methods to simulate the responses with sleap threads.
Please check the code bellow
public class PipelineOfTasksExample {
private Map<Long, String> db = new HashMap<>();
PipelineOfTasksExample() {
db.put(1L, "user1");
db.put(2L, "user2");
db.put(3L, "user3");
db.put(4L, "user4");
}
private CompletableFuture<List<Long>> returnUserIdsFromDb() {
try {
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("building the list of Ids" + " - thread: " + Thread.currentThread().getName());
return CompletableFuture.supplyAsync(() -> Arrays.asList(1L, 2L, 3L, 4L));
}
private CompletableFuture<String> fetchById(Long id) {
CompletableFuture<String> cfId = CompletableFuture.supplyAsync(() -> db.get(id));
try {
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("fetching id: " + id + " -> " + db.get(id) + " thread: " + Thread.currentThread().getName());
return cfId;
}
public static void main(String[] args) {
PipelineOfTasksExample example = new PipelineOfTasksExample();
CompletableFuture<List<String>> result = example.returnUserIdsFromDb()
.thenCompose(listOfIds ->
CompletableFuture.supplyAsync(
() -> listOfIds.parallelStream()
.map(id -> example.fetchById(id).join())
.collect(Collectors.toList()
)
)
);
System.out.println(result.join());
}
}
My question is, does the join call (example.fetchById(id).join()) ruins the nonblocking nature of process. If the answer is positive how can I solve this problem?
Thank you in advance

Your example is a bit odd as you are slowing down the main thread in returnUserIdsFromDb(), before any operation even starts and likewise, fetchById slows down the caller rather than the asynchronous operation, which defeats the entire purpose of asynchronous operations.
Further, instead of .thenCompose(listOfIds -> CompletableFuture.supplyAsync(() -> …)) you can simply use .thenApplyAsync(listOfIds -> …).
So a better example might be
public class PipelineOfTasksExample {
private final Map<Long, String> db = LongStream.rangeClosed(1, 4).boxed()
.collect(Collectors.toMap(id -> id, id -> "user"+id));
PipelineOfTasksExample() {}
private static <T> T slowDown(String op, T result) {
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
System.out.println(op + " -> " + result + " thread: "
+ Thread.currentThread().getName()+ ", "
+ POOL.getPoolSize() + " threads");
return result;
}
private CompletableFuture<List<Long>> returnUserIdsFromDb() {
System.out.println("trigger building the list of Ids - thread: "
+ Thread.currentThread().getName());
return CompletableFuture.supplyAsync(
() -> slowDown("building the list of Ids", Arrays.asList(1L, 2L, 3L, 4L)),
POOL);
}
private CompletableFuture<String> fetchById(Long id) {
System.out.println("trigger fetching id: " + id + " thread: "
+ Thread.currentThread().getName());
return CompletableFuture.supplyAsync(
() -> slowDown("fetching id: " + id , db.get(id)), POOL);
}
static ForkJoinPool POOL = new ForkJoinPool(2);
public static void main(String[] args) {
PipelineOfTasksExample example = new PipelineOfTasksExample();
CompletableFuture<List<String>> result = example.returnUserIdsFromDb()
.thenApplyAsync(listOfIds ->
listOfIds.parallelStream()
.map(id -> example.fetchById(id).join())
.collect(Collectors.toList()
),
POOL
);
System.out.println(result.join());
}
}
which prints something like
trigger building the list of Ids - thread: main
building the list of Ids -> [1, 2, 3, 4] thread: ForkJoinPool-1-worker-1, 1 threads
trigger fetching id: 2 thread: ForkJoinPool-1-worker-0
trigger fetching id: 3 thread: ForkJoinPool-1-worker-1
trigger fetching id: 4 thread: ForkJoinPool-1-worker-2
fetching id: 4 -> user4 thread: ForkJoinPool-1-worker-3, 4 threads
fetching id: 2 -> user2 thread: ForkJoinPool-1-worker-3, 4 threads
fetching id: 3 -> user3 thread: ForkJoinPool-1-worker-2, 4 threads
trigger fetching id: 1 thread: ForkJoinPool-1-worker-3
fetching id: 1 -> user1 thread: ForkJoinPool-1-worker-2, 4 threads
[user1, user2, user3, user4]
which might be a surprising number of threads on the first glance.
The answer is that join() may block the thread, but if this happens inside a worker thread of a Fork/Join pool, this situation will be detected and a new compensation thread will be started, to ensure the configured target parallelism.
As a special case, when the default Fork/Join pool is used, the implementation may pick up new pending tasks within the join() method, to ensure progress within the same thread.
So the code will always make progress and there’s nothing wrong with calling join() occasionally, if the alternatives are much more complicated, but there’s some danger of too much resource consumption, if used excessively. After all, the reason to use thread pools, is to limit the number of threads.
The alternative is to use chained dependent operations where possible.
public class PipelineOfTasksExample {
private final Map<Long, String> db = LongStream.rangeClosed(1, 4).boxed()
.collect(Collectors.toMap(id -> id, id -> "user"+id));
PipelineOfTasksExample() {}
private static <T> T slowDown(String op, T result) {
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
System.out.println(op + " -> " + result + " thread: "
+ Thread.currentThread().getName()+ ", "
+ POOL.getPoolSize() + " threads");
return result;
}
private CompletableFuture<List<Long>> returnUserIdsFromDb() {
System.out.println("trigger building the list of Ids - thread: "
+ Thread.currentThread().getName());
return CompletableFuture.supplyAsync(
() -> slowDown("building the list of Ids", Arrays.asList(1L, 2L, 3L, 4L)),
POOL);
}
private CompletableFuture<String> fetchById(Long id) {
System.out.println("trigger fetching id: " + id + " thread: "
+ Thread.currentThread().getName());
return CompletableFuture.supplyAsync(
() -> slowDown("fetching id: " + id , db.get(id)), POOL);
}
static ForkJoinPool POOL = new ForkJoinPool(2);
public static void main(String[] args) {
PipelineOfTasksExample example = new PipelineOfTasksExample();
CompletableFuture<List<String>> result = example.returnUserIdsFromDb()
.thenComposeAsync(listOfIds -> {
List<CompletableFuture<String>> jobs = listOfIds.parallelStream()
.map(id -> example.fetchById(id))
.collect(Collectors.toList());
return CompletableFuture.allOf(jobs.toArray(new CompletableFuture<?>[0]))
.thenApply(_void -> jobs.stream()
.map(CompletableFuture::join).collect(Collectors.toList()));
},
POOL
);
System.out.println(result.join());
System.out.println(ForkJoinPool.commonPool().getPoolSize());
}
}
The difference is that first, all asynchronous jobs are submitted, then, a dependent action calling join on them is scheduled, to be executed only when all jobs have completed, so these join invocations will never block. Only the final join call at the end of the main method may block the main thread.
So this prints something like
trigger building the list of Ids - thread: main
building the list of Ids -> [1, 2, 3, 4] thread: ForkJoinPool-1-worker-1, 1 threads
trigger fetching id: 3 thread: ForkJoinPool-1-worker-1
trigger fetching id: 2 thread: ForkJoinPool-1-worker-0
trigger fetching id: 4 thread: ForkJoinPool-1-worker-1
trigger fetching id: 1 thread: ForkJoinPool-1-worker-0
fetching id: 4 -> user4 thread: ForkJoinPool-1-worker-1, 2 threads
fetching id: 3 -> user3 thread: ForkJoinPool-1-worker-0, 2 threads
fetching id: 2 -> user2 thread: ForkJoinPool-1-worker-1, 2 threads
fetching id: 1 -> user1 thread: ForkJoinPool-1-worker-0, 2 threads
[user1, user2, user3, user4]
showing that no compensation threads had to be created, so the number of threads matches the configured target parallelism.
Note that if the actual work is done in a background thread rather than within the fetchById method itself, you now don’t need a parallel stream anymore, as there is no blocking join() call. For such scenarios, just using stream() will usually result in a higher performance.