I am getting below info message every time in kafka consumer.
2020-07-04 14:54:27.640 INFO 1 --- [istener-0-0-C-1] c.n.o.c.h.p.n.PersistenceKafkaConsumer : beginning to consume batch messages , Message Count :11
2020-07-04 14:54:27.809 INFO 1 --- [istener-0-0-C-1] c.n.o.c.h.p.n.PersistenceKafkaConsumer : Execution Time :169
2020-07-04 14:54:27.809 INFO 1 --- [istener-0-0-C-1] essageListenerContainer$ListenerConsumer : Committing: {nbi.cm.changes.mo.test23-1=OffsetAndMetadata{offset=5705, leaderEpoch=null, metadata=''}}
2020-07-04 14:54:27.812 INFO 1 --- [istener-0-0-C-1] c.n.o.c.h.p.n.PersistenceKafkaConsumer : Acknowledgment Success
2020-07-04 14:54:27.813 INFO 1 --- [istener-0-0-C-1] o.a.k.c.consumer.internals.Fetcher : [Consumer clientId=consumer-1, groupId=cm-persistence-notification] Fetch offset 5705 is out of range for partition nbi.cm.changes.mo.test23-1, resetting offset
2020-07-04 14:54:27.820 INFO 1 --- [istener-0-0-C-1] o.a.k.c.c.internals.SubscriptionState : [Consumer clientId=consumer-1, groupId=cm-persistence-notification] Resetting offset for partition nbi.cm.changes.mo.test23-1 to offset 666703.
Got OFFSET_OUT_OF_RANGE error in debug log and resetting to some other partition that actually not exist. Same all messages able to receive in consumer console.
But actually I committed offset before that only , offset are available in kafka , log retention policy is 24hr, so it's not deleted in kafka.
In debug log, I got below messages:
beginning to consume batch messages , Message Count :710
2020-07-02 04:58:31.486 DEBUG 1 --- [ce-notification] o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-1, groupId=cm-persistence-notification] Node 1002 sent an incremental fetch response for session 253529272 with 1 response partition(s)
2020-07-02 04:58:31.486 DEBUG 1 --- [ce-notification] o.a.k.c.consumer.internals.Fetcher : [Consumer clientId=consumer-1, groupId=cm-persistence-notification] Fetch READ_UNCOMMITTED at offset 11372 for partition nbi.cm.changes.mo.test12-1 returned fetch data (error=OFFSET_OUT_OF_RANGE, highWaterMark=-1, lastStableOffset = -1, logStartOffset = -1, preferredReadReplica = absent, abortedTransactions = null, recordsSizeInBytes=0)
When all we will get OFFSET_OUT_OF_RANGE.
Listener Class :
#KafkaListener( id = "batch-listener-0", topics = "topic1", groupId = "test", containerFactory = KafkaConsumerConfiguration.CONTAINER_FACTORY_NAME )
public void receive(
#Payload List<String> messages,
#Header( KafkaHeaders.RECEIVED_MESSAGE_KEY ) List<String> keys,
#Header( KafkaHeaders.RECEIVED_PARTITION_ID ) List<Integer> partitions,
#Header( KafkaHeaders.RECEIVED_TOPIC ) List<String> topics,
#Header( KafkaHeaders.OFFSET ) List<Long> offsets,
Acknowledgment ack )
{
long startTime = System.currentTimeMillis();
handleNotifications( messages ); // will take more than 5s to process all messages
long endTime = System.currentTimeMillis();
long timeElapsed = endTime - startTime;
LOGGER.info( "Execution Time :{}", timeElapsed );
ack.acknowledge();
LOGGER.info( "Acknowledgment Success" );
}
Do i need to close consumer here , i thought spring-kafka automatically take care those , if no could you please tell how to close in apring-kafka and also how to check if rebalance happened or not , because in DEBUG log not able to see any log related to rebalance.
I think your consumer may be rebalancing, because you are not calling consumer.close() at the end of your process.
This is a guess, but if the retention policy isn't kicking in (and the logs are not being deleted), this is the only reason I can tell for that behaviour.
Update:
As you set them as #KafkaListeners, you could just call stop() on the KafkaListenerEndpointRegistry: kafkaListenerEndpointRegistry.stop()
Related
I'm trying to reset offset to a partition to 0, like this:
final KafkaConsumer<String, String> stateConsumer = new KafkaConsumer<>(stateConsumerProperties.getConsumerProps());
stateConsumer.subscribe(STATE_TOPIC);
...
stateConsumer.seekToBeginning(stateConsumer.assignment());
...
stateConsumer.poll(Duration.ofMillis(1000)) // timeout too long, testing only
.forEach(record -> {
log.info("Warmup state read: " + record.value() + ", partition: " + record.partition());
stateMessages.add(record.value());
});
Consumer config is only this:
this.stateConsumerProperties.setConsumer(Map.of(
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka:9092",
ConsumerConfig.GROUP_ID_CONFIG, "state",
ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest",
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer",
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer",
ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "30",
ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, "2000",
ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, CustomPartitionAssignor.class.getName()
));
The "custom" assignor is only for logging purposes, these are the only overrides:
public class CustomPartitionAssignor extends CooperativeStickyAssignor {
...
#Override
public GroupAssignment assign(Cluster metadata, GroupSubscription groupSubscription) {
return super.assign(metadata, groupSubscription);
}
#Override
public void onAssignment(Assignment assignment, ConsumerGroupMetadata metadata) {
super.onAssignment(assignment, metadata);
Arrays.toString(ASSIGNED_PARTITIONS.toArray()));
ASSIGNED_PARTITIONS = assignment.partitions();
log.info("New Assigned partitions: " + assignment.partitions());
}
...
}
I have only two identical consumers for testing and each consumer runs this code on join.
The first joiner has no issue since there is not state yet ("consumers" read another topic and produce "state" to state topic).
Problem is, this is what I get in the log when the second consumer joins:
consumer_2 | 2022-10-10 18:46:26.833 INFO 7 --- [pool-1-thread-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-state-2, groupId=state] Setting offset for partition state-1 to the committed offset FetchPosition{offset=361, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[179bd3c6448e:9092 (id: 1001 rack: null)], epoch=0}}
consumer_2 | 2022-10-10 18:46:26.875 INFO 7 --- [pool-1-thread-1] c.company.fdi.poc.kafka.service.Consumer : Warmup state read: state 723, partition: 1
consumer_2 | 2022-10-10 18:46:26.876 INFO 7 --- [pool-1-thread-1] c.company.fdi.poc.kafka.service.Consumer : Warmup state read: state 725, partition: 1
consumer_2 | 2022-10-10 18:46:26.876 INFO 7 --- [pool-1-thread-1] c.company.fdi.poc.kafka.service.Consumer : Warmup state read: state 727, partition: 1
State "production" is just an atomic counter that starts at 0.
What I want to happen is that when a new consumer joins, it resets its assigned partition offset to 0 and starts reading from the first record.
I have a suspicion that this has something to do with the CooperativeStickyAssignor, but I have no clue as of now.
You can assume 2 consumers, 2 partitions for state topic if this helps. Cluster should be as balanced as possible.
Start is two partitions one consumer, then I add another consumer.
Any help much appreciated, thanks in advance.
I have a requirement where I need to process messages from Kafka without losing any message and also need to maintain the message order. Therefore, I used transactions and enabled 'exactly_once' processing guarantee in my Kafka streams topology. As I assume that the topology processing will be 'all or nothing', that the message offset is committed only after the last node successfully processed the message.
However in a failure scenario, for example when the database is down and the processor fails to store message and throws an exception. At this point, the topology dies as intended and is recreated automatically on rebalance. I assume that the topology should either re-consume the original message again from the Kafka topic OR on application restart, it should re-consume that original message from Kafka topic. However, it seems that original message disappears and is never consumed or processed after that topology died.
What do I need to do to reprocess the original message sent to Kafka topic? Or what Kafka configuration requires change? Do I need manually assign a state store and keep track of messages processed on a changelog topic?
Topology:
#Singleton
public class EventTopology extends Topology {
private final Deserializer<String> deserializer = Serdes.String().deserializer();
private final Serializer<String> serializer = Serdes.String().serializer();
private final EventLogMessageSerializer eventLogMessageSerializer;
private final EventLogMessageDeserializer eventLogMessageDeserializer;
private final EventLogProcessorSupplier eventLogProcessorSupplier;
#Inject
public EventTopology(EventsConfig eventsConfig,
EventLogMessageSerializer eventLogMessageSerializer,
EventLogMessageDeserializer eventLogMessageDeserializer,
EventLogProcessorSupplier eventLogProcessorSupplier) {
this.eventLogMessageSerializer = eventLogMessageSerializer;
this.eventLogMessageDeserializer = eventLogMessageDeserializer;
this.eventLogProcessorSupplier = eventLogProcessorSupplier;
init(eventsConfig);
}
private void init(EventsConfig eventsConfig) {
var topics = eventsConfig.getTopicConfig().getTopics();
String eventLog = topics.get("eventLog");
addSource("EventsLogSource", deserializer, eventLogMessageDeserializer, eventLog)
.addProcessor("EventLogProcessor", eventLogProcessorSupplier, "EventsLogSource");
}
}
Processor:
#Singleton
#Slf4j
public class EventLogProcessor implements Processor<String, EventLogMessage> {
private final EventLogService eventLogService;
private ProcessorContext context;
#Inject
public EventLogProcessor(EventLogService eventLogService) {
this.eventLogService = eventLogService;
}
#Override
public void init(ProcessorContext context) {
this.context = context;
}
#Override
public void process(String key, EventLogMessage value) {
log.info("Processing EventLogMessage={}", value);
try {
eventLogService.storeInDatabase(value);
context.commit();
} catch (Exception e) {
log.warn("Failed to process EventLogMessage={}", value, e);
throw e;
}
}
#Override
public void close() {
}
}
Configuration:
eventsConfig:
saveTopicsEnabled: false
topologyConfig:
environment: "LOCAL"
broker: "localhost:9093"
enabled: true
initialiseWaitInterval: 3 seconds
applicationId: "eventsTopology"
config:
auto.offset.reset: latest
session.timeout.ms: 6000
fetch.max.wait.ms: 7000
heartbeat.interval.ms: 5000
connections.max.idle.ms: 7000
security.protocol: SSL
key.serializer: org.apache.kafka.common.serialization.StringSerializer
value.serializer: org.apache.kafka.common.serialization.StringSerializer
max.poll.records: 5
processing.guarantee: exactly_once
metric.reporters: com.simple.metrics.kafka.DropwizardReporter
default.deserialization.exception.handler: org.apache.kafka.streams.errors.LogAndContinueExceptionHandler
enable.idempotence: true
request.timeout.ms: 8000
acks: all
batch.size: 16384
linger.ms: 1
enable.auto.commit: false
state.dir: "/tmp"
topicConfig:
topics:
eventLog: "EVENT-LOG-LOCAL"
kafkaTopicConfig:
partitions: 18
replicationFactor: 1
config:
retention.ms: 604800000
Test:
Feature: Feature covering the scenarios to process event log messages produced by external client.
Background:
Given event topology is healthy
Scenario: event log messages produced are successfully stored in the database
Given database is down
And the following event log messages are published
| deptId | userId | eventType | endDate | eventPayload_partner |
| dept-1 | user-1234 | CREATE | 2021-04-15T00:00:00Z | PARTNER-1 |
When database is up
And database is healthy
Then event log stored in the database as follows
| dept_id | user_id | event_type | end_date | event_payload |
| dept-1 | user-1234 | CREATE | 2021-04-15T00:00:00Z | {"partner":"PARTNER-1"} |
Logs:
INFO [data-plane-kafka-request-handler-1] kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Preparing to rebalance group eventsTopology in state PreparingRebalance with old generation 0 (__consumer_offsets-0) (reason: Adding new member eventsTopology-57fdac0e-09fb-4aa0-8b0b-7e01809b31fa-StreamThread-1-consumer-96a3e980-4286-461e-8536-5f04ccb2c778 with group instance id None)
INFO [executor-Rebalance] kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Stabilized group eventsTopology generation 1 (__consumer_offsets-0)
INFO [data-plane-kafka-request-handler-2] kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Assignment received from leader for group eventsTopology for generation 1
INFO [data-plane-kafka-request-handler-1] kafka.coordinator.transaction.TransactionCoordinator - [TransactionCoordinator id=0] Initialized transactionalId eventsTopology-0_0 with producerId 0 and producer epoch 0 on partition __transaction_state-4
INFO [data-plane-kafka-request-handler-6] kafka.coordinator.transaction.TransactionCoordinator - [TransactionCoordinator id=0] Initialized transactionalId eventsTopology-0_1 with producerId 1 and producer epoch 0 on partition __transaction_state-3
...
INFO [data-plane-kafka-request-handler-0] kafka.coordinator.transaction.TransactionCoordinator - [TransactionCoordinator id=0] Initialized transactionalId eventsTopology-0_16 with producerId 17 and producer epoch 0 on partition __transaction_state-37
INFO [data-plane-kafka-request-handler-4] kafka.coordinator.transaction.TransactionCoordinator - [TransactionCoordinator id=0] Initialized transactionalId eventsTopology-1_1 with producerId 18 and producer epoch 0 on partition __transaction_state-42
INFO [data-plane-kafka-request-handler-6] kafka.coordinator.transaction.TransactionCoordinator - [TransactionCoordinator id=0] Initialized transactionalId eventsTopology-1_0 with producerId 19 and producer epoch 0 on partition __transaction_state-43
...
INFO [data-plane-kafka-request-handler-3] kafka.coordinator.transaction.TransactionCoordinator - [TransactionCoordinator id=0] Initialized transactionalId eventsTopology-1_17 with producerId 34 and producer epoch 0 on partition __transaction_state-45
INFO [data-plane-kafka-request-handler-5] kafka.coordinator.transaction.TransactionCoordinator - [TransactionCoordinator id=0] Initialized transactionalId eventsTopology-1_16 with producerId 35 and producer epoch 0 on partition __transaction_state-46
INFO [pool-26-thread-1] ManagerClient - Manager request {uri:http://localhost:8081/healthcheck, method:GET, body:'', headers:{}}
INFO [pool-26-thread-1] ManagerClient - Manager response from with body {"Database":{"healthy":true},"eventsTopology":{"healthy":true}}
INFO [dw-admin-130] KafkaConnectionCheck - successfully connected to kafka broker: localhost:9093
INFO [kafka-producer-network-thread | EVENT-LOG-LOCAL-test-client-id] LocalTestEnvironment - Message: ProducerRecord(topic=EVENT-LOG-LOCAL, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value={"endDate":1618444800000,"deptId":"dept-1","userId":"user-1234","eventType":"CREATE","eventPayload":{"previousEndDate":null,"partner":"PARTNER-1","info":null}}, timestamp=null) pushed onto topic: EVENT-LOG-LOCAL
INFO [eventsTopology-b21df600-cd39-4c9d-9e7a-f55f53ac9fd3-StreamThread-1] EventLogProcessor - Processing EventLogMessage=EventLogMessage(endDate=Thu Apr 15 01:00:00 BST 2021, deptId=dept-1, userId=user-1234, eventType=CREATE, eventPayload=EventLogMessage.EventPayload(previousEndDate=null, partner=PARTNER-1, info=null))
WARN [eventsTopology-b21df600-cd39-4c9d-9e7a-f55f53ac9fd3-StreamThread-1] EventLogProcessor - Failed to process EventLogMessage=EventLogMessage(endDate=Thu Apr 15 01:00:00 BST 2021, deptId=dept-1, userId=user-1234, eventType=CREATE, eventPayload=EventLogMessage.EventPayload(previousEndDate=null, partner=PARTNER-1, info=null))
exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
at manager.service.EventLogService.storeInDatabase(EventLogService.java:24)
at manager.topology.processor.EventLogProcessor.process(EventLogProcessor.java:47)
at manager.topology.processor.EventLogProcessor.process(EventLogProcessor.java:19)
at org.apache.kafka.streams.processor.internals.ProcessorNode.lambda$process$2(ProcessorNode.java:142)
at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:836)
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:142)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:236)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:216)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:168)
at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:96)
at org.apache.kafka.streams.processor.internals.StreamTask.lambda$process$1(StreamTask.java:679)
at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:836)
at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:679)
at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1033)
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:690)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
ERROR [eventsTopology-b21df600-cd39-4c9d-9e7a-f55f53ac9fd3-StreamThread-1] org.apache.kafka.streams.processor.internals.TaskManager - stream-thread [eventsTopology-b21df600-cd39-4c9d-9e7a-f55f53ac9fd3-StreamThread-1] Failed to process stream task 0_8 due to the following error:
org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=0_8, processor=EventsLogSource, topic=EVENT-LOG-LOCAL, partition=8, offset=0, stacktrace=exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
ERROR [eventsTopology-b21df600-cd39-4c9d-9e7a-f55f53ac9fd3-StreamThread-1] org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [eventsTopology-b21df600-cd39-4c9d-9e7a-f55f53ac9fd3-StreamThread-1] Encountered the following exception during processing and the thread is going to shut down:
org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=0_8, processor=EventsLogSource, topic=EVENT-LOG-LOCAL, partition=8, offset=0, stacktrace=exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
ERROR [eventsTopology-b21df600-cd39-4c9d-9e7a-f55f53ac9fd3-StreamThread-1] org.apache.kafka.streams.KafkaStreams - stream-client [eventsTopology-b21df600-cd39-4c9d-9e7a-f55f53ac9fd3] All stream threads have died. The instance will be in error state and should be closed.
Exception: java.lang.IllegalStateException thrown from the UncaughtExceptionHandler in thread "eventsTopology-b21df600-cd39-4c9d-9e7a-f55f53ac9fd3-StreamThread-1"
INFO [executor-Heartbeat] kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Member eventsTopology-b21df600-cd39-4c9d-9e7a-f55f53ac9fd3-StreamThread-1-consumer-f11ca299-2a68-4317-a559-dd1b96cd431f in group eventsTopology has failed, removing it from the group
INFO [executor-Heartbeat] kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Preparing to rebalance group eventsTopology in state PreparingRebalance with old generation 1 (__consumer_offsets-0) (reason: removing member eventsTopology-b21df600-cd39-4c9d-9e7a-f55f53ac9fd3-StreamThread-1-consumer-f11ca299-2a68-4317-a559-dd1b96cd431f on heartbeat expiration)
INFO [data-plane-kafka-request-handler-2] kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Stabilized group eventsTopology generation 2 (__consumer_offsets-0)
INFO [data-plane-kafka-request-handler-6] kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Assignment received from leader for group eventsTopology for generation 2
INFO [data-plane-kafka-request-handler-0] kafka.coordinator.transaction.TransactionCoordinator - [TransactionCoordinator id=0] Initialized transactionalId eventsTopology-0_0 with producerId 0 and producer epoch 1 on partition __transaction_state-4
...
INFO [data-plane-kafka-request-handler-0] kafka.coordinator.transaction.TransactionCoordinator - [TransactionCoordinator id=0] Initialized transactionalId eventsTopology-1_16 with producerId 35 and producer epoch 1 on partition __transaction_state-46
INFO [main] Cluster - New databse host localhost/127.0.0.1:59423 added
com.jayway.awaitility.core.ConditionTimeoutException: Condition defined as a lambda expression in steps.EventLogSteps
Expecting:
<0>
to be equal to:
<1>
but was not. within 20 seconds.
So, there is an external server (game). There is a market there. A lot of products and their combinations. Their total number is 2146.
I want to receive up-to-date pricing information from time to time.
When the application starts, I create 2146 tasks, each of which is responsible for its own type of product. Tasks run in a separate thread with a delay of 2.5 seconds.
#EventListener(ApplicationReadyEvent.class)
public void start() {
log.info("Let's get party started!");
Set<MarketplaceCollector> collectorSet = marketplaceCollectorProviders.stream()
.flatMap(provider -> provider.getCollectors().stream())
.peek(this::subscribeOfferDBSubscriber)
.collect(Collectors.toSet());
collectors.addAll(collectorSet);
runTasks();
}
private void subscribeOfferDBSubscriber(MarketplaceCollector marketplaceCollector) {
marketplaceCollector.subscribe(marketplaceOfferDBSubscriber);
}
private void runTasks() {
Thread thread = new Thread(() -> collectors.forEach(this::runWithDelay));
thread.setName("tasks-runner");
thread.start();
}
private void runWithDelay(Collector collector) {
try {
collector.collect();
Thread.sleep(2_500);
counter += 1;
} catch (InterruptedException e) {
log.error(e);
}
log.debug(counter);
}
Using RestTemplate, I access the server. If the price has changed, this task will be completed again after 1 minute. If the price remains the same, add one minute to the wait and again make a request. Thus, if the price does not change, the maximum time between requests for one product will be 20 minutes. I assume that my application will execute up to 200 requests per minute, otherwise I will get a "too many requests" error.
#Override
public void collect() {
executorService.schedule(new MarketplaceTask(), INIT_DELAY, MILLISECONDS);
}
private MarketplaceRequest request() {
return MarketplaceRequest.builder()
.country(country)
.industry(industry)
.quality(quality)
.orderBy(ASC)
.currentPage(1)
.build();
}
private class MarketplaceTask implements Runnable {
private long MIN_DELAY = 60; // 1 minute
private long MAX_DELAY = 1200; // 20 minutes
private Double PREVIOUS_PRICE = Double.MAX_VALUE;
private long DELAY = 0; // seconds
#Override
public void run() {
log.debug(format("Collecting offer of %s %s in %s after %d m delay", industry, quality, country, DELAY / 60));
MarketplaceResponse response = marketplaceClient.getOffers(request());
subscribers.forEach(s -> s.onSubscription(response));
updatePreviousPriceAndPeriod(response);
executorService.schedule(this, DELAY, SECONDS);
}
private void updatePreviousPriceAndPeriod(MarketplaceResponse response) {
if (response.isError() || response.getOffers().isEmpty()) {
increasePeriod();
} else {
Double currentPrice = response.getOffers().get(0).getPriceWithTaxes();
if (isPriceChanged(currentPrice)) {
setMinimalDelay();
PREVIOUS_PRICE = currentPrice;
} else {
increasePeriod();
}
}
}
private void increasePeriod() {
if (DELAY < MAX_DELAY) {
DELAY += 60;
}
}
private boolean isPriceChanged(Double currentPrice) {
return !Objects.equals(currentPrice, PREVIOUS_PRICE);
}
private void setMinimalDelay() {
DELAY = MIN_DELAY;
}
}
public MarketplaceClient(#Value("${app.host}") String host,
AuthenticationService authenticationService,
RestTemplateBuilder restTemplateBuilder,
CommonHeadersComposer headersComposer) {
this.host = host;
this.authenticationService = authenticationService;
this.restTemplateList = restTemplateBuilder.build();
this.headersComposer = headersComposer;
}
public MarketplaceResponse getOffers(MarketplaceRequest request) {
var authentication = authenticationService.getAuthentication();
var requestEntity = new HttpEntity<>(requestString(request, authentication), headersComposer.getHeaders());
log.debug(message("PING for", request));
var responseEntity = restTemplate.exchange(host + MARKET_URL, POST, requestEntity, MarketplaceResponse.class);
log.debug(message("PONG for", request));
if (responseEntity.getBody().isError()) {
log.warn("{}: {} {} in {}", responseEntity.getBody().getMessage(), request.getIndustry(), request.getQuality(), request.getCountry());
}
return responseEntity.getBody();
}
private String requestString(MarketplaceRequest request, Authentication authentication) {
return format("countryId=%s&industryId=%s&quality=%s&orderBy=%s¤tPage=%s&ajaxMarket=1&_token=%s",
request.getCountry().getId(), request.getIndustry().getId(), request.getQuality().getValue(),
request.getOrderBy().getValue(), request.getCurrentPage(), authentication.getToken());
}
But I have a problem after a few minutes of the application. Some tasks cease to care. The request may go to the server and not return. However, other tasks work without problems. Logs how it behaves (for example):
2020-04-04 14:11:58.267 INFO 3546 --- [ main] c.g.d.e.harvesting.CollectorManager : Let's get party started!
2020-04-04 14:11:58.302 DEBUG 3546 --- [pool-1-thread-1] c.g.d.e.harvesting.MarketplaceCollector : Collecting offer of WEAPONS Q5 in GREECE after 0 m delay
2020-04-04 14:11:58.379 DEBUG 3546 --- [pool-1-thread-1] c.g.d.e.market.api.MarketplaceClient : PING for: WEAPONS Q5 in GREECE
2020-04-04 14:11:59.217 DEBUG 3546 --- [pool-1-thread-1] c.g.d.e.market.api.MarketplaceClient : PONG for: WEAPONS Q5 in GREECE
2020-04-04 14:12:00.805 DEBUG 3546 --- [ tasks-runner] c.g.d.e.harvesting.CollectorManager : 1
2020-04-04 14:12:00.806 DEBUG 3546 --- [pool-1-thread-1] c.g.d.e.harvesting.MarketplaceCollector : Collecting offer of WEAPONS Q4 in PAKISTAN after 0 m delay
2020-04-04 14:12:00.807 DEBUG 3546 --- [pool-1-thread-1] c.g.d.e.market.api.MarketplaceClient : PING for: WEAPONS Q4 in PAKISTAN
2020-04-04 14:12:03.308 DEBUG 3546 --- [ tasks-runner] c.g.d.e.harvesting.CollectorManager : 2
2020-04-04 14:12:03.309 DEBUG 3546 --- [pool-1-thread-2] c.g.d.e.harvesting.MarketplaceCollector : Collecting offer of FOOD_RAW Q1 in SAUDI_ARABIA after 0 m delay
2020-04-04 14:12:03.311 DEBUG 3546 --- [pool-1-thread-2] c.g.d.e.market.api.MarketplaceClient : PING for: FOOD_RAW Q1 in SAUDI_ARABIA
2020-04-04 14:12:05.810 DEBUG 3546 --- [ tasks-runner] c.g.d.e.harvesting.CollectorManager : 3
2020-04-04 14:12:05.810 DEBUG 3546 --- [pool-1-thread-1] c.g.d.e.harvesting.MarketplaceCollector : Collecting offer of WEAPONS Q5 in COLOMBIA after 0 m delay
2020-04-04 14:12:05.811 DEBUG 3546 --- [pool-1-thread-1] c.g.d.e.market.api.MarketplaceClient : PING for: WEAPONS Q5 in COLOMBIA
2020-04-04 14:12:08.314 DEBUG 3546 --- [ tasks-runner] c.g.d.e.harvesting.CollectorManager : 4
2020-04-04 14:12:08.315 DEBUG 3546 --- [pool-1-thread-4] c.g.d.e.harvesting.MarketplaceCollector : Collecting offer of WEAPONS Q1 in CZECH_REPUBLIC after 0 m delay
2020-04-04 14:12:08.316 DEBUG 3546 --- [pool-1-thread-4] c.g.d.e.market.api.MarketplaceClient : PING for: WEAPONS Q1 in CZECH_REPUBLIC
2020-04-04 14:12:10.818 DEBUG 3546 --- [ tasks-runner] c.g.d.e.harvesting.CollectorManager : 5
#Configuration
public class BeanConfiguration {
#Bean
public ScheduledExecutorService scheduledExecutorService() {
return Executors.newScheduledThreadPool(8);
}
}
I tried to change the connection pool for one host, but I only made it worse. I even created 200 instances of RestTemplate, but over time the access to the server ceased.
I would not want to use Spring Webflux for this purpose.
What should I do to make the app work as expected?
I'm learning Kafka Stream and I want to build a simple application that reads lines of text from one topic and puts number of letter occurrences to InfluxDB. My second goal is to run it in parallel. I would like to have two running instances of Kafka Stream both of them processing input at the same time but from different partitions. Unfortunately, after I launch both instances only one of them is working second one is waiting.
My configuration is as follows:
* input topic has 4 partitions
* Stream application has a property num.stream.threads set to 2
Here is the source code:
public class LineStatisticsStream {
static Pattern letterPattern = Pattern.compile("[a-z]");
InfluxDB influxDB;
public static void main(String[] args) {
new LineStatisticsStream().start();
}
Properties getStreamProperties() {
Properties properties = new Properties();
properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092");
properties.put(StreamsConfig.APPLICATION_ID_CONFIG,"lineStatistics4");
properties.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
properties.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG,2);
return properties;
}
Topology getStreamTopology(ForeachAction<String,Long> action) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String,String> linesStream = builder.stream("lines", Consumed.with(Serdes.String(),Serdes.String()));
linesStream.flatMapValues(this::findAllLetters)
.map((k,v)-> KeyValue.pair(v,v))
.groupByKey(Grouped.with(Serdes.String(),Serdes.String()))
.count()
.toStream()
.to("letterCount",Produced.with(Serdes.String(),Serdes.Long()));
KStream<String,Long> countStream = builder.stream("letterCount", Consumed.with(Serdes.String(),Serdes.Long()));
countStream.peek(action);
return builder.build();
}
private void insertToInfluxDB(String letter, Long count) {
influxDB.write(Point.measurement("lettersCount")
.tag("letter",letter)
.time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
.addField("count", count)
.build()
);
}
private List<String> findAllLetters(String s) {
List<String> letters = new ArrayList<>();
Matcher matcher= letterPattern.matcher(s.toLowerCase());
while(matcher.find()) {
letters.add(matcher.group(0));
}
return letters;
}
private void start() {
influxDB = InfluxDBFactory.connect("http://127.0.0.1:8086");
influxDB.setDatabase("letters");
influxDB.enableBatch(BatchOptions.DEFAULTS);
final KafkaStreams streams = new KafkaStreams(getStreamTopology(this::insertToInfluxDB), getStreamProperties());
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(()->{streams.close();influxDB.close();}));
}
}
And last few lines of a log from "waiting" application:
2019-11-18 21:27:16 INFO ConsumerCoordinator:982 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-1-consumer, groupId=lineStatistics4] Found no committed offset for partition lines4-2
2019-11-18 21:27:16 INFO ConsumerCoordinator:982 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-2-consumer, groupId=lineStatistics4] Found no committed offset for partition lines4-1
2019-11-18 21:27:16 INFO RocksDBTimestampedStore:82 - Opening store KSTREAM-AGGREGATE-STATE-STORE-0000000003 in regular mode
2019-11-18 21:27:16 INFO RocksDBTimestampedStore:82 - Opening store KSTREAM-AGGREGATE-STATE-STORE-0000000003 in regular mode
2019-11-18 21:27:16 INFO KafkaConsumer:1068 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-2-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions
2019-11-18 21:27:16 INFO KafkaConsumer:1068 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-2-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions
2019-11-18 21:27:16 INFO StreamThread:212 - stream-thread [lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING
2019-11-18 21:27:16 INFO KafkaConsumer:1068 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-1-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions
2019-11-18 21:27:16 INFO KafkaConsumer:1068 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-1-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions
2019-11-18 21:27:16 INFO StreamThread:212 - stream-thread [lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING
2019-11-18 21:27:16 INFO KafkaStreams:263 - stream-client [lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099] State transition from REBALANCING to RUNNING
2019-11-18 21:27:16 INFO ConsumerCoordinator:982 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-2-consumer, groupId=lineStatistics4] Found no committed offset for partition lines4-1
2019-11-18 21:27:16 INFO ConsumerCoordinator:982 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-1-consumer, groupId=lineStatistics4] Found no committed offset for partition lines4-2
2019-11-18 21:27:16 INFO ConsumerCoordinator:525 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-2-consumer, groupId=lineStatistics4] Setting offset for partition lineStatistics4-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0 to the committed offset FetchPosition{offset=73, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=localhost:9092 (id: 1 rack: null), epoch=0}}
2019-11-18 21:27:16 INFO ConsumerCoordinator:525 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-1-consumer, groupId=lineStatistics4] Setting offset for partition lines4-0 to the committed offset FetchPosition{offset=63, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=localhost:9092 (id: 1 rack: null), epoch=0}}
2019-11-18 21:27:16 INFO ConsumerCoordinator:525 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-1-consumer, groupId=lineStatistics4] Setting offset for partition lineStatistics4-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-3 to the committed offset FetchPosition{offset=40, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=localhost:9092 (id: 1 rack: null), epoch=0}}
2019-11-18 21:27:17 INFO SubscriptionState:348 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-2-consumer, groupId=lineStatistics4] Resetting offset for partition lines4-1 to offset 0.
2019-11-18 21:27:17 INFO SubscriptionState:348 - [Consumer clientId=lineStatistics4-5e5e2a71-2a3b-4e51-9d4c-4e470a8a8099-StreamThread-1-consumer, groupId=lineStatistics4] Resetting offset for partition lines4-2 to offset 0.
I have two scenarios in my script. 1st "getAssets" scenario will fetch all asset IDs and save it in a list, 2nd scenario "fetchMetadata" will iterate those IDs.
I have to exeecute "getAssets" scenario only once to fetch all the IDs, and then "fetchMetadata" scenario will execute till given time duration.
Here is the Json response of "/api/assets;limit=$limit" request (We are fetching id from here using $.assets[*].id),
{
"assets": [
{
"id": 3010411,
"name": "Asset 2016-11-22 20:06:07",
....
....
},
{
"id": 3010231,
"name": "Asset 2016-11-22 20:07:07",
....
....
}, and so on..
Here is the code
import java.util.concurrent.ThreadLocalRandom
import scala.concurrent.duration._
import io.gatling.core.Predef._
import io.gatling.http.Predef._
class getAssetsMetadata extends Simulation {
val getAssetURL = System.getProperty("getAssetURL", "https://performancetesting.net")
val username = System.getProperty("username", "performanceuser")
val password = System.getProperty("password", "performanceuser")
val limit = Integer.getInteger("limit", 1000).toInt
val userCount = Integer.getInteger("userCount", 100).toInt
val duration = Integer.getInteger("duration",1).toInt //in minutes
var IdList: Seq[String] = _
val httpProtocol = http
.basicAuth(username, password)
.baseURL(getAssetURL)
.contentTypeHeader("""application/vnd.v1+json""")
// Scenario 1 get assets
val getAssets = scenario("Get Assets")
.exec(http("List of Assets")
.get(s"""/api/assets;limit=$limit""")
.check(jsonPath("$.assets[*].id").findAll.transform {v => IdList = v; v }.saveAs("IdList"))
)
// Scenario 2 Fetch Metadata
val fetchMetadata = scenario("Fetch Metadata")
.exec(_.set("IdList", IdList))
.exec(http("Metadata Request")
.get("""/api/assets/${IdList.random()}/metadata""")
)
val scn = List(getAssets.inject(atOnceUsers(1)), fetchMetadata.inject(constantUsersPerSec(userCount) during(duration minutes)))
setUp(scn).protocols(httpProtocol)
}
:::ERROR:::
It throws "Value is null" (While we have 10 million asset IDs here). here is the Gatling log
14883 [GatlingSystem-akka.actor.default-dispatcher-4] INFO io.gatling.http.config.HttpProtocol - Warm up done
14907 [GatlingSystem-akka.actor.default-dispatcher-4] INFO io.gatling.http.config.HttpProtocol - Start warm up
14909 [GatlingSystem-akka.actor.default-dispatcher-4] INFO io.gatling.http.config.HttpProtocol - Warm up done
14911 [GatlingSystem-akka.actor.default-dispatcher-4] INFO i.g.core.controller.Controller - Total number of users : 6001
14918 [GatlingSystem-akka.actor.default-dispatcher-6] INFO i.g.c.r.writer.ConsoleDataWriter - Initializing
14918 [GatlingSystem-akka.actor.default-dispatcher-3] INFO i.g.c.result.writer.FileDataWriter - Initializing
14923 [GatlingSystem-akka.actor.default-dispatcher-6] INFO i.g.c.r.writer.ConsoleDataWriter - Initialized
14928 [GatlingSystem-akka.actor.default-dispatcher-3] INFO i.g.c.result.writer.FileDataWriter - Initialized
14931 [GatlingSystem-akka.actor.default-dispatcher-4] DEBUG i.g.core.controller.Controller - Launching All Scenarios
14947 [GatlingSystem-akka.actor.default-dispatcher-12] ERROR i.g.http.action.HttpRequestAction - 'httpRequest-2' failed to execute: Value is null
14954 [GatlingSystem-akka.actor.default-dispatcher-4] DEBUG i.g.core.controller.Controller - Finished Launching scenarios executions
14961 [GatlingSystem-akka.actor.default-dispatcher-4] DEBUG i.g.core.controller.Controller - Setting up max duration
14962 [GatlingSystem-akka.actor.default-dispatcher-4] INFO i.g.core.controller.Controller - Start user #7187317726850756780-0
14963 [GatlingSystem-akka.actor.default-dispatcher-4] INFO i.g.core.controller.Controller - Start user #7187317726850756780-1
14967 [GatlingSystem-akka.actor.default-dispatcher-4] INFO i.g.core.controller.Controller - End user #7187317726850756780-1
14970 [GatlingSystem-akka.actor.default-dispatcher-4] INFO i.g.core.controller.Controller - Start user #7187317726850756780-2
14970 [GatlingSystem-akka.actor.default-dispatcher-5] INFO io.gatling.http.ahc.HttpEngine - Sending request=List of Assets uri=https://performancetesting.net/api/assets;limit=1000: scenario=Get Assets, userId=7187317726850756780-0
14970 [GatlingSystem-akka.actor.default-dispatcher-7] ERROR i.g.http.action.HttpRequestAction - 'httpRequest-2' failed to execute: Value is null
14972 [GatlingSystem-akka.actor.default-dispatcher-7] INFO i.g.core.controller.Controller - End user #7187317726850756780-2
14980 [GatlingSystem-akka.actor.default-dispatcher-7] ERROR i.g.http.action.HttpRequestAction - 'httpRequest-2' failed to execute: Value is null
14980 [GatlingSystem-akka.actor.default-dispatcher-4] INFO i.g.core.controller.Controller - Start user #7187317726850756780-3
14984 [GatlingSystem-akka.actor.default-dispatcher-4] INFO i.g.core.controller.Controller - End user #7187317726850756780-3
.....
.....
.....
61211 [GatlingSystem-akka.actor.default-dispatcher-12] INFO i.g.core.controller.Controller - Start user #7187317726850756780-4626
61211 [GatlingSystem-akka.actor.default-dispatcher-7] ERROR i.g.http.action.HttpRequestAction - 'httpRequest-2' failed to execute: Value is null
61211 [GatlingSystem-akka.actor.default-dispatcher-7] INFO i.g.core.controller.Controller - End user #7187317726850756780-4626
61224 [GatlingSystem-akka.actor.default-dispatcher-2] INFO i.g.core.controller.Controller - Start user #7187317726850756780-4627
61225 [GatlingSystem-akka.actor.default-dispatcher-5] INFO io.gatling.http.ahc.HttpEngine - Sending request=Metadata Request uri=https://performancetesting.net/api/assets/3010320/metadata: scenario=Fetch Metadata, userId=7187317726850756780-4627
61230 [GatlingSystem-akka.actor.default-dispatcher-12] INFO i.g.core.controller.Controller - Start user #7187317726850756780-4628
61230 [GatlingSystem-akka.actor.default-dispatcher-7] INFO io.gatling.http.ahc.HttpEngine - Sending request=Metadata Request uri=https://performancetesting.net/api/assets/3009939/metadata: scenario=Fetch Metadata, userId=7187317726850756780-4628
61233 [GatlingSystem-akka.actor.default-dispatcher-2] INFO i.g.core.controller.Controller - End user #7187317726850756780-0
61233 [New I/O worker #12] DEBUG c.n.h.c.p.netty.handler.Processor - Channel Closed: [id: 0x8c94a1ae, /192.168.100.108:56739 :> performancetesting.net/10.20.14.176:443] with attribute INSTANCE
---- Requests ------------------------------------------------------------------
> Global (OK=261 KO=40 )
> Metadata Request (OK=260 KO=40 )
> List of Assets (OK=1 KO=0 )
---- Errors --------------------------------------------------------------------
> Value is null 40 (100.0%)
================================================================================
Thank you.
The list of ID is not passed to the other scenario, because it is obtained by another "user" (session).
Your simulation reads like the following
1 user is obtaining the list of IDs (and keeps the list for itself)
AT THE SAME TIME, n users try to fetch assets with an (undefined) list of IDs
The first user doesnt talk to the others, both scenario injections are independent of each other.
A way to solve this is to store the list in a shared container (i.e. AtomicReference) and access this container from the second scenario.
To ensure, the container is populated, inject a nothingFor step at the beginning of the second scenario in order to wait for the first scenario to finish.
Another way is to fetch the list of IDs in the beginning of the second scenario - if it has not been fetched before (see container above).
I'm sure there are ways to accomplish this as well (i.e. using some feeder and fetch the list of ids before the actual test)