Develop Reference

Retry HTTP Request using Spring's Webclient

I'm using springframework's reactive WebClient to make a client HTTP request to another service.
I currently have:
PredictionClientService.java
var response = externalServiceClient.sendPostRequest(predictionDto);
if (response.getStatusCode() == HttpStatusCode.OK) {
predictionService.updateStatus(predictionDto, Status.OK);
} else {
listOfErrors.add(response.getPayload());
predictionService.updateStatus(predictionDto, Stage.FAIL);
//Perhaps change above line to Stage.PENDING and then
//Poll the DB every 30, 60, 120 mins
//if exhausted, then call
// predictionService.updateStatus(predictionDto, Stage.FAILED);??
}
}
ExternalServiceClient.java
public PredictionResponseDto sendPostRequest(PredictionDto predictionDto) {
var response = webClient.post()
.uri(url)
.contentType(MediaType.APPLICATION_JSON)
.body(BodyInserters.fromValue(predictionDto.getPayload()))
.exchange()
.retryWhen(Retry.backoff(3, Duration.ofMinutes(30)))
//Maybe I can remove the retry logic here
//and handle retrying in PredictionClientService?
.onErrorResume(throwable ->
Mono.just(ClientResponse.create(TIMEOUT_HTTP_CODE,
ExchangeStrategies.empty().build()).build()))
.blockOptional();
return response.map(clientResponse ->
new PredictionResponseDto(
clientResponse.rawStatusCode(),
clientResponse.bodyToMono(String.class).block()))
.orElse(PredictionResponseDto.builder().build());
}
This will retry a maximum of 3 times on intervals 30, 60, 120 mins. The issue is, I don't want to keep a processing for running upwards of 30 mins.
The top code block is probably where I need to add the retry logic (poll from database if status = pending and retries < 3)?
Is there any sensible solution here? I was thinking if I could save the failed request to a DB with columns 'Request Body', "Retry attempt", "Status" and poll from this? Although not sure if cron is the way to go here.
How would I retry sending the HTTP request every 30, 60, 120 mins to avoid these issues? Would appreciate any code samples or links!

Reactor Sink that emits only 1 event at a time?

I am playing with Replaying Reactor Sinks, I am trying to achieve a mix of a unicast and a replay processor. I would like it to emit to only one subscriber at the same (UnicastProcessor), but that it can also emit a default value on subscribe (ReplayProcessor). Here is something similar to the real case:
Flux<Boolean> monoC = Sinks.many().replay().latestOrDefault(true).asFlux().doOnNext(integer -> System.out.println(new Date() + " - " + Thread.currentThread().getName() + " emiting next"));
for(int i = 0; i < 5; i++) {
new Thread(() -> {
monoC.flatMap(unused ->
webClientBuilder.build()
.get()
.uri("https://www.google.com")
.retrieve()
.toEntityFlux(String.class)
.doOnSuccess(stringResponseEntity -> {
System.out.println(new Date() + " - " + Thread.currentThread().getName() + " finished processing");
})
).subscribe();
}).start();
}
That is printing:
emiting next
...
emiting next
finished processing
...
finished processing
Instead, I would like it to print:
emiting next
finished processing
...
emiting next
finished processing
Update, some more clarifications on the real case scenario:
The real case scenario is: I have a Spring WebFlux application that acts like a relay, it receives a request on a specific endpoint A, and it relays it to another microservice B. This microservice can then reply with a 429 if I go too fast, and in a header with how long I have to wait before retrying again. The retrying thing I have already achieved it with a .retry operator and a Mono.delay, but in the meantime, I can receive another request on my first endpoint A which will have to be blocked until the Mono.delay finishes.
I am trying to achieve this with a Replay Sink, so that after receiving a 429, I emit a "false" to the sink and after Mono.delay is over, it emits a true to the sink, so if in the mean time I receive any further request on A it can filter out all the falses and wait for a true to be emitted.
The problem i have on top of that is that, when I receive too many request to relay on A, microservice B starts responding slow, and getting overloaded. Therefore, i would like to limit the rate that the Sink is emitting. To be precise, i would like the publisher to emit a value, but don't emit any more until the subscriber hits onCompleted.

As soon as I understood your issue correctly, you want the requests to B being processed sequentially. In that case you should have a look at https://projectreactor.io/docs/core/release/api/reactor/core/publisher/Flux.html#flatMap-java.util.function.Function-int-
public final <V> Flux<V> flatMap(Function<? super T, ? extends Publisher<? extends V>> mapper, int concurrency)
I think your case should look like
//sinks should be global variable for your controller, initialized in #PostConstruct
var sinks = Sinks
//unsafe is required for multithreading
.unsafe()
.many()
.replay()
.latest();
sinks.asFlux()
.doOnNext(it -> System.out.printf("%s is emitting %s\n", Thread.currentThread().getName(), it))
.flatMap(counter -> {
return webClientBuilder.build()
.get()
.uri("https://www.google.com")
.retrieve()
.toEntityFlux(String.class)
.doOnSuccess(stringResponseEntity -> {
System.out.println(counter + " " + new Date() + " - " + Thread.currentThread().getName() + " finished processing with " + stringResponseEntity.getStatusCode());
})
.then(Mono.just(counter));
//concurrency = 1 causes the flatMap being handled only once in parallel
}, 1)
.doOnError(Throwable::printStackTrace)
//this subscription also must be done in #PostConstruct
.subscribe(counter -> System.out.printf("%s completed in %s\n", counter, Thread.currentThread().getName()));
//and this is your endpoint method
for (int i = 0; i < 5; i++) {
int counter = i;
new Thread(() -> {
var result = sinks.tryEmitNext(counter);
if (result.isFailure()) {
//mb in that case you should retry
System.out.printf("%s emitted %s. with fail: %s\n", Thread.currentThread().getName(), counter, result);
} else {
System.out.printf("%s successfully emitted %s\n", Thread.currentThread().getName(), counter);
}
}).start();
}

Apache Pulsar - What is the behaviour of the Consumer.seek() method by timestamp?

https://pulsar.apache.org/api/client/2.4.0/org/apache/pulsar/client/api/Consumer.html#seek-long-
When calling seek(long timestamp) method on the consumer, does timestamp have to equal the exact time a message was published?
For example, if i sent three messages at t=1, 5, 7 and if i call consumer.seek(3), will i get an error? or will my consumer get reset to t=3, so that if i call consumer.next(), i'll get my second message?
Thanks in advance,

The Consumer#seek(long timestamp) allows you to reset your subscription to a given timestamp. After seeking the consumer will start receiving messages with a publish time equal to or greater than the timestamp passed to the seek method.
The below example show how to reset a consumer to the previous hour:
try (
// Create PulsarClient
PulsarClient client = PulsarClient
.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
// Create Consumer subscription
Consumer<String> consumer = client.newConsumer(Schema.STRING)
.topic("my-topic")
.subscriptionName("my-subscription")
.subscriptionMode(SubscriptionMode.Durable)
.subscriptionType(SubscriptionType.Key_Shared)
.subscriptionInitialPosition(SubscriptionInitialPosition.Latest)
.subscribe()
) {
// Seek consumer to previous hour
consumer.seek(Instant.now().minus( Duration.ofHours(1)).toEpochMilli());
while (true) {
final Message<String> msg = consumer.receive();
System.out.printf(
"Message received: key=%s, value=%s, topic=%s, id=%s%n",
msg.getKey(),
msg.getValue(),
msg.getTopicName(),
msg.getMessageId().toString());
consumer.acknowledge(msg);
}
}
Note that if you have multiple consumers that belong to the same subscriptio ( e.g., Key_Shared) then all consumers will be reset.

Rxjava approach the get last Observable from each thread

I'm thinking how to use RXJava for the scenario described bellow.
A List<Object>,each object will be sent to k8s and checked the status till the respone return true,so my polling active is that：
private Observable<Boolean> startPolling(String content) {
log.info("start polling "+ content);
return Observable.interval(2, TimeUnit.SECONDS)
.take(3)
.observeOn(Schedulers.newThread())
.flatMap(aLong -> Observable.just(new CheckSvcStatus().check(content)))
.takeUntil(checkResult -> checkResult)
.timeout(3000L, TimeUnit.MILLISECONDS, Observable.just(false))
;
}
Function of sent action:
Observable<Compo> sentYamlAndGet() {
log.info("sent yaml");
sentYaml()
return Observable.just(content);
}
I try to use the foreach to get each object status which like this:
public void rxInstall() throws JsonProcessingException {
List<Boolean>observables = Lists.newArrayList();
Observable.from(list)
.subscribeOn(Schedulers.newThread())
.concatMap(s -> sendYamlAndGet())
.timeout(3000l, TimeUnit.MILLISECONDS)
.subscribe()
;
Observable.from(list).forEach(s -> {
observables.add(Observable.just(s)
.flatMap(this::startPolling)
.toBlocking()
.last()
)
;
System.out.println(new ObjectMapper().writeValueAsString(observables));
}
Objects of outputs list is :{"o1","o2","o3","o4","o5"}
the last status of objest which I want is : [false,true,false,false,true].
All above style is not much 'ReactX',check object status action do not affect to each other.
How to throw foreach? I trid toIterable(),toList() but failed.
Observable.from(list)
.concatMap(s -> sentYamlAndGet())
.concatMap(this::startPolling)
....
;
Wanted to know if it's good practice to do that and what would be the best way to do that?
Thanks in advance.
pps: currentlly I'm using rxjava1 <version>1.2.0</version> but can change to 2(´▽`)ﾉ

I am new to kafka. The consumer that I am creating is not getting messages that the producer publishes

I am creating consumers (a consumer group with single consumer in it) :
Properties properties = new Properties();
properties.put("zookeeper.connect","localhost:2181");
properties.put("auto.offset.reset", "largest");
properties.put("group.id", groupId);
properties.put("auto.commit.enable", "true");
ConsumerConfig consumerConfig = new ConsumerConfig(properties);
ConsumerConnector consumerConnector = Consumer.createJavaConsumerConnector(consumerConfig);
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumerConnector.createMessageStreams(topicCountMap);
consumerMap.entrySet().stream().forEach(
streams -> {
streams.getValue().stream().forEach(
stream -> {
KafkaBasicConsumer customConsumer = new KafkaBasicConsumer();
try {
Future<?> consumerFuture = kafkaConsumerExecutor.submit(customConsumer);
kafkaConsumersFuture.put(groupId, consumerFuture);
} catch (Exception e) {
logger.error("---- Got error : "+ e.getMessage());
logger.error("Exception : ", e);
}
}
);
}
);
I have subscribed 2 consumers for the same topic.
I am unsubscribing the consumer by storing its future object and then invoking
consumerFuture.cancel(Boolean.TRUE);
Now I subscribe the same consumer again with above code and it gets successfully registered.
However, when the publisher now publishes the newly subscribed consumer is not getting messages whereas the other consumer which was registered is getting messages
I am also checking offsets of consumers, they are getting updated when producer publishes but consumers are not getting messages.
Before producing :
Group Topic Pid Offset logSize Lag
A T1 0 94 94 1
Group Topic Pid Offset logSize Lag
B T1 0 94 94 1
After producing :
Group Topic Pid Offset logSize Lag
A T1 0 95 97 2
Group Topic Pid Offset logSize Lag
B T1 0 94 97 2
I am not able to figure out that if this an issue from producer side (partitions not enough) or if I have created consumer in an incorrect way
Also, I am not able to figure out what is log and lag column means in this.
Let me know if anyone can help or need more details.

I found to solution to my problem, thanks #nautilus for reminding to update.
My main intent was to provide endpoint to subscribe and unsubscribe a consumer in kafka.
Since kafka provides only subscribing and not unsubscribing (only manually possible) I had to write layer over kafka implementation.
I stored the consumer object in a static map with key as group id (since my consumer group can have only one consumer)
Problem was I was not closing consumer once created when unsubscribing and old consumer with same group id was preventing new from getting messages
private static Map kafkaConsumersFuture
Based on some parameter, find out group id
kafkaConsumersFuture.put(groupId, consumerConnector);
And while unsubcribing I did
ConsumerConnector consumerConnector = kafkaConsumersFuture.get(groupId);
if(consumerConnector!=null) {
consumerConnector.shutdown();
kafkaConsumersFuture.remove(groupId);
}