How can you create a failsafe Spring XD stream, which will keep running properly after an exception is triggered for one specific message (i.e.logs the error but continues consuming the next messages in the stream), without having to add try catch(Throwable) in every Stream step?
Is there any easy way of doing this with the Reactor or RxJava model?
Example stream using Reactor:
#Override
public Publisher<Tuple> process(Stream<GenericMessage> inputStream) {
return inputStream
.flatMap(SomeClass::someFlatMap)
.filter(SomeClass::someFilter)
.when(Throwable.class, t -> log.error("error", t));
}
RxJava can be used by a processor module. On creation the subscription needs to be created and to handle errors the subscriber needs to add an onError handler:
subject = new SerializedSubject(PublishSubject.create());
Observable<?> outputStream = processor.process(subject);
subscription = outputStream.subscribe(new Action1<Object>() {
#Override
public void call(Object outputObject) {
if (ClassUtils.isAssignable(Message.class, outputObject.getClass())) {
getOutputChannel().send((Message) outputObject);
} else {
getOutputChannel().send(MessageBuilder.withPayload(outputObject).build());
}
}
}, new Action1<Throwable>() {
#Override
public void call(Throwable throwable) {
logger.error(throwable.getMessage(), throwable);
}
}, new Action0() {
#Override
public void call() {
logger.error("Subscription close for [" + subscription + "]");
}
});
Look at more examples here: https://github.com/spring-projects/spring-xd/tree/master/spring-xd-rxjava/src
Related
I am using executeBlocking from vertx in a for loop to parallelise the processing of a result, and collating the results using a CompositeFuture. Based on all the results, I want to return some value from the method, but lambda function inside CompositeFuture's handler is not letting me do so. How to work with this usecase?
Code for reference:
public Response call() throws Exception {
List<Future> futureList = new ArrayList<>();
//services is a global variable, an arraylist of services
for(String service: services) {
Future serviceFuture = Future.future();
futureList.add(serviceFuture);
vertx.executeBlocking(implementTask(service, serviceFuture), false, asyncResult -> {
if (asyncResult.failed()) {
LOGGER.error("Task FAILED for service {}", service, asyncResult.cause());
}
});
}
CompositeFuture.all(futureList).setHandler(asyncResult -> {
if(asyncResult.succeeded()) {
LOGGER.debug("Task completed successfully");
return new Response(ResponseStatus.SUCCESS);
} else {
LOGGER.error("Task FAILED", asyncResult.cause());
return new Response(ResponseStatus.FAILED);
}
});
}
You can't do this.
Your call() method should return Future<Result> and not the Result. Then you would need to attach the callback handler on your original caller. This is the way that async methods propagate the result in Vert.x.
public Future<Response> call() throws Exception {
Promise<Response> promise = Promise.promise();
List<Future> futureList = new ArrayList<>();
//services is a global variable, an arraylist of services
for(String service: services) {
Future serviceFuture = Future.future();
futureList.add(serviceFuture);
vertx.executeBlocking(implementTask(service, serviceFuture), false, asyncResult -> {
if (asyncResult.failed()) {
LOGGER.error("Task FAILED for service {}", service, asyncResult.cause());
}
});
}
CompositeFuture.all(futureList).setHandler(asyncResult -> {
if(asyncResult.succeeded()) {
LOGGER.debug("Task completed successfully");
promise.complete(new Response(ResponseStatus.SUCCESS));
} else {
LOGGER.error("Task FAILED", asyncResult.cause());
promise.fail("Failed");
}
});
return promise.future();
}
Then the call would look like this:
object.call().onSuccess(resultHandler - > {
//Code when call succeeded
}).onFailure(failureHandler - > {
//Code when call failed
});
Note: This example is using Vert.x 4. If you are using a version of Vert.x less than 4, then the syntax is a bit different, but the point stays the same.
I am using #StreamListener (spring cloud stream) to consume messages from a topic (input-channel), do some processing and save into some cache or database.
My requirement is, if DB goes down while processing the consumed message, I want to pause the main consumer(input-channel), and start consuming from another TOPIC (INPUT56-CHANNEL), and as soon as It consume all the message (doesn't have many) from INPUT56-CHANNEL, I want to resume the main consumer (input-channel) again.
Can that be achieved??
#StreamListener is deprecated; you should convert to the functional programming model instead.
Here is an example using that model (but the same techniques apply to the deprecated listeners).
spring.cloud.function.definition=input1;input2
spring.cloud.stream.bindings.input1-in-0.group=grp1
spring.cloud.stream.bindings.input2-in-0.consumer.auto-startup=false
spring.cloud.stream.bindings.input2-in-0.group=grp2
spring.cloud.stream.kafka.bindings.input2-in-0.consumer.idle-event-interval=5000
#SpringBootApplication
public class So69726610Application {
public static void main(String[] args) {
SpringApplication.run(So69726610Application.class, args);
}
boolean dbIsDown = true;
#Autowired
BindingsLifecycleController controller;
TaskExecutor exec = new SimpleAsyncTaskExecutor();
#Bean
public Consumer<String> input1() {
return str -> {
System.out.println(str);
if (this.dbIsDown) {
this.controller.changeState("input1-in-0", State.PAUSED);
this.controller.changeState("input2-in-0", State.STARTED);
throw new RuntimeException("Paused");
}
};
}
#Bean
public Consumer<String> input2() {
return System.out::println;
}
#EventListener
public void idle(ListenerContainerIdleEvent event) {
System.out.println(event);
// assumes concurrency = 1 (default)
if (event.getListenerId().contains("input2-in-0")) {
this.controller.changeState("input1-in-0", State.RESUMED);
this.exec.execute(() -> this.controller.changeState("input2-in-0", State.STOPPED));
}
}
}
We have a Spring Integration DSL pipeline connected to a GCP Pubsub and things "work": The data is received and processed as defined in the pipeline, using a collection of Function implementations and .handle().
The problem we have (and why I used "work" in quotes) is that, in some handlers, when some of the data isn't found in the companion database, we raise IllegalStateException, which forces the data to be reprocessed (along the way, another service may complete the companion database and then function will now work). This exception is never shown anywhere.
We tried to capture the content of errorHandler, but we really can't find the proper way of doing it programmatically (no XML).
Our Functions have something like this:
Record record = recordRepository.findById(incomingData).orElseThrow(() -> new IllegalStateException("Missing information: " + incomingData));
This IllegalStateException is the one that is not appearing anywhere in the logs.
Also, maybe it's worth mentioning that we have our channels defined as
#Bean
public DirectChannel cardInputChannel() {
return new DirectChannel();
}
#Bean
public PubSubInboundChannelAdapter cardChannelAdapter(
#Qualifier("cardInputChannel") MessageChannel inputChannel,
PubSubTemplate pubSubTemplate) {
PubSubInboundChannelAdapter adapter = new PubSubInboundChannelAdapter(pubSubTemplate, SUBSCRIPTION_NAME);
adapter.setOutputChannel(inputChannel);
adapter.setAckMode(AckMode.AUTO);
adapter.setPayloadType(CardDto.class);
return adapter;
}
I am not familiar with the adapter, but I just looked at the code and it looks like they just nack the message and don't log anything.
You can add an Advice to the handler's endpoint to capture and log the exception
.handle(..., e -> e.advice(exceptionLoggingAdvice)
#Bean
public MethodInterceptor exceptionLoggingAdvice() {
return invocation -> {
try {
return invocation.proceed();
}
catch (Exception thrown) {
// log it
throw thrown;
}
}
}
EDIT
#SpringBootApplication
public class So57224614Application {
public static void main(String[] args) {
SpringApplication.run(So57224614Application.class, args);
}
#Bean
public IntegrationFlow flow(MethodInterceptor myAdvice) {
return IntegrationFlows.from(() -> "foo", endpoint -> endpoint.poller(Pollers.fixedDelay(5000)))
.handle("crasher", "crash", endpoint -> endpoint.advice(myAdvice))
.get();
}
#Bean
public MethodInterceptor myAdvice() {
return invocation -> {
try {
return invocation.proceed();
}
catch (Exception e) {
System.out.println("Failed with " + e.getMessage());
throw e;
}
};
}
}
#Component
class Crasher {
public void crash(Message<?> msg) {
throw new RuntimeException("test");
}
}
and
Failed with nested exception is java.lang.RuntimeException: test
i'm having a problem with rx java.
I have a current stream that in some point gives to me an Either
That response has external resources, like image urls, and i want to send each url to an external class, download it asyncronously, and if all of them are ok, continue with that either received or if one of that resources fails while is being downloaded return an Either.error(MyError());
My problem is that as i'm creating a new observable inside the resources provider, it needs to be subscribed to start run, but i do not know how can i do.
This is my current code (not sure if compiles but you get the idea):
private Observable<Either<Error, Response>> prefetchResourcesOrError(final Either<Error, Response> errorOrResponse) {
if (errorOrResponse.isResponseWithImages()) {
ResponseImages responseImages = (ResponseImages) responseImages.getResponse();
return
Observable.fromIterable(responseImages.getResources()
.map(resourcesProvider::prefetch)
.onErrorReturn(throwable -> Observable.<Either<Error, Response>>just(Either.left(new MyError())))
.map(observable -> errorOrResponse);
} else {
return Observable.just(errorOrResponse);
}
}
//Resource prefetch method
Observable prefetch(Resource resource) {
return Observable.just(resource)
.flatMap((Function<Resource, ObservableSource<?>>) res1 ->
Observable.create((ObservableOnSubscribe<Void>) emitter ->
resourceLoader.prefetch(res1.getUrl(), new ImageLoaderListenerAdapter() {
#Override
public void onException(Exception e) {
emitter.onError(e);
}
#Override
public void onResourceReady() {
emitter.onNext(null);
}
})
)
);
}
}
//The main Stream
//MainObservable is an Either<Error, Response> errorOrResponse
return mainObservable.flatMap(this::prefetchResourcesOrError);
I'm forwarding some real-time data to web clients using chunked encoding.
In the following way, I could make HTTP streaming responses,
public Result action() {
return ok(new StringChunks() {
public void onReady(final Out<String> out) {
openStream().onData(new Consumer<String>() {
public void accept(String string) {
out.write(string);
}
}
}
}
}
But I need to clean up some resources after the client has disconnected. (by e.g. closing the browser window or when the stream reaches EOF)
When using WebSocket I could detect client disconnection using Iteratee.mapDone.
Is there an equivalent method to detect it when using Chunks?
Thanks
Well, just figured it out.
Results.Chunked.Out<A> object has onDisconnected(Callback0) method that I can register a disconnection callback. so
public Result action() {
return ok(new StringChunks() {
public void onReady(final Out<String> out) {
out.onDisconnected(new F.Callback0() {
public void invoke() throws Throwable {
// clean up things ...
}
}
openStream().onData(new Consumer<String>() {
public void accept(String string) {
out.write(string);
}
}
}
}
}