I have 2 data sources: DB and server. When I start the application, I call the method from the repository (MyRepository):
public Observable<List<MyObj>> fetchMyObjs() {
Observable<List<MyObj>> localData = mLocalDataSource.fetchMyObjs();
Observable<List<MyObj>> remoteData = mRemoteDataSource.fetchMyObjs();
return Observable.concat(localData, remoteData);
}
I subscribe to it as follows:
mMyRepository.fetchMyObjs()
.compose(applySchedulers())
.subscribe(
myObjs -> {
//do somthing
},
throwable -> {
//handle error
}
);
I expect that the data from the database will be loaded faster, and when the download of data from the network is completed, I will simply update the data in Activity.
When the Internet is connected, everything works well. But when we open the application without connecting to the network, then mRemoteDataSource.fetchMyObjs(); throws UnknownHostException and on this all Observable ends (the subscriber for localData does not work (although logs tell that the data from the database was taken)). And when I try to call the fetchMyObjs() method again from the MyRepository class (via SwipeRefresh), the subscriber to localData is triggered.
How can I get rid of the fact that when the network is off, when the application starts, does the subscriber work for localData?
Try some of error handling operators:
https://github.com/ReactiveX/RxJava/wiki/Error-Handling-Operators
I'd guess onErrorResumeNext( ) will be fine but you have to test it by yourself. Maybe something like this would work for you:
Observable<List<MyObj>> remoteData = mRemoteDataSource.fetchMyObjs()
.onErrorResumeNext()
Addidtionally I am not in position to judge if your idea is right or not but maybe it's worth to think about rebuilding this flow. It is not the right thing to ignore errors - that's for sure ;)
You can observe your chain with observeOn(Scheduler scheduler, boolean delayError) and delayError set to true.
delayError - indicates if the onError notification may not cut ahead of onNext notification on the other side of the scheduling boundary. If true a sequence ending in onError will be replayed in the same order as was received from upstream
Related
I am implementing Spring Boot application in Java, using Spring Cloud Stream with Kafka Streams binder.
I need to implement blocking operation inside of KStream map method like so:
public Consumer<KStream<?, ?>> sink() {
return input -> input
.mapValues(value -> methodReturningCompletableFuture(value).get())
.foreach((key, value) -> otherMethod(key, value));
}
completableFuture.get() throws exceptions (InterruptedException, ExecutionException)
How to handle these exceptions so that the chained method doesn't get executed and the Kafka message is not acknowledged? I cannot afford message loss, sending it to a dead letter topic is not an option.
Is there a better way of blocking inside map()?
You can try the branching feature in Kafka Streams to control the execution of the chained methods. For example, here is a pseudo-code that you can try.
You can possibly use this as a starting point and adapt this to your particular use case.
final Map<String, ? extends KStream<?, String>> branches =
input.split()
.branch(k, v) -> {
try {
methodReturningCompletableFuture(value).get();
return true;
}
catch (Exception e) {
return false;
}
}, Branched.as("good-records"))
.defaultBranch();
final KStream<?, String> kStream = branches.get("good-records");
kStream.foreach((key, value) -> otherMethod(key, value));
The idea here is that you will only send the records that didn't throw an exception to the named branch good-records, everything else goes into a default branch which we simply ignore in this pseudo-code. Then you invoke additional chained methods (as this foreach call shows) only for those "good" records.
This does not solve the problem of not acknowledging the message after an exception is thrown. That seems to be a bit challenging. However, I am curious about that use case. When an exception happens and you handle it, why don't you want to ack the message? The requirements seem to be a bit rigid without using a DLT. The ideal solution here is that you might want to introduce some retries and once exhausted from the retries, send the record to a DLT which makes Kafka Streams consumer acknowledges the message. Then the application moves on to the next offset.
The call methodReturningCompletableFuture(value).get() simply waits until a default or configured timeout is reached, assuming that methodReturningCompletableFuture() returns a Future object. Therefore, that is already a good approach to wait inside the KStream map operation. I don't think anything else is necessary to make it wait further.
Been recently experiencing some timeouts with scala.concurrent.Future objects created awaiting processing within an Akka actor and I was wondering how to handle those timeout'd events. Are they really lost? Are they retried and preserved in memory or how does it work?
To put a bit of context, the code goes the following.
List<Future<MyMessage>> futureMessageList = plainMessages.stream()
.map(this::toFuture)
.collect(Collectors.toList());
Futures.sequence(futureMessageList, ExecutionContexts.global())
.onComplete(new OnComplete<Iterable<MyMessage>>() {
#Override
public void onComplete(Throwable throwable, Iterable<MyMessage> messages) {
... // iterate futureMessageList list
Within the onComplete an iteration over futureMessageList takes place, which is basically composed of Future objects which encapsulate MyMessage.
However, the function toFuture does a Patterns.ask() with a given dispatcher and that seems to be taking more than the timeout I sent (60 seconds). Take into account that the response times depend on an underlying system which may be under high load or without the fastest network depending on the environment it runs.
Future<MyMessage> message = Patterns.ask(actorSystem.getSampleDispatcher(), msg, TIMEOUT_60_SECS)
So my question is, after the onComplete throws the following exception due to the Future not being processed in time...
java.lang.NullPointerException
at my.package.Clazz.onComplete(Clazz.java:4)
at my.package.Clazz$1.onComplete(Clazz.java:5)
at akka.dispatch.OnComplete.internal(Future.scala:258)
at akka.dispatch.OnComplete.internal(Future.scala:256)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
at scala.concurrent.impl.CallbackRunnable.run$$$capture(Promise.scala:32)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala)
at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
Are those MyMessage objs saved within memory and retried afterwards? Should I somehow handle the exception and handle those timeout'd messages with an in-memory list or how should I workaround this?
When ask times out from not getting a reply it completes the Future (or CompletionStage) with a failure. The message may still be somewhere being processed and if there is a response it will end up in dead letters (https://doc.akka.io/docs/akka/current/general/message-delivery-reliability.html#dead-letters). Other scenarios where the timeout could hit are if the actor has stopped or crashed processing the message, the request or response got lost (not likely unless the responding actor is remote).
Future.sequence will either complete successfully when all futures passed to it has completed successfully or fail if any of them fails.
This means that if any of the asks time out you will get null as the messages parameter and the exception from the first failing future as the throwable parameter in your onComplete callback.
If you rather would like to get a partial list of results, each being either a successful value or an exception. You can do that with with the help of recover on each future before passing them to Future.sequence.
So here's the situation: I'm implementing the caching of our webapp using vertx-redis (we were formerly using lettuce). Pretty simple mechanism, there is an anotation we use on endpoints which is responsible to invoke the redis-client (whatever implementation we are using) and, if there is cached info for the given key it should be used as response body and the request should be finished with no processing.
But there's this really annoying behavior with the vertx-redis implementation in which ending the request doesn't stop the processing. I make the request, get the quick response since there was cached info, but I can still see in the logs that the app keeps the processing going on, as if the request was still open. I believe that it's because I'm ending the response inside the handler for the Redis client call, like this:
client.get("key", onResponse -> {
if (onResponse.succeeded() && onResponse.result() != null) {
//ending request from here
}
});
I realize that I could maybe reproduce the behavior as it was before if I could do something like this:
String cachedInfo = client.get("key").map(onResponse -> onResponse.result());
// endResponse
But as we know, vertx-redis is a semantic API and every method returns the same instance of RedisClient. I also thought about doing something like this:
private String cachedInfo;
...
client.get("key", onResult -> {
if (onResponse.succeeded()) {
this.cachedInfo = onResponse.result();
}
});
if (cachedInfo != null) { // The value could be unset since the lambda is running in other thread
//end request
}
Really don't know what to do, is there a way to return the contents of the AsyncResult to a variable or maybe set it to a variable synchronously somehow? I've also been searching for ways to somehow stop the whole flow of the current request but couldn't find any satisfactory, non-aggressive solution so far, but I'm really open to this option either.
I'm trying to implement excel export for some amount of data. After 5 minutes I receive a 504 Gateway timeout. In the backend the process continues with its work.
For the whole service to finish, I need approximately 15 minutes. Is there anything I can do to prevent this? I dont have access to the servers in production.
The app is Spring boot with Oracle database. I'm using POI for this export.
One common way to handle these kinds of problems is to have the first request start the process in the background, and when the file has been generated, download the results from another place. The first request finishes immediately, and the user can then check another view to see if the file has been generated, and download the results.
You can export the data in smaller chunks. Run a test with say 10K records, make a note of the id of the last record and repeat the export starting at the next record. If 10K finishes quickly, then try 50K. If you have a timer that might come in handy. Good luck.
I had the same situation where the timeout of the network calls wasn't in our hand, so I guess you have something where it is 5 mins to receive the 1st byte and then the timeout is gone.
My solution was, let's assume you have a controller and a query layer to talk to the database. In this case, you make your process in the Async way. The call to this controller should just trigger that async execution and return the success status immediately, without waiting. Here execution will happen in the background. Futures can be used here as they are async and you can also handle the result once completed by using callback methods of Future.
You can implement using Future and callback methods in java8 like below:
Futures.addCallback(
exportData,
new FutureCallback<String>() {
public void onSuccess(String message) {
System.out.println(message);
}
public void onFailure(Throwable thrown) {
thrown.getCause();
}
},
service)
and in Scala like:
val result = Future {
exportData(data)
}
result.onComplete {
case Success(message) => println(s"Got the callback result:
$message")
case Failure(e) => e.printStackTrace
}
I'm trying to write a non-blocking proxy with netty 4.1. I have a "FrontHandler" which handles incoming connections, and then a "BackHandler" which handles outgoing ones. I'm following the HexDumpProxyHandler (https://github.com/netty/netty/blob/ed4a89082bb29b9e7d869c5d25d6b9ea8fc9d25b/example/src/main/java/io/netty/example/proxy/HexDumpProxyFrontendHandler.java#L67)
In this code I have found:
#Override
public void channelRead(final ChannelHandlerContext ctx, Object msg) {
if (outboundChannel.isActive()) {
outboundChannel.writeAndFlush(msg).addListener(new ChannelFutureListener() {, I've seen:
Meaning that the incoming message is only written if the outbound client connection is already ready. This is obviously not ideal in a HTTP proxy case, so I am thinking what would be the best way to handle it.
I am wondering if disabling auto-read on the front-end connection (and only trigger reads manually once the outgoing client connection is ready) is a good option. I could then enable autoRead over the child socket again, in the "channelActive" event of the backend handler. However, I am not sure about how many messages would I get in the handler for each "read()" invocation (using HttpDecoder, I assume I would get the initial HttpRequest, but I'd really like to avoid getting the subsequent HttpContent / LastHttpContent messages until I manually trigger the read() again and enable autoRead over the channel).
Another option would be to use a Promise to get the Channel from the client ChannelPool:
private void setCurrentBackend(HttpRequest request) {
pool.acquire(request, backendPromise);
backendPromise.addListener((FutureListener<Channel>) future -> {
Channel c = future.get();
if (!currentBackend.compareAndSet(null, c)) {
pool.release(c);
throw new IllegalStateException();
}
});
}
and then do the copying from input to output thru that promise. Eg:
private void handleLastContent(ChannelHandlerContext frontCtx, LastHttpContent lastContent) {
doInBackend(c -> {
c.writeAndFlush(lastContent).addListener((ChannelFutureListener) future -> {
if (future.isSuccess()) {
future.channel().read();
} else {
pool.release(c);
frontCtx.close();
}
});
});
}
private void doInBackend(Consumer<Channel> action) {
Channel c = currentBackend.get();
if (c == null) {
backendPromise.addListener((FutureListener<Channel>) future -> action.accept(future.get()));
} else {
action.accept(c);
}
}
but I'm not sure about how good it is to keep the promise there forever and do all the writes from "front" to "back" by adding listeners to it. I'm also not sure about how to instance the promise so that the operations are performed in the right thread... right now I'm using:
backendPromise = group.next().<Channel> newPromise(); // bad
// or
backendPromise = frontCtx.channel().eventLoop().newPromise(); // OK?
(where group is the same eventLoopGroup as used in the ServerBootstrap of the frontend).
If they're not handled thru the right thread, I assume it could be problematic to have the "else { }" optimization in the "doInBackend" method to avoid using the Promise and write to the channel directly.
The no-autoread approach doesn't work by itself, because the HttpRequestDecoder creates several messages even if only one read() was performed.
I have solved it by using chained CompletableFutures.
I have worked on a similar proxy application based on the MQTT protocol. So it was basically used to create a real-time chat application. The application that I had to design however was asynchronous in nature so I naturally did not face any such problem. Because in case the
outboundChannel.isActive() == false
then I can simply keep the messages in a queue or a persistent DB and then process them once the outboundChannel is up. However, since you are talking about an HTTP application, so this means that the application is synchronous in nature meaning that the client cannot keep on sending packets until the outboundChannel is up and running. So the option you suggest is that the packet will only be read once the channel is active and you can manually handle the message reads by disabling the auto read in ChannelConfig.
However, what I would like to suggest is that you should check if the outboundChannel is active or not. In case the channel is active, send he packet forward for processing. In case the channel is not active, you should reject the packet by sending back a response similar to Error404
Along with this you should configure your client to keep on retrying sending the packets after certain intervals and accordingly handle what needs to be done in case the channel takes too long a time to become active and become readable. Manually handling channelRead is generally not preferred and is an anti pattern. You should let Netty handle that for you in the most efficient way.