I would like to implement a microservice which after receive a request (via message queue) will try to execute it via REST/SOAP calls to the external services. On success the reply should be sent back via MQ, but on failure the request should be rescheduled for the execution later (using some custom algorithm like 10 seconds, 1 minute, 10 minutes, timeout - give up). After specified amount of time the failure message should be sent back to the requester.
It should run on Java 8 and/or Groovy. Event persistence is not required.
First I though about Executor and Runnable/Future together with ScheduledExecutorService.scheduleWithFixedDelay, but it looks to much low level for me. The second idea was actors with Akka and Scheduler (for rescheduling), but I'm sure there could be some other approaches.
Question. What technique would you use for reactive event processing with an ability to reschedule them on failure?
"Event" is quite fuzzy term, but most of definitions I met was talking about one of techniques of Inversion of Control. This one was characterized with fact, that you don't care WHEN and BY WHOM some piece of code will be called, but ON WHAT CONDITION. That means that you invert (or more precisely "lose") control over execution flow.
Now, you want event-driven processing (so you don't want to handle WHEN and BY WHOM), yet you want to specify TIMED (so strictly connected to WHEN) behaviour on failure. This is some kind of paradox to me.
I'd say you would do better, if you'd use callbacks for reactive programming, and on failure you'd just start new thread that will sleep for 10 seconds and re-run callback.
In the end I have found the library async-retry which was written just for this purpose. It allows to asynchronously retry the execution in a very customizable way. Internally it leverages ScheduledExecutorService and CompletableFuture (or ListenableScheduledFuture from Guava when Java 7 has to be used).
Sample usage (from the project web page):
ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();
RetryExecutor executor = new AsyncRetryExecutor(scheduler).
retryOn(SocketException.class).
withExponentialBackoff(500, 2). //500ms times 2 after each retry
withMaxDelay(10_000). //10 seconds
withUniformJitter(). //add between +/- 100 ms randomly
withMaxRetries(20);
final CompletableFuture<Socket> future = executor.getWithRetry(() ->
new Socket("localhost", 8080)
);
future.thenAccept(socket ->
System.out.println("Connected! " + socket)
);
Related
i am working on some performance testing task. The main goal is to compare duration of old NCP protocol calls againts new REST API calls. I have this scenario:
Client has an authenticated session
Client access protected resource
I have to create two variants:
a) One-by-one variant: The question is: How long does it take to perform 2000 requests sent one by one?
b) Concurrent variant: The question is: How long does it take to perform 2000 Request which are sent concurrently (300 Threads ideal)
I dont know the best way to solve this problem. My idea is:
a) Creation of 2000 Http clients -> Each client sends HTTP Post with credentials in body -> Each client sends HTTP GET and get the response (I will measure the time between sending the GET request and getting a response for each iteration and Sum it.
b) Creation of 2000 Httpclients -> Use executor service with fixed thread pool (300) -> each thread will perform sending get request.
Is there any other way? I know that Jmeter is a great tool but i am not sure that this scenario could by performed on Jmeter. Thanks!
For the second variant: you need to determine what is you targeted throughput (TP). 2000 request per hour? Per minute? Per second? Once you get the TP, and a guesstimate for the scenario response time (RT), you could estimate the number of VUsers using the Little's Law. Alternatively, you can use a calculator to determine that number.
Jmeter provides a mechanism to submit this workload (scenarios) by using Arrivals Thread Group. This TG will instantiate the number of threads needed to sustain the targeted TP.
Be aware that there is possibility that you might not reach the TP goal due to:
the SUT does not have the capacity to handle the load
a bottleneck (resource saturation) somewhere in the environment
the client (JMeter) does have enough resources to produce the load
JUnit itself doesn't provide any multithreading logic, you will have to construct the HTTP requests yourself (or with a 3rd-party library like RestAssured) and then execute them using i.e. ExecutorService or jmh and then come up with something for results analysis.
JMeter has everything out of the box so you won't need to write a single line of code, reporting is also included, it might be not that CI friendly as JMeter .jmx scripts are XML but on the other hand you will get nice protocol metrics and ability to correlate increasing load with increasing response time
We are using Akka actors in our application. Actors are invoking services and in some cases those services are calling rest API exposed by third party. Now the third party APIs are taking a very long time to respond. Due to this during peak time , the system through put is getting impacted as threads are blocked as they wait for client API to respond.
Sometimes during peak time , because threads are waiting , messages just keep waiting in akka mail box for long time and are picked up, once blocked threads are made available.
I am looking for some solution where in i can improve the throughput of the system and can free up threads so that actor messages can start processing.
I am thinking to change the rest API call from blocking to non blocking call using Future and creating a dedicated actor for such kind of rest API invocations . Actor will periodically check if future is completed and then sends a completion message and then rest of process can continue.This way the number of blocking threads will reduce and they will be available for actor message processing.
Also would like to have separate execution context for actors which are doing blocking operations. As of now we have global execution context.
Need further inputs on this.
Firstly, you are correct that you should not block when processing a message, so you need to start an asynchronous process and return immediately from the message handler. The Actor can then send another message when the process completes.
You don't say what technology you are using for the REST API calls but libraries like Akka HTTP will return a Future when making an HTTP request so you shouldn't need to add your own Future around it. E.g:
def getStatus(): Future[Status] =
Http().singleRequest(HttpRequest(uri = "http://mysite/status")).flatMap {
res =>
Unmarshal(res).to[Status]
}
If for some reason you are using a synchronous library, wrap the call in Future(blocking{ ??? }) to notify the execution context that it needs to create a new thread. You don't need a separate execution context unless you need to control the scheduling of threads for this operation.
You also don't need to poll the Future, simply add an onComplete handler that sends an internal message back to your actor which then processes the result of the API call.
I just read the Hystrix guide and am trying to wrap my head around how the default circuit breaker and recovery period operate, and then how to customize their behavior.
Obviously, if the circuit is tripped, Hystrix will automatically call the command's getFallBack() method; this much I understand. But what criteria go into making the circuit tripped in the first place? Ideally, I'd like to try hitting a backing service several times (say, a max of 3 attempts) before we consider the service to be offline/unhealthy and trip the circuit breaker. How could I implement this, and where?
But I imagine that if I override the default circuit breaker, I must also override whatever mechanism handles the default recovery period. If a backing service goes down, it could be for any one of several reasons:
There is a network outage between the client and server
The service was deployed with a bug that makes it incapable of returning valid responses to the client
The client was deployed with a bug that makes it incapable of sending valid requests to the server
Some weird, momentary service hiccup (perhaps the service is doing a major garbage collection, etc.)
etc.
In most of these cases, it is not sufficient to have a recovery period that merely waits N seconds and then tries again. If the service has a bug in it, or if someone pulled some network cables in the data center, we will always get failures from this service. Only in a small number of cases will the client-service automagically heal itself without any human interaction.
So I guess my next question is partially "How do I customize the default recovery period strategy?", but I guess it is mainly: "How do I use Hystrix to notify devops when a service is down and requires manual intervention?"
there are basically four reasons for Hystrix to call the fallback method: an exception, a timeout, too many parallel requests, or too many exceptions in the previous calls.
You might want to do a retry in your run() method if the return code or the exception you receive from your service indicate that a retry makes sense.
In your fallback method of the command you might retry when there was a timeout - when there where too many parallel requests or too many exceptions it usually makes no sense to call the same service again.
As also asked how to notify devops: You should connect a monitoring system to Hystrix that polls the status of the circuit breaker and the ratio of successful and unsuccessful calls. You can use the metrics publishers provided, JMX, or write your own adapter using Hystrix' API. I've written two adapters for Riemann and Zabbix in a tutorial I prepared; you'll new very few lines of code for that.
The tutorial also has a example application and a load driver to try out some scenarios.
Br,
Alexander.
I need to call an url and it's possible that call return bad http code like 404, 500 etc. I want to implement a retry feature when I get those errors: a new call will be proceed every hour and ten times maximum.
I use the async-http-client library to do my POST call async.
Do you have any idea?
Thanks in advance for your help.
It's worth considering the Spring Retry functionality.
The API is constructed to be agnostic to what you want to retry, and concerns itself with retry policies, backoffs, limiting number of retries etc.
Another possibility if you're using Java 7/8 is the AsyncRetryExecutor. e.g.
ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();
RetryExecutor executor = new AsyncRetryExecutor(scheduler).
retryOn(SocketException.class).
withExponentialBackoff(500, 2). //500ms times 2 after each retry
withMaxDelay(10_000). //10 seconds
withUniformJitter(). //add between +/- 100 ms randomly
withMaxRetries(20);
I need to wait for a condition in a Spring MVC request handler while I call a third party service to update some entities for a user.
The wait averages about 2 seconds.
I'm calling Thread.sleep to allow the remote call to complete and for the entities to be updated in the database:
Thread.currentThread().sleep(2000);
After this, I retrieve the updated models from the database and display the view.
However, what will be the effect on parallel requests that arrive for processing at this controller/request handler?
Will parallel requests also experience a wait?
Or will they be spawned off into separate threads and so not be affected by the delay experienced by the current request?
What are doing may work sometimes, but it is not a reliable solution.
The Java Future interface, along with a configured ExecutorService allows you to begin some operation and have one or more threads wait until the result is ready (or optionally until a certain amount of time has passed).
You can find documentation for it here:
http://download.oracle.com/javase/6/docs/api/java/util/concurrent/Future.html