I have 2 services running with spring boot and I have the following situation:
Service A (N instances) -> Queue (multiple instances) -> Service B (N instances)
Service A enqueue events async
Service B dequeue events
Traffic is increasing and we've noticed that events spend a lot of time in queue. We need to process events faster. Payload for each event is small and this solution has been working for some years now and a couple of years ago they thought that having a queue was a good idea but now I'm having this performance issue.
I thinking about creating an endpoint in service B and hit this endpoint from service A.
This call should be async and also implement a circuit breaker to avoid lossing messages if B goes down. If B goes down I could use the queue to keep messages and once B is running and up again pull messages from queue.
I have 2 questions:
Is it possible to implement circuit breaker or a failover mechanism for an async call?
do you think there is some other approach which could be better?
Thanks
for question
yes, it is possible, I am assuming you are using spring boot as you have mentioned java in tags, so there is a mechanism called RetryTemplate that you can use for async calls as well, you can find more details here -> https://www.openprogrammer.info/2019/09/03/using-spring-boot-async-with-retry/
There are better approaches I would say, what did you implement when you say queue here? like is it a LIFO or something like AWS SQS? if that is the case then you can try and look for topic-based queue mechanisms like apache kafka
If the bottleneck is in the queue, I don't think you should just remove it. You can just try to use a SQS or other cloud-based queue, which doesn't have that problem.
What happens if service A calls service B directly, and service B is down? The request will be lost. With a queue, the message will remain there until service B recovers.
Circuit breakers are used to avoid overwhelming a failed service with requests (in order to allow it to recover). So when it kicks in, there is a clear data loss there.
Related
Small question regarding Resilience4j please.
Currently, Resilience4J propose a very nice TimeLimiter pattern.
With it, one can very easily configure the time limit on an outbound http call. For instance, my service A is using an http client to call a service B. The time limit is to wait for service B for 1 second.
When properly configured, we can see when service B takes more than 1 second, to return a fallback instead, some kind of "no need to wait for service B anymore".
I was wondering, is it possible to configure similar, but for my own service/handler?
By that, I mean I would like to set a TimeLimiter on my own service A. Since I have an SLA defined by a contract of 5 seconds, instead of telling all my clients ("don't wait for me if I take more than 5 seconds") and to let them configure some kind of TimeLimiter on their end.
Very naive, I put the Resilience4J time limiter annotation on my own handler, and put very long computation in it, but not working. I think the Resilience4J time limiter is more for outbound calls.
Would it be possible to have the same mechanism on my own service A please?
Thank you
Yes, it also possible for your own service A.
But your service must return a CompletableFuture, Mono or Flux so that the TimeLimiter can throw an exception.
In gRPC I would like some more information on the way the server handles requests.
Are requests executed in parallel? Or does the server spawn a new thread for each request, and execute them in parallel? Is there a way to modify this behavior? I understand that in client-streaming rpc's that message order is guaranteed.
If I send Request A followed by Request B to the same RPC, is it guaranteed that A will be executed first before B begins processing? Or are they each their own thread and executed in parallel with no guarantee that A finishes before B.
Ideally I would like to send a request to the server, the server acknowledges receipt of the request, and then the request is added to a queue to be processed sequentially, and returns a response once it's been processed. An approach I was exploring is to use an external task queue (like RabbitMQ) to queue the work done by the service but I want to know if there is a better approach.
Also -- on a somewhat related note -- does gRPC have a native retry counter mechanism? I have a particularly error-prone RPC that may have to retry up to 3 times (with an arbitrary delay between retries) before it is successful. This is something that could be implemented with RabbitMQ as well.
grpc-java passes RPCs to the service using the Executor provided by ServerBuilder.executor(Executor), or a cached thread pool if no executor is provided.
There is no ordering between simultaneous RPCs. RPCs can arrive in any order.
You could use a server-streaming RPC to allow the server to respond twice, once for acknowledgement and once for completion. You can use a oneof in the response message to allow sending the two different responses.
grpc-java as experimental retry support. gRFC A6 describes the support. The configuration is delivered to the client via service config. Retries are disabled by default, so overall you would want something like channelBuilder.defaultServiceConfig(serviceConfig).enableRetry(). You can also reference the hedging example which is very similar to retries.
I'm creating a spring-mvc restful app, and I'm questionning about performance when it gonna be on production.
I found this link about async controllers, but I still have few questions. In general, what happens when 3 clients try to access a page?
Is this async or synchronous that is to say, I) client A will be processed, then B, and then C, like a waiting queue, or II) do they have a different and single thread each?
If I), would I have to make all my controllers aync?
Requests are processed concurrently using a worker pool of threads.
No, you don't need to make anything async. You can get an advantage from them if you have long blocking requests, but unless you understand how and why, don't bother worrying about it.
I have been doing Java for a few years but I have not had much experience with Asynchronous programming.
I am working on an application that makes SOAP web service calls to some Synchronous web services and currently the implementation of my consuming application is Synchronous also ie. my applications threads block while waiting for the response.
I am trying to learn how to handle these SOAP calls in an asynchronous way - just for the hell of it but I have some high-level questions which I cant seem to find any answers to.
I am using CXF but my question is not specifically about CXF or SOAP, but higher-level, in terms of asynchronous application architecture I think.
What I want to know (working thru a scenario) - at a high level - is:
So I have a Thread (A) running in my JVM that makes a call to a remote web service
It registers a callback method and returns a Future
Thread (A) has done its bit and gets returned to its pool once it has returned the Future
The remote web service response returns and Thread (B) gets allocated and calls the callback method (which generally populates the Future with a result I believe)
Q1. I cant get my head off the blocking thread model - if Thread (A) is no longer listening to that network socket then how does the response that comes back from the remote service get allocated Thread (B) - is it simply treated as a new request coming into the server/container which then allocates a thread to service it?
Q2. Closely related to Q1 I imagine: if no Thread has the Future, or handler (with its callback method) on its stack, then how does the response from the remote web service get associated with the callback method it needs to call?
Or, in another way of asking, how does Thread B (now dealing with the response) get given a reference to the Future/Callback object?
Very sorry my question is so long - and thanks to anyone who gave their time to read through it! :)
I don't see why you'd add all this complexity using asynchronous Threading.
The way to design an asynchronous soap service:
You have one service sending out a response to a given client / clients.
Those clients work on the response given asynchronously.
When done, they would call another soap method to return their response.
The response will just be stored in a queue (e.g. a database table), without any extra logic. You'd have a "Worker" Service working on the incoming tasks. If a response is needed again another method on the other remote service would be called. The requests I would store as events in the database, which would later be asynchronously handled by an EventHandler. See
Hexagonal Architecture:
https://www.youtube.com/watch?v=fGaJHEgonKg
Your Q1 and Q2 seem to have more to do with multithreading than they have to do with asynchronous calls.
The magic of asynchronous web service calls is that you don't have to worry about multithreading to handle blocking while waiting for a response.
It's a bit unclear from the question what the specific problem statement is (i.e., what you are hoping to have your application do while blocking or rather than blocking), but here are a couple ways that you could use asynchronous web service calls that will allow you to do other work.
For the following cases, assume that the dispatch() method calls Dispatch.invokeAsync(T msg, AsyncHandler handler) and returns a Future:
1) Dispatch multiple web service requests, so that they run in parallel:
If you have multiple services to consume and they can all execute independently, dispatch them all at once and process the responses when you have received them all.
ArrayList<Future<?>> futures = new ArrayList<Future<?>>();
futures.add(serviceToConsume1.dispatch());
futures.add(serviceToConsume2.dispatch());
futures.add(serviceToConsume3.dispatch());
// now wait until all services return
for(Future f<?> : futures) {
f.get();
}
// now use responses to continue processing
2) Polling:
Future<?> f = serviceToConsume.dispatch();
while(!f.isDone()) {
// do other work here
}
// now use response to continue processing
I use Apache Camel 2.10.0 with spring-ws component to route some (20+) WS/SOAP operations.
The sample code looks like:
from("spring-ws:rootqname:{http://my/name/space}myOperation1?endpointMapping=#endpointMapping")
from("spring-ws:rootqname:{http://my/name/space}myOperation2?endpointMapping=#endpointMapping")
from("spring-ws:rootqname:{http://my/name/space}myOperation3?endpointMapping=#endpointMapping")
Operations normally access several DB and could last up to couple of seconds
It works perfectly, but now I have a new requirement: 3 of the operations must be synchronized.
For example: if client1 calls operation1 1ms before client2 calling operation1, the client1’s call must be finished before starting client2’s one.
Same is valid for 1 client calling 2 different operations.
For example: if client1 calls operation1 1ms before calling operation2, the operation1’s call must be finished before starting operation2’s one. Clients call the WS asynchronously and this cannot be changed
The application is running with WebLogic 10.3.5.
Reducing the container threads to 1 only would affect all operations, thus I was thinking about adding of some custom queue (JMS style) only to these 3 operations.
Do you have any better idea?
It looks all the calling some be put into the queue first, then we can decide which one should be invoked then.
ActiveMQ is easy to setup and would work with Camel very well.
You need to route the request to JMS queue first, the queue itself is the transaction break point, then you consume JMS messages sequentially. You have much more control of the threading and message consuming by using the message pattern