I have 2 services running with spring boot and I have the following situation:
Service A (N instances) -> Queue (multiple instances) -> Service B (N instances)
Service A enqueue events async
Service B dequeue events
Traffic is increasing and we've noticed that events spend a lot of time in queue. We need to process events faster. Payload for each event is small and this solution has been working for some years now and a couple of years ago they thought that having a queue was a good idea but now I'm having this performance issue.
I thinking about creating an endpoint in service B and hit this endpoint from service A.
This call should be async and also implement a circuit breaker to avoid lossing messages if B goes down. If B goes down I could use the queue to keep messages and once B is running and up again pull messages from queue.
I have 2 questions:
Is it possible to implement circuit breaker or a failover mechanism for an async call?
do you think there is some other approach which could be better?
Thanks
for question
yes, it is possible, I am assuming you are using spring boot as you have mentioned java in tags, so there is a mechanism called RetryTemplate that you can use for async calls as well, you can find more details here -> https://www.openprogrammer.info/2019/09/03/using-spring-boot-async-with-retry/
There are better approaches I would say, what did you implement when you say queue here? like is it a LIFO or something like AWS SQS? if that is the case then you can try and look for topic-based queue mechanisms like apache kafka
If the bottleneck is in the queue, I don't think you should just remove it. You can just try to use a SQS or other cloud-based queue, which doesn't have that problem.
What happens if service A calls service B directly, and service B is down? The request will be lost. With a queue, the message will remain there until service B recovers.
Circuit breakers are used to avoid overwhelming a failed service with requests (in order to allow it to recover). So when it kicks in, there is a clear data loss there.
In gRPC I would like some more information on the way the server handles requests.
Are requests executed in parallel? Or does the server spawn a new thread for each request, and execute them in parallel? Is there a way to modify this behavior? I understand that in client-streaming rpc's that message order is guaranteed.
If I send Request A followed by Request B to the same RPC, is it guaranteed that A will be executed first before B begins processing? Or are they each their own thread and executed in parallel with no guarantee that A finishes before B.
Ideally I would like to send a request to the server, the server acknowledges receipt of the request, and then the request is added to a queue to be processed sequentially, and returns a response once it's been processed. An approach I was exploring is to use an external task queue (like RabbitMQ) to queue the work done by the service but I want to know if there is a better approach.
Also -- on a somewhat related note -- does gRPC have a native retry counter mechanism? I have a particularly error-prone RPC that may have to retry up to 3 times (with an arbitrary delay between retries) before it is successful. This is something that could be implemented with RabbitMQ as well.
grpc-java passes RPCs to the service using the Executor provided by ServerBuilder.executor(Executor), or a cached thread pool if no executor is provided.
There is no ordering between simultaneous RPCs. RPCs can arrive in any order.
You could use a server-streaming RPC to allow the server to respond twice, once for acknowledgement and once for completion. You can use a oneof in the response message to allow sending the two different responses.
grpc-java as experimental retry support. gRFC A6 describes the support. The configuration is delivered to the client via service config. Retries are disabled by default, so overall you would want something like channelBuilder.defaultServiceConfig(serviceConfig).enableRetry(). You can also reference the hedging example which is very similar to retries.
I am using ActiveMQ in order to communicate between servers.
Every server holds one queue in order to send messages and temporaryQueue per each thread in order to receive messages.
When i am using about ~>32 threads i am receiving
Cannot publish to a deleted Destination: temp-queue: xxx
After a while.
When i change from temporaryQueue to "regular" queue everything works perfectly.
(session.createQueue(...) instead of session.createTemporaryQueue())
Why am i getting this error ?
Does it cost more me when i use "regular" queue ?
So implementing request reply using non temporary queues you need some way to correlate the response to a request. You can use the correlation-id header. Since you are not really supposed to create request-unique regular queues, but a fixed set of queues. Like ORDER.CONFIRMATION.RESPONSE or similar. So, it's less costly to use regular queues - IF you reuse them.
To read messages using multiple threads from a common response queue - use a message selector to select JMSCorrelationID header for your particular answer.
However you should be able to use temp queues as well. Your problem is likely some kind of usage issue - but the information provided does not give any clues as it reveals no implementation or analysis.
I would like to implement a distinct thread for each route in apache camel.I do not want to use a thread pool or async as I want my process to remain synchronous.Could I please get a code example for the same in java DSL format.
Each route uses its own thread, unless a route is using the direct component (http://camel.apache.org/direct), which will re-use the caller thread.
For example having 2 routes
from("file:foo").to("bean:blah");
from("jms:queue:bar").to("bean:great")
Is 2 routes, and each route uses its own thread.
On the other hand the following 2 routes
from("file:foo").to("bean:blah").to("direct:bar");
from("direct:bar").to("bean:great")
Then the 2nd route being a direct endpoint, will re-use the caller thread, from the 1st route, when the 1st route routes the message to it, using: .to("direct:bar")
you can use camel-direct to have to a single-threaded, synchronous request/response route...
There is one controlling entity and several 'worker' entities. The controlling entity requests certain data from the worker entities, which they will fetch and return in their own manner.
Since the controlling entity can agnostic about the worker entities (and the working entities can be added/removed at any point), putting a JMS provider in between them sounds like a good idea. That's the assumption at least.
Since it is an one-to-many relation (controller -> workers), a JMS Topic would be the right solution. But, since the controlling entity is depending on the return values of the workers, request/reply functionality would be nice as well (somewhere, I read about the TopicRequester but I cannot seem to find a working example). Request/reply is typical Queue functionality.
As an attempt to use topics in a request/reply sort-of-way, I created two JMS topis: request and response. The controller publishes to the request topic and is subscribed to the response topic. Every worker is subscribed to the request topic and publishes to the response topic. To match requests and responses the controller will subscribe for each request to the response topic with a filter (using a session id as the value). The messages workers publish to the response topic have the session id associated with them.
Now this does not feel like a solution (rather it uses JMS as a hammer and treats the problem (and some more) as a nail). Is JMS in this situation a solution at all? Or are there other solutions I'm overlooking?
Your approach sort of makes sense to me. I think a messaging system could work. I think using topics are wrong. Take a look at the wiki page for Enterprise Service Bus. It's a little more complicated than you need, but the basic idea for your use case, is that you have a worker that is capable of reading from one queue, doing some processing and adding the processed data back to another queue.
The problem with a topic is that all workers will get the message at the same time and they will all work on it independently. It sounds like you only want one worker at a time working on each request. I think you have it as a topic so different types of workers can also listen to the same queue and only respond to certain requests. For that, you are better off just creating a new queue for each type of work. You could potentially have them in pairs, so you have a work_a_request queue and work_a_response queue. Or if your controller is capable of figuring out the type of response from the data, they can all write to a single response queue.
If you haven't chosen an Message Queue vendor yet, I would recommend RabbitMQ as it's easy to set-up, easy to add new queues (especially dynamically) and has really good spring support (although most major messaging systems have spring support and you may not even be using spring).
I'm also not sure what you are accomplishing the filters. If you ensure the messages to the workers contain all the information needed to do the work and the response messages back contain all the information your controller needs to finish the processing, I don't think you need them.
I would simply use two JMS queues.
The first one is the one that all of the requests go on. The workers will listen to the queue, and process them in their own time, in their own way.
Once complete, they will put bundle the request with the response and put that on another queue for the final process to handle. This way there's no need for the the submitting process to retain the requests, they just follow along with the entire procedure. A final process will listen to the second queue, and handle the request/response pairs appropriately.
If there's no need for the message to be reliable, or if there's no need for the actual processes to span JVMs or machines, then this can all be done with a single process and standard java threading (such as BlockingQueues and ExecutorServices).
If there's a need to accumulate related responses, then you'll need to capture whatever grouping data is necessary and have the Queue 2 listening process accumulate results. Or you can persist the results in a database.
For example, if you know your working set has five elements, you can queue up the requests with that information (1 of 5, 2 of 5, etc.). As each one finishes, the final process can update the database, counting elements. When it sees all of the pieces have been completed (in any order), it marks the result as complete. Later you would have some audit process scan for incomplete jobs that have not finished within some time (perhaps one of the messages erred out), so you can handle them better. Or the original processors can write the request to a separate "this one went bad" queue for mitigation and resubmission.
If you use JMS with transaction, if one of the processors fails, the transaction will roll back and the message will be retained on the queue for processing by one of the surviving processors, so that's another advantage of JMS.
The trick with this kind of processing is to try and push the state with message, or externalize it and send references to the state, thus making each component effectively stateless. This aids scaling and reliability since any component can fail (besides catastrophic JMS failure, naturally), and just pick up where you left off when you get the problem resolved an get them restarted.
If you're in a request/response mode (such as a servlet needing to respond), you can use Servlet 3.0 Async servlets to easily put things on hold, or you can put a local object on a internal map, keyed with the something such as the Session ID, then you Object.wait() in that key. Then, your Queue 2 listener will get the response, finalize the processing, and then use the Session ID (sent with message and retained through out the pipeline) to look up
the object that you're waiting on, then it can simply Object.notify() it to tell the servlet to continue.
Yes, this sticks a thread in the servlet container while waiting, that's why the new async stuff is better, but you work with the hand you're dealt. You can also add a timeout to the Object.wait(), if it times out, the processing took to long so you can gracefully alert the client.
This basically frees you from filters and such, and reply queues, etc. It's pretty simple to set it all up.
Well actual answer should depend upon whether your worker entities are external parties, physical located outside network, time expected for worker entity to finish their work etc..but problem you are trying to solve is one-to-many communication...u added jms protocol in your system just because you want all entities to be able to talk in jms protocol or asynchronous is reason...former reason does not make sense...if it is latter reason, you can choose other communication protocol like one-way web service call.
You can use latest java concurrent APIs to create multi-threaded asynchronous one-way web service call to different worker entities...