Develop Reference

Handle high number of traditional synchronous/blocking HTTP client requests with few threads Java?

I am working with Java. Another software developer has provided me his code performing synchronous HTTP calls and is responsible of maintaining it - he is using com.google.api.client.http. Updating his code to use an asynchronous HTTP client with a callback is not an available option, and I can't contact the developer to make changes to it. But I still want the efficient asynchronous behaviour of attaching a callback to an HTTP request.
(I am working in Spring Boot and my system is built using RabbitMQ AMQP if it has any effect.)
The simple HTTP GET (it is actually an API call) is performed as follows:
HttpResponse<String> response = httpClient.send(request, BodyHandlers.ofString());
This server I'm communicating with via HTTP takes some time to reply back... say 3-4 seconds. So my thread of execution is blocked for this duration, waiting for a reply. This scales very poorly, my single thread isn't doing is just waiting back for a reply to arrive - this is very heavy.
Sure, I can add the number of threads performing this call if I want to send more HTTP requests concurrently, i.e. I can scale in that way, but this doesn't sound efficient or correct. If possible, I would really like to get a better ratio than 1 thread waiting for 1 HTTP request in this situation.
In other words, I want to send thousands of HTTP requests with 2-3 available threads and handle the response once it arrives; I don't want to incur any significant delay between the execution of each request.
I was wondering: how can I achieve a more scalable solution? How can I handle thousands of this HTTP call per thread? What should I be looking at or do I just have no options and I am asking for the impossible?
EDIT: I guess this is another way to phrase my problem. Assume I have 1000 requests to be sent right now, each will last 3-4 seconds, but only 4-5 available threads of execution on which to send them. I would like to send them all at the same time, but that's not possible; if I manage to send them ALL within the span of 0.5s or less and handle their requests via some callback or something like that, I would consider that a great solution. But I can't switch to an asynchronous HTTP client library.

Using an asynchronous HTTP client is not an available option - I can't change my HTTP client library.
In that case, I think you are stuck with non-scalable synchronous behavior on the client side.
The only work-around I can think of is to run your requests as tasks in an ExecutorService with a bounded thread pool. That will limit the number of threads that are used ... but will also limit the number of simultaneous HTTP requests in play. This is replacing one scaling problem with another one: you are effectively rate-limiting your HTTP requests.
But the flip-side is that launching too many simultaneous HTTP requests is liable to overwhelm the target service(s) and / or the client or server-side network links. From that perspective, client-side rate limiting could be a good thing.
Assume I have 1000 requests to be sent right now, each will last 3-4 seconds, but only 4-5 available threads of execution on which to send them. I would like to send them all at the same time, but that's not possible; if I manage to send them ALL within the span of 0.5s or less and handle their requests via some callback or something like that, I would consider that a great solution. But I can't switch to an asynchronous HTTP client.
The only way you are going to be able to run > N requests at the same time with N threads is to use an asynchronous client. Period.
And "... callback or something like that ...". That's a feature you will only get with an asynchronous client. (Or more precisely, you can only get real asynchronous behavior via callbacks if there is a real asynchronous client library under the hood.)
So the solution is akin to sending the HTTP requests in a staggering manner i.e. some delay between one request and another, where each delay is limited by the number of available threads? If the delay between each request is not significant, I can find that acceptable, but I am assuming it would be a rather large delay between the time each thread is executed as each thread has to wait for each other to finish (3-4s)? In that case, it's not what I want.
With my proposed work-around, the delay between any two requests is difficult to quantify. However, if you are trying to submit a large number of requests at the same time and wait for all of the responses, then the delay between individual requests is not relevant. For that scenario, the relevant measure is the time taken to complete all of the requests. Assuming that nothing else is submitting to the executor, the time taken to complete the requests will be approximately:
nos_requests * average_request_time / nos_worker_threads
The other thing to note is that if you did manage to submit a huge number of requests simultaneously, the server delay of 3-4s per request is liable to increase. The server will only have the capacity to process a certain number of requests per second. If that capacity is exceeded, requests will either be delayed or dropped.
But if there are no other options.
I suppose, you could consider changing your server API so that you can submit multiple "requests" in a single HTTP request.
I think that the real problem here is there is a mismatch between what the server API was designed to support, and what you are trying to do with it.
And there is definitely a problem with this:
Another software developer has provided me his code performing synchronous HTTP calls and is responsible of maintaining it - he is using com.google.api.client.http. Updating his code to use an asynchronous HTTP client with a callback is not an available option, and I can't contact the developer to make changes to it.
Perhaps you need to "bite the bullet" and stop using his code. Work out what it is doing and replace it with your own implementation.
There is no magic pixie dust that will give scalable performance from a synchronous HTTP client. Period.

How to maintain order of tasks for asynchrounous operations using multithreading

I am writing a java application where the central datastructure will be updated both with the request and corresponding response from external systems.
How can I make sure there are no race conditions? Below is how I implemented.
I receive request from GUI and I process it and store in hashmap of hashmap and then forward the request to external system for which I get the response asynchronously. When I receive the response based on some id that I sent earlier, I update the datastructure (hashmap of hashmap)
I created one thread that handles request from GUI and another for handling responses from external system.
I have created 2 linkedblockingqueues - one for requests and another for responses
I am using executor service to create multiple threads for request & response.
How do I make sure things are executed in order ?
This is an order management system and I dont want an amend to be sent before new order is sent.

Use Hastable, it is the synchronized implementation of a Map.
The synchronized implementation will prevent that more than one thread accesses it at the same time.
https://docs.oracle.com/javase/8/docs/api/java/util/Hashtable.html

Request Aggregator / Middle-tier design pattern for costly requests

I'm working on a program that will have multiple threads requiring information from a web-service that can handle requests such as:
"Give me [Var1, Var2, Var3] for [Object1, Object2, ... Object20]"
and the resulting reply will give me a, in this case, 20-node XML (one for each object), each node with 3 sub-nodes (one for each var).
My challenge is that each request made of this web-service costs the organization money and, whether it be for 1 var for 1 object or 20 vars for 20 objects, the cost is the same.
So, that being the case, I'm looking for an architecture that will:
Create a request on each thread as data is required
Have a middle-tier "aggregator" that gets all the requests
Once X number of requests have been aggregated (or a time-limit has reached), the middle-tier performs a single request of the web-service
Middle-tier receives reply from web-service
Middle-tier routes information back to waiting objects
Currently, my thoughts are to use a library such as NetMQ with my middle-tier as a server and each thread as a poller, but I'm getting stuck on the actual implementation and, before going too far down the rabbit-hole, am hoping there's already a design pattern / library out there that does this substantially more efficiently than I'm conceiving of.
Please understand that I'm a noob, and, so, ANY help / guidance would be really greatly appreciated!!
Thanks!!!

Overview
From the architectural point of view, you just sketched out a good approach for the problem:
Insert a proxy between the requesting applications and the remote web service
In the proxy, put the requests in the request queue, until at least one of the following events occurs
The request queue reaches a given length
The oldest request in the request queue reaches a certain age
Group all requests in the request queue in one single request, removing duplicate objects or attributes
Send this request to the remote web service
Move the requests into the (waiting for) response queue
Wait for the response until one of the following occurs
the oldest request in the response queue reaches a certain age (time out)
a response arrives
Get the response (if applicable) and map it to the according requests in the response queue
Answer all requests in the response queue that have an answer
Send a timeout error for all requests older than the timeout limit
Remove all answered requests from the response queue
Technology
You probably won't find an off-the-shelf product or a framework that exactly matches you requirements. But there are several frameworks / architectural patterns that you can use to build a solution.
C#: RX and LINQ
When you want to use C#, you could use reactive extensions for getting the timing and the grouping right.
You could then use LINQ to select the attributes from the requests to build the response and to select the requests in the response queue that either match to a certain part of a response or that timed out.
Scala/Java: Akka
You could model the solution as an actor system, using several actors:
An actor as the gateway for the requests
An actor holding the request queue
An actor sending the request to the remote web service and getting the response back
An actor holding the response queue
An actor sending out the responses or the timeouts
An actor system makes it easy to deal with concurrency and to separate the concerns in a testable way.
When using Scala, you could use its "monadic" collection API (filter, map, flatMap) to do basically the same as with LINQ in the C# approach.
The actor approach really shines when you want to test the individual elements. It is very easy to test each actor individually, without having to mock the whole workflow.
Erlang/Elixir: Actor System
This is similar to the Akka approach, just with a different (functional!) language. Erlang / Elixir has a lot of support for distributed actor systems, so when you need an ultra stable or scalable solution, you should look into this one.
NetMQ / ZeroMQ
This is probably too low level and brings in to few infrastructure. When you use an actor system, you could try to bring in NetMQ / ZeroMQ as the transport system.

Your idea of using a queue looks good to me.
This is one possible solution to your problem and I'm sure there are countless other solutions that can do what you need.
Have a "publish queue" (PQ) and a "consume queue" (CQ)
Clients subscribe to CQ and MT subscribes to PQ
Clients publish the requests to PQ
MT Listens to PQ, aggregates requests and dispatches to farm in a thread
Once the results are back, this thread separates the results into req/res pair
It then publishes the req/res pairs to the CQ
Each client picks the correct message and processes it
Long(er) version:
Have your "middle tier" to listen to a queue (to which, the clients publish messages) and aggregate the requests until N number of requests have come through or X amount of time has passed.
One you are ready, offload the aggregated request to a thread to call your farm and get the results. A bigger problem will most likely arise when you need to communicate this back to the clients.
For that, you probably need another queue that all your clients subscribe to and once your result batch is ready (say 20 responses in XML) from the farm, the thread that called the farm will separate the XML results into their corresponding request/response pair and publish to this queue. Each client will need to pick up the correct request/response pair from the queue and process it.
This will not be a webservice in the traditional sense since the wait times can be prohibitively long and you don't want to maintain a connection which is why I suggest the queue.
You can also have your consumer queue to be topic based, meaning you only publish the req/res pairs to the consumer that asked for it and don't broadcast it (so the client doesn't have to "pick the correct req/res". It will be taken care of based on the topic name). Almost all queues support this.

asynchronous webservice call

I am attempting to make an asynchronous web service call for speed via Java API. I wish to send a payload of over 1000 users and their corresponding attributes. My question is, would making the call multi threaded so that each thread sends a single insert user request be faster than sending all users in one payload?

It depends on how your webservice is organized, mainly, whether it is itself multithreaded.
If it is, sending several simultaneous requests make sense, but chances are optimal number of simultaneous requests is less than 1000. I would start with 10 requests with 100 users each, and then change that parameters to find optimal number of requests.

Since each thread uses the same network port, I would not assume the multi-threading will speed up the sending - on the contrary, I think the overhead of negotiating the usage .

Would a JMS Topic suffice in this situation? Or should I look elsewhere?

There is one controlling entity and several 'worker' entities. The controlling entity requests certain data from the worker entities, which they will fetch and return in their own manner.
Since the controlling entity can agnostic about the worker entities (and the working entities can be added/removed at any point), putting a JMS provider in between them sounds like a good idea. That's the assumption at least.
Since it is an one-to-many relation (controller -> workers), a JMS Topic would be the right solution. But, since the controlling entity is depending on the return values of the workers, request/reply functionality would be nice as well (somewhere, I read about the TopicRequester but I cannot seem to find a working example). Request/reply is typical Queue functionality.
As an attempt to use topics in a request/reply sort-of-way, I created two JMS topis: request and response. The controller publishes to the request topic and is subscribed to the response topic. Every worker is subscribed to the request topic and publishes to the response topic. To match requests and responses the controller will subscribe for each request to the response topic with a filter (using a session id as the value). The messages workers publish to the response topic have the session id associated with them.
Now this does not feel like a solution (rather it uses JMS as a hammer and treats the problem (and some more) as a nail). Is JMS in this situation a solution at all? Or are there other solutions I'm overlooking?

Your approach sort of makes sense to me. I think a messaging system could work. I think using topics are wrong. Take a look at the wiki page for Enterprise Service Bus. It's a little more complicated than you need, but the basic idea for your use case, is that you have a worker that is capable of reading from one queue, doing some processing and adding the processed data back to another queue.
The problem with a topic is that all workers will get the message at the same time and they will all work on it independently. It sounds like you only want one worker at a time working on each request. I think you have it as a topic so different types of workers can also listen to the same queue and only respond to certain requests. For that, you are better off just creating a new queue for each type of work. You could potentially have them in pairs, so you have a work_a_request queue and work_a_response queue. Or if your controller is capable of figuring out the type of response from the data, they can all write to a single response queue.
If you haven't chosen an Message Queue vendor yet, I would recommend RabbitMQ as it's easy to set-up, easy to add new queues (especially dynamically) and has really good spring support (although most major messaging systems have spring support and you may not even be using spring).
I'm also not sure what you are accomplishing the filters. If you ensure the messages to the workers contain all the information needed to do the work and the response messages back contain all the information your controller needs to finish the processing, I don't think you need them.

I would simply use two JMS queues.
The first one is the one that all of the requests go on. The workers will listen to the queue, and process them in their own time, in their own way.
Once complete, they will put bundle the request with the response and put that on another queue for the final process to handle. This way there's no need for the the submitting process to retain the requests, they just follow along with the entire procedure. A final process will listen to the second queue, and handle the request/response pairs appropriately.
If there's no need for the message to be reliable, or if there's no need for the actual processes to span JVMs or machines, then this can all be done with a single process and standard java threading (such as BlockingQueues and ExecutorServices).
If there's a need to accumulate related responses, then you'll need to capture whatever grouping data is necessary and have the Queue 2 listening process accumulate results. Or you can persist the results in a database.
For example, if you know your working set has five elements, you can queue up the requests with that information (1 of 5, 2 of 5, etc.). As each one finishes, the final process can update the database, counting elements. When it sees all of the pieces have been completed (in any order), it marks the result as complete. Later you would have some audit process scan for incomplete jobs that have not finished within some time (perhaps one of the messages erred out), so you can handle them better. Or the original processors can write the request to a separate "this one went bad" queue for mitigation and resubmission.
If you use JMS with transaction, if one of the processors fails, the transaction will roll back and the message will be retained on the queue for processing by one of the surviving processors, so that's another advantage of JMS.
The trick with this kind of processing is to try and push the state with message, or externalize it and send references to the state, thus making each component effectively stateless. This aids scaling and reliability since any component can fail (besides catastrophic JMS failure, naturally), and just pick up where you left off when you get the problem resolved an get them restarted.
If you're in a request/response mode (such as a servlet needing to respond), you can use Servlet 3.0 Async servlets to easily put things on hold, or you can put a local object on a internal map, keyed with the something such as the Session ID, then you Object.wait() in that key. Then, your Queue 2 listener will get the response, finalize the processing, and then use the Session ID (sent with message and retained through out the pipeline) to look up
the object that you're waiting on, then it can simply Object.notify() it to tell the servlet to continue.
Yes, this sticks a thread in the servlet container while waiting, that's why the new async stuff is better, but you work with the hand you're dealt. You can also add a timeout to the Object.wait(), if it times out, the processing took to long so you can gracefully alert the client.
This basically frees you from filters and such, and reply queues, etc. It's pretty simple to set it all up.

Well actual answer should depend upon whether your worker entities are external parties, physical located outside network, time expected for worker entity to finish their work etc..but problem you are trying to solve is one-to-many communication...u added jms protocol in your system just because you want all entities to be able to talk in jms protocol or asynchronous is reason...former reason does not make sense...if it is latter reason, you can choose other communication protocol like one-way web service call.
You can use latest java concurrent APIs to create multi-threaded asynchronous one-way web service call to different worker entities...