I am working on webcrawler. It is possible to perform many requests (say, 500-1000 per second) without creation thread per each request(I don't mean thread pools, reusing and so on)?
I think what you want here is that a single thread can handle n number of requests simultaneously.
Now that would mean interleaving the steps for handling 2 threads. That would make sense only if there was some "blocking" operation.
Now, one might say, yes we do block. So what I want is
Request 1 is made and I am waiting for the response
Initiate request 2 while waiting for request 1s response is to come response.
Get request 1's response and process it
Get request 2's response and process it.
This would be possible only if HTTP was "asynchronous". Unfortunately it is not.
(An ok read -> http://wiki.answers.com/Q/Why_http_is_asynchronous)
There are some "asynchronous" HTTP clients which do what AJAX in browser does.
It allows the thread initiating the call to continue. The response is provided back in the call back.
Truth is that they have a thread pool which process these calls synchronously. Only it appears asynchronous.
Exampe:
http://hc.apache.org/httpcomponents-asyncclient-dev/index.html
Related
I am working with Java. Another software developer has provided me his code performing synchronous HTTP calls and is responsible of maintaining it - he is using com.google.api.client.http. Updating his code to use an asynchronous HTTP client with a callback is not an available option, and I can't contact the developer to make changes to it. But I still want the efficient asynchronous behaviour of attaching a callback to an HTTP request.
(I am working in Spring Boot and my system is built using RabbitMQ AMQP if it has any effect.)
The simple HTTP GET (it is actually an API call) is performed as follows:
HttpResponse<String> response = httpClient.send(request, BodyHandlers.ofString());
This server I'm communicating with via HTTP takes some time to reply back... say 3-4 seconds. So my thread of execution is blocked for this duration, waiting for a reply. This scales very poorly, my single thread isn't doing is just waiting back for a reply to arrive - this is very heavy.
Sure, I can add the number of threads performing this call if I want to send more HTTP requests concurrently, i.e. I can scale in that way, but this doesn't sound efficient or correct. If possible, I would really like to get a better ratio than 1 thread waiting for 1 HTTP request in this situation.
In other words, I want to send thousands of HTTP requests with 2-3 available threads and handle the response once it arrives; I don't want to incur any significant delay between the execution of each request.
I was wondering: how can I achieve a more scalable solution? How can I handle thousands of this HTTP call per thread? What should I be looking at or do I just have no options and I am asking for the impossible?
EDIT: I guess this is another way to phrase my problem. Assume I have 1000 requests to be sent right now, each will last 3-4 seconds, but only 4-5 available threads of execution on which to send them. I would like to send them all at the same time, but that's not possible; if I manage to send them ALL within the span of 0.5s or less and handle their requests via some callback or something like that, I would consider that a great solution. But I can't switch to an asynchronous HTTP client library.
Using an asynchronous HTTP client is not an available option - I can't change my HTTP client library.
In that case, I think you are stuck with non-scalable synchronous behavior on the client side.
The only work-around I can think of is to run your requests as tasks in an ExecutorService with a bounded thread pool. That will limit the number of threads that are used ... but will also limit the number of simultaneous HTTP requests in play. This is replacing one scaling problem with another one: you are effectively rate-limiting your HTTP requests.
But the flip-side is that launching too many simultaneous HTTP requests is liable to overwhelm the target service(s) and / or the client or server-side network links. From that perspective, client-side rate limiting could be a good thing.
Assume I have 1000 requests to be sent right now, each will last 3-4 seconds, but only 4-5 available threads of execution on which to send them. I would like to send them all at the same time, but that's not possible; if I manage to send them ALL within the span of 0.5s or less and handle their requests via some callback or something like that, I would consider that a great solution. But I can't switch to an asynchronous HTTP client.
The only way you are going to be able to run > N requests at the same time with N threads is to use an asynchronous client. Period.
And "... callback or something like that ...". That's a feature you will only get with an asynchronous client. (Or more precisely, you can only get real asynchronous behavior via callbacks if there is a real asynchronous client library under the hood.)
So the solution is akin to sending the HTTP requests in a staggering manner i.e. some delay between one request and another, where each delay is limited by the number of available threads? If the delay between each request is not significant, I can find that acceptable, but I am assuming it would be a rather large delay between the time each thread is executed as each thread has to wait for each other to finish (3-4s)? In that case, it's not what I want.
With my proposed work-around, the delay between any two requests is difficult to quantify. However, if you are trying to submit a large number of requests at the same time and wait for all of the responses, then the delay between individual requests is not relevant. For that scenario, the relevant measure is the time taken to complete all of the requests. Assuming that nothing else is submitting to the executor, the time taken to complete the requests will be approximately:
nos_requests * average_request_time / nos_worker_threads
The other thing to note is that if you did manage to submit a huge number of requests simultaneously, the server delay of 3-4s per request is liable to increase. The server will only have the capacity to process a certain number of requests per second. If that capacity is exceeded, requests will either be delayed or dropped.
But if there are no other options.
I suppose, you could consider changing your server API so that you can submit multiple "requests" in a single HTTP request.
I think that the real problem here is there is a mismatch between what the server API was designed to support, and what you are trying to do with it.
And there is definitely a problem with this:
Another software developer has provided me his code performing synchronous HTTP calls and is responsible of maintaining it - he is using com.google.api.client.http. Updating his code to use an asynchronous HTTP client with a callback is not an available option, and I can't contact the developer to make changes to it.
Perhaps you need to "bite the bullet" and stop using his code. Work out what it is doing and replace it with your own implementation.
There is no magic pixie dust that will give scalable performance from a synchronous HTTP client. Period.
In Servlet 3.0 the concept of async processing was introduced. So all the books says that eliminates the requirement one thread per request. I have tested that and yeah, it actually works. Now, I have a simple servlet where users initiate an HTTP request, in sync mode. The thread is simply sleeps for 1 second and then replies to the client. When I do a load testing against this mode, the server can handle 4 request per second only.
Now, I change the sync mode to async mode and create a new thread on request an release the original http thread back to pool. Again, the new thread initiated sleeps for 1 second and replies back.That mode however, scales very good, it handles hundreds of requests per second.
Now, the question is, all the books and articles says the server has limited number of recourses so keeping one thread per request is not good. etc. In both scenarios, I have one thread per request, the main difference is in first one it is an http thread and in the second one it is my custom thread. Now the question is, is there anything special about the HTTP threads than my custom threads? After all, in both case we have the one thread per request, why the first one performs bad but the second one does not? I have gone through many documents, books none explains that tricky detail. Can you suggest me something? Thanks
AFAIK, there is no difference between HTTP threads and Asychronous Threads...
You see a performance upgrade is because HTTP threads (mostly) are implemented as a fixed N-size pool of threads; meaning that at last N server requests will be processed concurrently... any other requests will block (or rejected) until one or more threads become free.
In the Asynchronous mode, those N threads are used and released very quickly because the hard work (creating the response object) is being done in another thread(s); allowing you to continue receiving more hit requests.
Depending how you implement the background threads, you will see the performance upgrade. For example, if you also implement your threads as a fixed M-size pool, where M is bigger than N, you will see an increment of M - N requests in processing.
Here are two links which seem to be contradicting each other. I'd sooner trust the docs:
Link 1
Request processing on the server works by default in a synchronous processing mode
Link 2
It already is multithreaded.
My question:
Which is correct. Can it be both synchronous and multithreaded?
Why do the docs say the following?:
in cases where a resource method execution is known to take a long time to compute the result, server-side asynchronous processing model should be used
If the docs are correct, why is the default action synchronous? All requests are asynchronous on client-side javascript by default for user experience, it would make sense then that the default action for server-side should also be asynchronous too.
If the client does not need to serve requests in a specific order, then who cares how "EXPENSIVE" the operation is. Shouldn't all operations simply be asynchronous?
Request processing on the server works by default in a synchronous processing mode
Each request is processed on a separate thread. The request is considered synchronous because that request holds up the thread until the request is finished processing.
It already is multithreaded.
Yes, the server (container) is multi-threaded. For each request that comes in, a thread is taken from the thread pool, and the request is tied to the particular request.
in cases where a resource method execution is known to take a long time to compute the result, server-side asynchronous processing model should be used
Yes, so that we don't hold up the container thread. There are only so many threads in the container thread pool to handle requests. If we are holding them all up with long processing requests, then the container may run out of threads, blocking other requests from coming in. In asynchronous processing, Jersey hands the thread back to the container, and handle the request processing itself in its own thread pool, until the process is complete, then send the response up to the container, where it can send it back to the client.
If the client does not need to serve requests in a specific order, then who cares how "EXPENSIVE" the operation is.
Not really sure what the client has to do with anything here. Or at least in the context of how you're asking the question. Sorry.
Shouldn't all operations simply be asynchronous?
Not necessarily, if all the requests are quick. Though you could make an argument for it, but that would require performance testing, and numbers you can put up against each other and make a decision from there. Every system is different.
In context of Servlet request/response I read somewhere that:
Using a different thread to do work required by a request will, as you expect, allow the response to be sent immediately.
I am just wondering when a Servlet thread is handing over the actual processing to another thread then that means that it does not have the expected response with it at that point of time anyways, then what is the value in sending the immediate but meaningless response back to the browser?
Could someone please give me a valuable usecase for it.
That quote is talking about a scenario where you can return a meaningful response without actually finishing all the work required by the request. For instance you might upload a file to be processed and respond immediately with a processing ID, but pass the processing to another thread. Later on the client could make another request with that ID to find out if processing completed.
An asynchronous servlet scenario would hand off processing to another thread to do the work while blocking the request. But the blocked request would not tie up a servlet request thread during processing like a normal synchronous servlet request.
Suppose you had a single threaded processor and 10 requests were made at the same time. With a synchronous servlet that waited for the processing to finish, you'd have 10 blocked request threads + 1 processor thread. But with an asynchronous servlet, you'd have 0 blocked threads + 1 processor thread. That's a pretty significant gain.
I have a client end (say Customer) that sends request (with RequestID 1) to server end and receives ack for the sent request. My server end (say SomeStore) processes the request 1 and sends to Customer and receives ack (or resends three times). I have another thread listening at Customer. Upon receiving the Customer's listener thread should update HashMap at key 1. All I need is to wait and retrieve this updated value at key 1.
I have a thread from a threadpool to send request and recieve ack on both ends. I see that both threads do the process of sending. I also have a threadpool for listener. After receiving ack, if I make my main thread wait in a while loop, I don't see the listener's update. (Here I cannot make it with wait()). I don't understand this behavior. Shouldn't both threads be working?
I tried changing my implementation and created a separate class upon receiving and synchroned with this.wait() on myHashMap.get(key) and this.notify() on myHashMap.set(key, value). It works a couple of times and not always. My understanding is that it depends on which thread gets the lock first.
How else do I wait and listen at the same time? Maybe I am overseeing something obvious...
It is easy to receive reply instead of ack but my request gets lost in the network. Therefore using ack. I am already using Callable<> for ack. Any idea is appreciated...
I suspect you are not using thread safe access to the map.
If it's not a ConcurrentHashMap and you are not using synchronization, there is no guarentee you will ever see a change in a HashMap.
Instead of using wait/notify and your own threads, I suggest you use ConcurrentHashMap and ExecutorService and add tasks to perform the update. This will ensure you process and see every update.