In Servlet 3.0 the concept of async processing was introduced. So all the books says that eliminates the requirement one thread per request. I have tested that and yeah, it actually works. Now, I have a simple servlet where users initiate an HTTP request, in sync mode. The thread is simply sleeps for 1 second and then replies to the client. When I do a load testing against this mode, the server can handle 4 request per second only.
Now, I change the sync mode to async mode and create a new thread on request an release the original http thread back to pool. Again, the new thread initiated sleeps for 1 second and replies back.That mode however, scales very good, it handles hundreds of requests per second.
Now, the question is, all the books and articles says the server has limited number of recourses so keeping one thread per request is not good. etc. In both scenarios, I have one thread per request, the main difference is in first one it is an http thread and in the second one it is my custom thread. Now the question is, is there anything special about the HTTP threads than my custom threads? After all, in both case we have the one thread per request, why the first one performs bad but the second one does not? I have gone through many documents, books none explains that tricky detail. Can you suggest me something? Thanks
AFAIK, there is no difference between HTTP threads and Asychronous Threads...
You see a performance upgrade is because HTTP threads (mostly) are implemented as a fixed N-size pool of threads; meaning that at last N server requests will be processed concurrently... any other requests will block (or rejected) until one or more threads become free.
In the Asynchronous mode, those N threads are used and released very quickly because the hard work (creating the response object) is being done in another thread(s); allowing you to continue receiving more hit requests.
Depending how you implement the background threads, you will see the performance upgrade. For example, if you also implement your threads as a fixed M-size pool, where M is bigger than N, you will see an increment of M - N requests in processing.
Related
I have a api(GET REST) developed using java & Quarkus. I want to understand the default mechanism on how this api handle the multiple requests to same api. Is there any queuing mechanism used by default.? is there any multithreading used by default?
Please help to understand this.
Quarkus became popular for it's optimization of resources and benchmarks on heavy loaded systems. It will by default use 2 kind of different threads.
I/O threads or otherwise called event-loop threads
Worker threads
I/O threads or otherwise called event-loop threads. Those threads are responsible, among other things, for reading bytes from the HTTP request and writing bytes back to the HTTP response. The important here is that those threads are usually not blocked at all. You can see a simple of the functionality of those threads as illustrated in the following picture
The number of those I/o threads as described in documentation
The number if IO threads used to perform IO. This will be
automatically set to a reasonable value based on the number of CPU
cores if it is not provided. If this is set to a higher value than the
number of Vert.x event loops then it will be capped at the number of
event loops. In general this should be controlled by setting
quarkus.vertx.event-loops-pool-size, this setting should only be used
if you want to limit the number of HTTP io threads to a smaller number
than the total number of IO threads.
Worker threads. Here a pool of threads again are maintained by the system and the system assigns a worker thread to a execute some scheduled work on some request. Then this thread can be used from another thread to execute some other task. These threads normally take over long running tasks or blocking code.
The default number of these type of threads are 20 if not otherwise configured as indicated by documentation
So to sum up a request in Quarkus will be executed either by some I/O thread or some Worker thread and those threads will be shared between other requests too. An I/O thread will normally take over non blocking tasks that do not take long to be executed. A Worker thread will normally take over blocking tasks and long running processes.
Taking in consideration the above it makes sense that Quarkus will have configured much more Worker threads in the worker thread pool than I/O threads in the i/o thread pool.
What is very important to take from the above information is the following:
A worker thread will serve a specific request (ex request1) and if during this serve it get's blocked to do some I/O operation it will continue to wait for the I/O in order to complete the request it serves. When this request is finished the thread is able to move on and serve some other request (ex request2).
An I/O thread or event-loop thread will serve a specific request (ex request1) and if during this serve it get's blocked for some I/O operation which is needed for this request, it will pause this request, and continue to serve another request (ex request2). When the I/O of first request is completed the thread will return according to some algorithm that schedules the job again to request1 to continue from where it was left.
Now someone may question what is the case then, since usually every request requires some type of I/O operation then how can someone use I/O thread to have better performance. In that case the programmer has 2 choices when he declares the controller of quarkus to use I/O thread:
Spawn manually inside the controller method which is declared to be I/O some other thread to do the blocking code block work while the outer thread that serves the request is of type I/O (read http request data, write http response). The manual thread can be of type worker inside some service layer. This is a bit complicated approach.
Use some external library for I/O operations that's expected to work with the same approach that I/O threads work in quarkus. For example for database operations the I/O could be operated by the library hibernate-reactive. This way full benefits of I/O approach can be achieved.
Some side notes
Considering that we are in the java ecosystem it will be very useful to also mention that the above architecture and efficiency of resources is similar (not exactly same) with Spring Reactive (Web Flux).
But quarkus is based on Jax-Rs and will by default provide this architecture of efficient use of resources, independently of whether you write reactive code or not. When using Spring Boot however in order to have a similar efficiency with quarkus you have to use Spring Reactive (Web Flux).
In case you use the basic spring boot web, the architecture used will be of a single thread per incoming request. A specific thread in this case is not able to switch between different threads. It will need to complete some request in order to handle the next request.
Also in quarkus making a controller method execute from an I/O thread is as simple as placing an annotation #NonBlocking in that method. The same for an endpoint method that needs to be executed from a worker thread with #Blocking.
In Spring boot however switching from those 2 type of threads may mean switching from spring-boot-web to spring-boot-webflux and vice versa. Spring-boot-web has some support however now with servlet-3 to optimize it's approach, article with such optimization, but this requires some programming optimization and not an out of the box functionality.
Here are two links which seem to be contradicting each other. I'd sooner trust the docs:
Link 1
Request processing on the server works by default in a synchronous processing mode
Link 2
It already is multithreaded.
My question:
Which is correct. Can it be both synchronous and multithreaded?
Why do the docs say the following?:
in cases where a resource method execution is known to take a long time to compute the result, server-side asynchronous processing model should be used
If the docs are correct, why is the default action synchronous? All requests are asynchronous on client-side javascript by default for user experience, it would make sense then that the default action for server-side should also be asynchronous too.
If the client does not need to serve requests in a specific order, then who cares how "EXPENSIVE" the operation is. Shouldn't all operations simply be asynchronous?
Request processing on the server works by default in a synchronous processing mode
Each request is processed on a separate thread. The request is considered synchronous because that request holds up the thread until the request is finished processing.
It already is multithreaded.
Yes, the server (container) is multi-threaded. For each request that comes in, a thread is taken from the thread pool, and the request is tied to the particular request.
in cases where a resource method execution is known to take a long time to compute the result, server-side asynchronous processing model should be used
Yes, so that we don't hold up the container thread. There are only so many threads in the container thread pool to handle requests. If we are holding them all up with long processing requests, then the container may run out of threads, blocking other requests from coming in. In asynchronous processing, Jersey hands the thread back to the container, and handle the request processing itself in its own thread pool, until the process is complete, then send the response up to the container, where it can send it back to the client.
If the client does not need to serve requests in a specific order, then who cares how "EXPENSIVE" the operation is.
Not really sure what the client has to do with anything here. Or at least in the context of how you're asking the question. Sorry.
Shouldn't all operations simply be asynchronous?
Not necessarily, if all the requests are quick. Though you could make an argument for it, but that would require performance testing, and numbers you can put up against each other and make a decision from there. Every system is different.
I am working on webcrawler. It is possible to perform many requests (say, 500-1000 per second) without creation thread per each request(I don't mean thread pools, reusing and so on)?
I think what you want here is that a single thread can handle n number of requests simultaneously.
Now that would mean interleaving the steps for handling 2 threads. That would make sense only if there was some "blocking" operation.
Now, one might say, yes we do block. So what I want is
Request 1 is made and I am waiting for the response
Initiate request 2 while waiting for request 1s response is to come response.
Get request 1's response and process it
Get request 2's response and process it.
This would be possible only if HTTP was "asynchronous". Unfortunately it is not.
(An ok read -> http://wiki.answers.com/Q/Why_http_is_asynchronous)
There are some "asynchronous" HTTP clients which do what AJAX in browser does.
It allows the thread initiating the call to continue. The response is provided back in the call back.
Truth is that they have a thread pool which process these calls synchronously. Only it appears asynchronous.
Exampe:
http://hc.apache.org/httpcomponents-asyncclient-dev/index.html
Because of browser compatibility issues, I have decided to use long polling for a real time syncing and notification system. I use Java on the backend and all of the examples I've found thus far have been PHP. They tend to use while loops and a sleep method. How do I replicate this sort of thing in Java? There is a Thread.sleep() method, which leads me to...should I be using a separate thread for each user issuing a poll? If I don't use a separate thread, will the polling requests be blocking up the server?
[Update]
First of all, yes it is certainly possible to do a straightforward, long polling request handler. The request comes in to the server, then in your handler you loop or block until the information you need is available, then you end the loop and provide the information. Just realize that for each long polling client, yes you will be tying up a thread. This may be fine and perhaps this is the way you should start. However - if your web server is becoming so popular that the sheer number of blocking threads is becoming a performance problem, consider an asynchronous solution where you can keep a large numbers of client requests pending - their request is blocking, that is not responding until there is useful data, without tying up one or more threads per client.
[original]
The servlet 3.0 spec provides a standard for doing this kind asynchronous processing. Google "servlet 3.0 async". Tomcat 7 supports this. I'm guessing Jetty does also, but I have not used it.
Basically in your servlet request handler, when you realize you need to do some "long" polling, you can call a method to create an asynchronous context. Then you can exit the request handler and your thread is freed up, however the client is still blocking on the request. There is no need for any sleep or wait.
The trick is storing the async context somewhere "convenient". Then something happens in your app and you want to push data to the client, you go find that context, get the response object from it, write your content and invoke complete. The response is sent back to the client without you having to tie up a thread for each client.
Not sure this is the best solution for what you want but usually if you want to do this at period intervals in java you use the ScheduleExecutorService. There is a good example at the top of the API document. The TimeUnit is a great enum as you can specify the period time easily and clearly. So you can specify it to run every x minutes, hours etc
A little help please.
I am designing a stateless server that will have the following functionality:
Client submits a job to the server.
Client is blocked while the server tries to perform the job.
The server will spawn one or multiple threads to perform the job.
The job either finishes, times out or fails.
The appropriate response (based on the outcome) is created, the client is unblocked and the response is handed off to the client.
Here is what I have thought of so far.
Client submits a job to the server.
The server assigns an ID to the job, places the job on a Queue and then places the Client on an another queue (where it will be blocked).
Have a thread pool that will execute the job, fetch the result and appropriately create the response.
Based on ID, pick the client out of the queue (thereby unblocking it), give it the response and send it off.
Steps 1,3,4 seems quite straight forward however any ideas about how to put the client in a queue and then block it. Also, any pointers that would help me design this puppy would be appreciated.
Cheers
Why do you need to block the client? Seems like it would be easier to return (almost) immediately (after performing initial validation, if any) and give client a unique ID for a given job. Client would then be able to either poll using said ID or, perhaps, provide a callback.
Blocking means you're holding on to a socket which obviously limits the upper number of clients you can serve simultaneously. If that's not a concern for your scenario and you absolutely need to block (perhaps you have no control over client code and can't make them poll?), there's little sense in spawning threads to perform the job unless you can actually separate it into parallel tasks. The only "queue" in that case would be the one held by common thread pool. The workflow would basically be:
Create a thread pool (such as ThreadPoolExecutor)
For each client request:
If you have any parts of the job that you can execute in parallel, delegate them to the pool.
And / or do them in the current thread.
Wait until pooled job parts complete (if applicable).
Return results to client.
Shutdown the thread pool.
No IDs are needed per se; though you may need to use some sort of latch for 2.1 / 2.3 above.
Timeouts may be a tad tricky. If you need them to be more or less precise you'll have to keep your main thread (the one that received client request) free from work and have it signal submitted job parts (by flipping a flag) when timeout is reached and return immediately. You'll have to check said flag periodically and terminate your execution once it's flipped; pool will then reclaim the thread.
How are you communicating to the client?
I recommend you create an object to represent each job which holds job parameters and the socket (or other communication mechanism) to reach the client. The thread pool will then send the response to unblock the client at the end of job processing.
The timeouts will be somewhat tricky, and will have hidden gotcha's but the basic design would seem to be to straightforward, write a class that takes a Socket in the constructor. on socket.accept we just do a new socket processing instantiation, with great foresight and planning on scalability or if this is a bench-test-experiment, then the socket processing class just goes to the data processing stuff and when it returns you have some sort of boolean or numeric for the state or something, handy place for null btw, and ether writes the success to the Output Stream from the socket or informs client of a timeout or whatever your business needs are
If you have to have a scalable, effective design for long-running heavy-haulers, go directly to nio ... hand coded one-off solutions like I describe probably won't scale well but would provide fundamental conceptualizing basis for an nio design of code-correct work.
( sorry folks, I think directly in code - design patterns are then applied to the code after it is working. What does not hold up gets reworked then, not before )