Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I have been thinking of different approaches to achieve asynchronous processing in Java. I thought of a few possibilities and wanted to get your opinion on whether some of them are better than other, and maybe get further suggestions on how this can be done. The most common use-case that comes to mind is sending packets of data over a connection (e.g. TCP) with the following API void sendData(Data data) or even void sendData(Data data, Handler handler). Here are a few ideas I had:
Dedicated Data sending loop - have a didicated thread, that behaves a bit like Event Dispatch Thread in Java, that all other threads call and submit a request. These requests are stored in a queue and periodically that queu is drained and all requests are sent.
Use background thread to drain the queue - the Connection could maintain the list of pending requests, and have a background thread to drain the queue (with some synchronization)
Executor service - pass the request to the service and call the Handler from the background thread.
Asynchronous channel - most high-level approach, delegate over to the implementation
Is any of those better to use, or if you have any other general ideas in mind, please share your comments.
Most of your suggestions are actually different ways of dressing up the same thing.
Behind the scenes the executor service has a pool of 1 or more background threads draining a queue. Requests are submitted to it.
1 and 2 are both ways of saying "queue up stuff to do, have a thread that processes it"
So basically 1 and 2 are both variants of each other. They are also a subset of 3.
For 4. I'm not sure what you mean here?
An ExecutorService is designed to do exactly what you are trying to do - if you have more than one thread to execute the tasks then it's clearly the way to do it.
If you only have a single thread you could still use an ExecutorService but the choice is less clear. It would be reasonably simple just to use a thread and a BlockingQueue so that might be the way to go.
I would always use an executor service to do concurrency. They are high level enough to hide the complexity of managing threads and allow thread reuse. You could either submit tasks to the executor, which will effectively act as a queue, or have many tasks running that use a synchronized queue (like a blocking queue) to share data. The latter might allow more flexibility, e.g., batching of queue items.
I also highly recommend the Guava ListenableFuture as it might solve a lot of problems you might run into when working with concurrency.
You should clearly design your asynchronous pipeline as a graph with data/event dependencies. Typical graph consists of two kinds of nodes:
fast handler:
immediately invoked when an event happens and:
probably stores the event,
and/or calls another fast handler,
and/or submits an asynchronous task to an executor
task:
runs and issues events (that is, calls fast handlers)
So basically you need to develop two independent layers: fast handlers and task executors. Executors are universal and can be taken from java.util.concurrent package. Fast handlers greatly depend on your problem domain and there is no universal library for all cases. For example, a pure queue is a fast handler which only stores events, and so is almost useless.
If you work with I/O, you need to employ standard I/O libraries so that that they issue I/O events for your handlers. It can be build using threads for synchronous I/O, or with Selector threads or Asynchronous channels for async I/O.
Fast handler example used with NIO2 Asynchronous channels:
class ConnectionAcceptor implements CompletionHandler<AsynchronousSocketChannel, Void>{
AsynchronousServerSocketChannel assc;
int maxConn;// max number of simultaneous connections
int connCount=0;
/* called on creation to start listening incoming client connection requests */
void allowAccept() {
assc.accept(null, this);
}
/* called by I/O layer when a client connection requested */
public synchronized void completed(AsynchronousSocketChannel result, Void attachment) {
executor.exec(new Connection(result));
connCount++;
if (connCount<maxConn) {
allowAccept();
}
}
/* called by Connection when it is closed */
synchronized void connClosed() {
if (connCount==maxConn) {
allowAccept();
}
connCount--;
}
}
This handler has 3 entry points and handles 3 kinds of events. Similary, handler to make reading or writing (but not both) can be created. Instead of connCount, its internal state contains a boolean flag indicating that an I/O operation is in progress, and a queue for buffers waiting for AsynchronousSocketChannel to end started operation.
Related
I have a api(GET REST) developed using java & Quarkus. I want to understand the default mechanism on how this api handle the multiple requests to same api. Is there any queuing mechanism used by default.? is there any multithreading used by default?
Please help to understand this.
Quarkus became popular for it's optimization of resources and benchmarks on heavy loaded systems. It will by default use 2 kind of different threads.
I/O threads or otherwise called event-loop threads
Worker threads
I/O threads or otherwise called event-loop threads. Those threads are responsible, among other things, for reading bytes from the HTTP request and writing bytes back to the HTTP response. The important here is that those threads are usually not blocked at all. You can see a simple of the functionality of those threads as illustrated in the following picture
The number of those I/o threads as described in documentation
The number if IO threads used to perform IO. This will be
automatically set to a reasonable value based on the number of CPU
cores if it is not provided. If this is set to a higher value than the
number of Vert.x event loops then it will be capped at the number of
event loops. In general this should be controlled by setting
quarkus.vertx.event-loops-pool-size, this setting should only be used
if you want to limit the number of HTTP io threads to a smaller number
than the total number of IO threads.
Worker threads. Here a pool of threads again are maintained by the system and the system assigns a worker thread to a execute some scheduled work on some request. Then this thread can be used from another thread to execute some other task. These threads normally take over long running tasks or blocking code.
The default number of these type of threads are 20 if not otherwise configured as indicated by documentation
So to sum up a request in Quarkus will be executed either by some I/O thread or some Worker thread and those threads will be shared between other requests too. An I/O thread will normally take over non blocking tasks that do not take long to be executed. A Worker thread will normally take over blocking tasks and long running processes.
Taking in consideration the above it makes sense that Quarkus will have configured much more Worker threads in the worker thread pool than I/O threads in the i/o thread pool.
What is very important to take from the above information is the following:
A worker thread will serve a specific request (ex request1) and if during this serve it get's blocked to do some I/O operation it will continue to wait for the I/O in order to complete the request it serves. When this request is finished the thread is able to move on and serve some other request (ex request2).
An I/O thread or event-loop thread will serve a specific request (ex request1) and if during this serve it get's blocked for some I/O operation which is needed for this request, it will pause this request, and continue to serve another request (ex request2). When the I/O of first request is completed the thread will return according to some algorithm that schedules the job again to request1 to continue from where it was left.
Now someone may question what is the case then, since usually every request requires some type of I/O operation then how can someone use I/O thread to have better performance. In that case the programmer has 2 choices when he declares the controller of quarkus to use I/O thread:
Spawn manually inside the controller method which is declared to be I/O some other thread to do the blocking code block work while the outer thread that serves the request is of type I/O (read http request data, write http response). The manual thread can be of type worker inside some service layer. This is a bit complicated approach.
Use some external library for I/O operations that's expected to work with the same approach that I/O threads work in quarkus. For example for database operations the I/O could be operated by the library hibernate-reactive. This way full benefits of I/O approach can be achieved.
Some side notes
Considering that we are in the java ecosystem it will be very useful to also mention that the above architecture and efficiency of resources is similar (not exactly same) with Spring Reactive (Web Flux).
But quarkus is based on Jax-Rs and will by default provide this architecture of efficient use of resources, independently of whether you write reactive code or not. When using Spring Boot however in order to have a similar efficiency with quarkus you have to use Spring Reactive (Web Flux).
In case you use the basic spring boot web, the architecture used will be of a single thread per incoming request. A specific thread in this case is not able to switch between different threads. It will need to complete some request in order to handle the next request.
Also in quarkus making a controller method execute from an I/O thread is as simple as placing an annotation #NonBlocking in that method. The same for an endpoint method that needs to be executed from a worker thread with #Blocking.
In Spring boot however switching from those 2 type of threads may mean switching from spring-boot-web to spring-boot-webflux and vice versa. Spring-boot-web has some support however now with servlet-3 to optimize it's approach, article with such optimization, but this requires some programming optimization and not an out of the box functionality.
I am new to this concept and want to have a great understanding of this topic.
To make my point clear I want to take an analogy.
Let's take a scenario of Node JS which is single-threaded and provide fast IO operation using an event loop. Now that makes sense since It is single-threaded and is not blocked for any task.
While studying reactive programming in Java using reactor. I came to a situation where the main thread is blocked when an object subscribes and some delay event took place.
Then I came to know the concept of subscribeOn.boundedElastic and many more pipelines like this.
I got it that they are trying to make it asynchronous by moving those subscribers to other threads.
But if it occurs like this then why is the asynchronous. Is it not thread-based programming?
If we are trying to achieve the async behaviour of Node JS then according to my view it should be in a single thread.
Summary of my question is:
So I don't get the fact of using or calling reactive programming as asynchronous or functional programming because of two reason
Main thread is blocked
We can manage the thread and can run it in another pool. Runnable service/ callable we can also define.
First of all you can't compare asynchronous with functional programming. Its like comparing a rock with a banana. Its two separate things.
Functional programming is compared to other types of programming, like object oriented programming or procedural programming etc. etc.
Reactor is a java library, and java is an object oriented programming language with functional features.
Asynchronous i will explain with what wikipedia says
Asynchrony, in computer programming, refers to the occurrence of events independent of the main program flow and ways to deal with such events.
So basically how to handle stuff "around" your application, that is not a part of the main flow of your program.
In comparison to Blocking, wikipedia again:
A process that is blocked is one that is waiting for some event, such as a resource becoming available or the completion of an I/O operation.
A traditional servlet application works by assigning one thread per request.
So every time a request comes in, a thread is spawned, and this thread follows along the request until the request returns. If there is something blocking during this request, for instance reading a file from the operating system, or making a request to another service. The assigned thread will block and wait until the reading of the file is completed, or the request has returned etc.
Reactive works with subscribers and producers and makes heavy use of the observer pattern. Which means that as soon as some thing blocks, reactor can take that thread and use it for something else. And then it is un-blocked any thread can pick up where it left off. This makes sure that every thread is always in use, and utilized at 100%.
All things processed in reactor is done by the event loop the event loop is a single threaded loop that just processes events as quick as possible. Schedulers schedule things to be processed on the event loop, and after they are processed a scheduler picks up the result and carries on.
If you just run reactor you get a default scheduler that will schedule things for you completely automatically.
But lets say you have something blocking. Well then you will stop the event loop. And everything needs to wait for that thing to finish.
When you run a fully reactive application you usually get one event loop per core during startup. Which means lets say you have 4 cores, you get 4 event loops and you block one, then during that period of blockages your application runs 25% slower.
25% slower is a lot!
Well sometimes you have something that is blocking that you can't avoid. For instance an old database that doesn't have a non-blocking driver. Or you need to read files from the operating system in a blocking manor. How do you do then?
Well the reactor team built in a fallback, so that if you use onSubscribe in combination with its own elastic thread pool, then you will get the old servlet behaviour back for that single subscriber to a specific say endpoint etc.
This makes sure that you can run fully reactive stuff side by side with old legacy blocking things. So that maybe some reaquests usese the old servlet behaviour, while other requests are fully non-blocking.
You question is not very clear so i am giving you a very unclear answer. I suggest you read the reactor documentation and try out all their examples, as most of this information comes from there.
To give some context here, I have been following Project Loom for some time now. I have read The state of Loom. I have done asynchronous programming.
Asynchronous programming (provided by Java NIO) returns the thread to the thread pool when the task waits and it goes to great lengths to not block threads. And this gives a large performance gain, we can now handle many more request as they are not directly bound by the number of OS threads. But what we lose here, is the context. The same task is now NOT associated with just one thread. All the context is lost once we dissociate tasks from threads. Exception traces do not provide very useful information and debugging is difficult.
In comes Project Loom with virtual threads that become the single unit of concurrency. And now you can perform a single task on a single virtual thread.
It's all fine until now, but the article goes on to state, with Project Loom:
A simple, synchronous web server will be able to handle many more requests without requiring more hardware.
I don't understand how we get performance benefits with Project Loom over asynchronous APIs? The asynchrounous API:s make sure to not keep any thread idle. So, what does Project Loom do to make it more efficient and performant that asynchronous API:s?
EDIT
Let me re-phrase the question. Let's say we have an http server that takes in requests and does some crud operations with a backing persistent database. Say, this http server handles a lot of requests - 100K RPM. Two ways of implementing this:
The HTTP server has a dedicated pool of threads. When a request comes in, a thread carries the task up until it reaches the DB, wherein the task has to wait for the response from DB. At this point, the thread is returned to the thread pool and goes on to do the other tasks. When DB responds, it is again handled by some thread from the thread pool and it returns an HTTP response.
The HTTP server just spawns virtual threads for every request. If there is an IO, the virtual thread just waits for the task to complete. And then returns the HTTP Response. Basically, there is no pooling business going on for the virtual threads.
Given that the hardware and the throughput remain the same, would any one solution fare better than the other in terms of response times or handling more throughput?
My guess is that there would not be any difference w.r.t performance.
We don't get benefit over asynchronous API. What we potentially will get is performance similar to asynchronous, but with synchronous code.
The answer by #talex puts it crisply. Adding further to it.
Loom is more about a native concurrency abstraction, which additionally helps one write asynchronous code. Given its a VM level abstraction, rather than just code level (like what we have been doing till now with CompletableFuture etc), It lets one implement asynchronous behavior but with reduce boiler plate.
With Loom, a more powerful abstraction is the savior. We have seen this repeatedly on how abstraction with syntactic sugar, makes one effectively write programs. Whether it was FunctionalInterfaces in JDK8, for-comprehensions in Scala.
With loom, there isn't a need to chain multiple CompletableFuture's (to save on resources). But one can write the code synchronously. And with each blocking operation encountered (ReentrantLock, i/o, JDBC calls), the virtual-thread gets parked. And because these are light-weight threads, the context switch is way-cheaper, distinguishing itself from kernel-threads.
When blocked, the actual carrier-thread (that was running the run-body of the virtual thread), gets engaged for executing some other virtual-thread's run. So effectively, the carrier-thread is not sitting idle but executing some other work. And comes back to continue the execution of the original virtual-thread whenever unparked. Just like how a thread-pool would work. But here, you have a single carrier-thread in a way executing the body of multiple virtual-threads, switching from one to another when blocked.
We get the same behavior (and hence performance) as manually written asynchronous code, but instead avoiding the boiler-plate to do the same thing.
Consider the case of a web-framework, where there is a separate thread-pool to handle i/o and the other for execution of http requests. For simple HTTP requests, one might serve the request from the http-pool thread itself. But if there are any blocking (or) high CPU operations, we let this activity happen on a separate thread asynchronously.
This thread would collect the information from an incoming request, spawn a CompletableFuture, and chain it with a pipeline (read from database as one stage, followed by computation from it, followed by another stage to write back to database case, web service calls etc). Each one is a stage, and the resultant CompletablFuture is returned back to the web-framework.
When the resultant future is complete, the web-framework uses the results to be relayed back to the client. This is how Play-Framework and others, have been dealing with it. Providing an isolation between the http thread handling pool, and the execution of each request. But if we dive deeper in this, why is it that we do this?
One core reason is to use the resources effectively. Particularly blocking calls. And hence we chain with thenApply etc so that no thread is blocked on any activity, and we do more with less number of threads.
This works great, but quite verbose. And debugging is indeed painful, and if one of the intermediary stages results with an exception, the control-flow goes hay-wire, resulting in further code to handle it.
With Loom, we write synchronous code, and let someone else decide what to do when blocked. Rather than sleep and do nothing.
The http server has a dedicated pool of threads ....
How big of a pool? (Number of CPUs)*N + C? N>1 one can fall back to anti-scaling, as lock contention extends latency; where as N=1 can under-utilize available bandwidth. There is a good analysis here.
The http server just spawns...
That would be a very naive implementation of this concept. A more realistic one would strive for collecting from a dynamic pool which kept one real thread for every blocked system call + one for every real CPU. At least that is what the folks behind Go came up with.
The crux is to keep the {handlers, callbacks, completions, virtual threads, goroutines : all PEAs in a pod} from fighting over internal resources; thus they do not lean on system based blocking mechanisms until absolutely necessary This falls under the banner of lock avoidance, and might be accomplished with various queuing strategies (see libdispatch), etc.. Note that this leaves the PEA divorced from the underlying system thread, because they are internally multiplexed between them. This is your concern about divorcing the concepts. In practice, you pass around your favourite languages abstraction of a context pointer.
As 1 indicates, there are tangible results that can be directly linked to this approach; and a few intangibles. Locking is easy -- you just make one big lock around your transactions and you are good to go. That doesn't scale; but fine-grained locking is hard. Hard to get working, hard to choose the fineness of the grain. When to use { locks, CVs, semaphores, barriers, ... } are obvious in textbook examples; a little less so in deeply nested logic. Lock avoidance makes that, for the most part, go away, and be limited to contended leaf components like malloc().
I maintain some skepticism, as the research typically shows a poorly scaled system, which is transformed into a lock avoidance model, then shown to be better. I have yet to see one which unleashes some experienced developers to analyze the synchronization behavior of the system, transform it for scalability, then measure the result. But, even if that were a win experienced developers are a rare(ish) and expensive commodity; the heart of scalability is really financial.
Even after reading http://krondo.com/?p=1209 or Does an asynchronous call always create/call a new thread? I am still confused about how to provide asynchronous calls on an inherently single-threaded system. I will explain my understanding so far and point out my doubts.
One of the examples I read was describing a TCP server providing asynch processing of requests - a user would call a method e.g. get(Callback c) and the callback would be invoked some time later. Now, my first issue here - we have already two systems, one server and one client. This is not what I mean, cause in fact we have two threads at least - one in the server and one on the client side.
The other example I read was JavaScript, as this is the most prominent example of single-threaded asynch system with Node.js. What I cannot get through my head, maybe thinking in Java terms, is this:If I execute the code below (apologies for incorrect, probably atrocious syntax):
function foo(){
read_file(FIle location, Callback c) //asynchronous call, does not block
//do many things more here, potentially for hours
}
the call to read file executes (sth) and returns, allowing the rest of my function to execute. Since there is only one thread i.e. the one that is executing my function, how on earth the same thread (the one and only one which is executing my stuff) will ever get to read in the bytes from disk?
Basically, it seems to me I am missing some underlying mechanism that is acting like round-robin scheduler of some sort, which is inherently single-threaded and might split the tasks to smaller ones or call into a multiothraded components that would spawn a thread and read the file in.
Thanks in advance for all comments and pointing out my mistakes on the way.
Update: Thanks for all responses. Further good sources that helped me out with this are here:
http://www.html5rocks.com/en/tutorials/async/deferred/
http://lostechies.com/johnteague/2012/11/30/node-js-must-know-concepts-asynchrounous/
http://www.interact-sw.co.uk/iangblog/2004/09/23/threadless (.NET)
http://ejohn.org/blog/how-javascript-timers-work/ (intrinsics of timers)
http://www.mobl-lang.org/283/reducing-the-pain-synchronous-asynchronous-programming/
The real answer is that it depends on what you mean by "single thread".
There are two approaches to multitasking: cooperative and interrupt-driven. Cooperative, which is what the other StackOverflow item you cited describes, requires that routines explicitly relinquish ownership of the processor so it can do other things. Event-driven systems are often designed this way. The advantage is that it's a lot easier to administer and avoids most of the risks of conflicting access to data since only one chunk of your code is ever executing at any one time. The disadvantage is that, because only one thing is being done at a time, everything has to either be designed to execute fairly quickly or be broken up into chunks that to so (via explicit pauses like a yield() call), or the system will appear to freeze until that event has been fully processed.
The other approach -- threads or processes -- actively takes the processor away from running chunks of code, pausing them while something else is done. This is much more complicated to implement, and requires more care in coding since you now have the risk of simultaneous access to shared data structures, but is much more powerful and -- done right -- much more robust and responsive.
Yes, there is indeed a scheduler involved in either case. In the former version the scheduler is just spinning until an event arrives (delivered from the operating system and/or runtime environment, which is implicitly another thread or process) and dispatches that event before handling the next to arrive.
The way I think of it in JavaScript is that there is a Queue which holds events. In the old Java producer/consumer parlance, there is a single consumer thread pulling stuff off this queue and executing every function registered to receive the current event. Events such as asynchronous calls (AJAX requests completing), timeouts or mouse events get pushed on to the Queue as soon as they happen. The single "consumer" thread pulls them off the queue and locates any interested functions and then executes them, it cannot get to the next Event until it has finished invoking all the functions registered on the current one. Thus if you have a handler that never completes, the Queue just fills up - it is said to be "blocked".
The system has more than one thread (it has at least one producer and a consumer) since something generates the events to go on the queue, but as the author of the event handlers you need to be aware that events are processed in a single thread, if you go into a tight loop, you will lock up the only consumer thread and make the system unresponsive.
So in your example :
function foo(){
read_file(location, function(fileContents) {
// called with the fileContents when file is read
}
//do many things more here, potentially for hours
}
If you do as your comments says and execute potentially for hours - the callback which handles fileContents will not fire for hours even though the file has been read. As soon as you hit the last } of foo() the consumer thread is done with this event and can process the next one where it will execute the registered callback with the file contents.
HTH
A little help please.
I am designing a stateless server that will have the following functionality:
Client submits a job to the server.
Client is blocked while the server tries to perform the job.
The server will spawn one or multiple threads to perform the job.
The job either finishes, times out or fails.
The appropriate response (based on the outcome) is created, the client is unblocked and the response is handed off to the client.
Here is what I have thought of so far.
Client submits a job to the server.
The server assigns an ID to the job, places the job on a Queue and then places the Client on an another queue (where it will be blocked).
Have a thread pool that will execute the job, fetch the result and appropriately create the response.
Based on ID, pick the client out of the queue (thereby unblocking it), give it the response and send it off.
Steps 1,3,4 seems quite straight forward however any ideas about how to put the client in a queue and then block it. Also, any pointers that would help me design this puppy would be appreciated.
Cheers
Why do you need to block the client? Seems like it would be easier to return (almost) immediately (after performing initial validation, if any) and give client a unique ID for a given job. Client would then be able to either poll using said ID or, perhaps, provide a callback.
Blocking means you're holding on to a socket which obviously limits the upper number of clients you can serve simultaneously. If that's not a concern for your scenario and you absolutely need to block (perhaps you have no control over client code and can't make them poll?), there's little sense in spawning threads to perform the job unless you can actually separate it into parallel tasks. The only "queue" in that case would be the one held by common thread pool. The workflow would basically be:
Create a thread pool (such as ThreadPoolExecutor)
For each client request:
If you have any parts of the job that you can execute in parallel, delegate them to the pool.
And / or do them in the current thread.
Wait until pooled job parts complete (if applicable).
Return results to client.
Shutdown the thread pool.
No IDs are needed per se; though you may need to use some sort of latch for 2.1 / 2.3 above.
Timeouts may be a tad tricky. If you need them to be more or less precise you'll have to keep your main thread (the one that received client request) free from work and have it signal submitted job parts (by flipping a flag) when timeout is reached and return immediately. You'll have to check said flag periodically and terminate your execution once it's flipped; pool will then reclaim the thread.
How are you communicating to the client?
I recommend you create an object to represent each job which holds job parameters and the socket (or other communication mechanism) to reach the client. The thread pool will then send the response to unblock the client at the end of job processing.
The timeouts will be somewhat tricky, and will have hidden gotcha's but the basic design would seem to be to straightforward, write a class that takes a Socket in the constructor. on socket.accept we just do a new socket processing instantiation, with great foresight and planning on scalability or if this is a bench-test-experiment, then the socket processing class just goes to the data processing stuff and when it returns you have some sort of boolean or numeric for the state or something, handy place for null btw, and ether writes the success to the Output Stream from the socket or informs client of a timeout or whatever your business needs are
If you have to have a scalable, effective design for long-running heavy-haulers, go directly to nio ... hand coded one-off solutions like I describe probably won't scale well but would provide fundamental conceptualizing basis for an nio design of code-correct work.
( sorry folks, I think directly in code - design patterns are then applied to the code after it is working. What does not hold up gets reworked then, not before )