I was going through the twitter finagle library which is an asynchronous service framework in scala and I have some question regarding asynchronous libraries in general.
So as I understand, the advantage of an synchronous library using a callback is that the application thread gets free and the library calls the callback as soon as the request is completed over the network. And in general the application threads might not have a 1:1 mapping with the library thread.
The service call in the library thread is blocking right?
If that's the case then we are just making the blocking call in some other thread. This makes the application thread free but some other thread is doing the same work. Can't we just increase the number of application threads to have that advantage?
It's possible that I mis-understand how the asynchronous libraries are implemented in Java/Scala or JVM in general. Can anyone help me understand how does this work?
Async approach is useful abstraction: your CPU-intensive thread offloads long-running IO operation to dedicated (maybe, belonging to a library) thread. When IO is done, some other thread will receive IO result.
Using blocking approach, you'll miss CPU ticks for your threads which are doing blocking IO call. And adding some more threads to ensure there's always free thread to do some CPU work means wasting OS/JVM resources for re-scheduling.
Blocking IO is used because it's simpler to program (no need to synchronize caller and callback).
Actually, IO is only one possible use-case where async style is useful. In general, whenever you feel that task at hand will benefit from splitting it into several activities, which can be run concurrently and would communicate with each other, this is the case for async style. Examples not connected to IO:
GUI: GUI event loop thread passes user input to background threads, and they do necessary processing;
Utilizing modern multi-core CPUs: if your task can be split in several subtasks, you can run these in separate threads, utilizing all available cores. Naturally, you'll need to gather results of subtasks, and you'll need async style here.
Related
Java Virtual Threads
In Java 19 were introduced Virtual Threads JEP-425 as a preview feature
After some investigation of concept of Java Virtual Threads (Project Loom), called sometimes lightweight threads (or sometimes fibers or green threads), I'm pretty interested in potential usage of them with reactive libraries, like, for example, with Spring WebFlux based on Project Reactor (reactive streams implementation) and Netty, for making blocking calls efficiently.
Most JVM implementations today implement Java threads as thin direct wrappers around operating system threads, called sometimes heavyweight, OS-managed threads platform threads.
While a platform thread can only execute a single thread at a time, the virtual threads have the ability to switch to executing a different virtual thread when the currently executed virtual thread makes a blocking call (e.g. network, file system, database call).
How do we deal with blocking calls in Reactor?
So, when dealing with blocking calls in Reactor we use the following construct:
Mono.fromCallable(() -> {
return blockingOperation();
}).subscribeOn(Schedulers.boundedElastic());
In subcribeOn() we provide a Scheduler that creates a dedicated thread for executing that blocking operation. However, it means that thread will eventually be blocked, so, as we are still on the old-fashioned threading model, we'll actually block the platform thread, which is still not really efficient way of dealing with CPU resources.
Here is the question:
So, the question is, could we use the virtual threads with reactive frameworks directly for making blocking calls like this, using, for example, Executors.newVirtualThreadPerTaskExecutor() :
Creates an Executor that starts a new virtual Thread for each task.
The number of threads created by the Executor is unbounded.
Mono.fromCallable(() -> {
return blockingOperation();
}).subscribeOn(Schedulers.fromExecutorService(Executors.newVirtualThreadPerTaskExecutor()));
Would it work out of the box? Will we actually get the benefits from this approach, in terms of dealing with our CPU resources more efficiently and increase performance of our application? Does it mean that we can easily integrate reactive library with any blocking library/framework, for example, Spring Data JPA (which is based on JDBC) and millions of others and magically turn them to non-blocking?
I am trying to understand the core principles of non-blocking programming (and frameworks like project reactor). The main idea is to have "thread pool" with determined number of threads (executors) and tasks which are executed there. We should not have any blocked threads. In "user code" we just run something to execute and leave callback (what to do with the result). Out "user" thread is not blocked, right. But what if my task depends on some jdbc query. My task will request this query and then will be blocked waiting for the result, right? So, this thread is blocked.
But we avoid thread creating (which is expensive). Is it the core benefit of this style?
If my thread pool consists of 2 executors and both are blocked waiting for something, other tasks will not be executed, right? How to avoid it? Create more than 2 threads?
Threads are relatively costly system resources. For example, each thread needs memory for the call stack. How much this is depends on the operating system, but typically it's something like 1 or 2 MB. This means it's not a good idea to start thousands of threads - you'd waste 1 or 2 GB memory just on the call stacks of 1000 threads.
So, to do things more efficiently you want to limit the number of threads, for example using a thread pool to handle work. The thread pool makes it possible to manage the number of threads that are being used.
However, imagine that you'd have a thread pool with 10 threads, and then 10 requests come in. Each of your threads will be reserved to handle a request. While they are busy, you can't handle request #11 because there is no thread free. When you are using blocking I/O, then, even though all your 10 threads are doing nothing (waiting for I/O to complete), request #11 cannot be handled...
When you use non-blocking I/O, threads will never need to wait for I/O - so when the handling request #3 is suspended because it needs the result of an I/O operation, the thread that was handling it can temporarily switch to handling other requests.
So, with non-blocking I/O, you never have waiting threads and you are using system resources more efficiently.
This will only work if you are using non-blocking I/O from the front to the back of your system. If at the back-end you are using JDBC, which is a blocking API, then you'll loose the full benefit of non-blocking I/O.
Therefore, if you have a database at the back-end, this works best if you have a DB which supports non-blocking I/O. Some NoSQL databases like MongoDB support this, and for some relational databases there are special drivers / APIs available that support this. You won't be using JDBC in that case, because JDBC is an inherently blocking API.
Oracle is working on a new API for relational databases tentatively called
ADBA which will allow you to do non-blocking / async I/O with relational databases but it's not ready yet.
Project Reactor is an implementation of Reactive Streams specification. The specification overview can be found at ReactiveManifest. It's not just creating a set of threads and letting them do their jobs, It's the framework or the runtime (in this case ProjectReactor) that will organize your code in such a way that it'll presumably behave as nonblocking. Also, the whole system implementation has to be in this fashion otherwise you won't be benefited from the reactive streams.
If my thread pool consists of 2 executors and both are blocked waiting for something, other tasks will not be executed, right? How to avoid it? Create more than 2 threads?
The answer to this will be yes, and no. The framework may are may not create threads. Since the code will be interleaved among the threads, Since the non-blocking system are event-driven including the low-level operations (ex, libuv I/O), It's not necessary for a thread to wait for the completion of an I/O operation. Meanwhile, the thread will be executing something meaningful. The completion of the task will be notified and the dependent code can be executed by any of the available thread. The goal of such a system is to utilize the CPU to the fullest with limited resources(threads).
Taken from http://www.reactive-streams.org.
The main goal of Reactive Streams is to govern the exchange of stream data across an asynchronous boundary—think passing elements on to another thread or thread-pool—while ensuring that the receiving side is not forced to buffer arbitrary amounts of data. In other words, back pressure is an integral part of this model in order to allow the queues which mediate between threads to be bounded. The benefits of asynchronous processing would be negated if the communication of back pressure were synchronous (see also the Reactive Manifesto), therefore care has to be taken to mandate fully non-blocking and asynchronous behavior of all aspects of a Reactive Streams implementation.
It's the Reactor framework that enforces and help you in building a completely non-blocking system from the ground up.
I'm learning about Vertx and it's ecosysteme, firstly i learned about Event loop, and the concept is really nice to me.
But since Servlet 3.1 we can use async support in JAVA based servers.
I'm using Spring and it's new class named deferredresult which can take thread from tomcat, give execution of logic to thread from executor thread pool that make thread from tomcat free to handle another requests and then when it done return response.
In event loop all blocking calls should be done by worker vertx, the concept is absolute the same, you give a thread to blocking call and provide callback when task is done event loop execute callback and return response.
These concepts looks really similar to me.
Maybe i miss something but what is the difference between these concepts?
Thread pool based web servers use worker threads as the primary execution context for user requests. When you develop a Spring or JavaEE application, you call a lot of blocking code (JDBC, Hibernate, JAX-RS client, ...etc).
Then the servlet 3.1 async API was added to solve problems like long polling (if all your worker threads are waiting, you cannot handle requests any more).
Event loop based systems (Vert.x, Node) however are built to handle a lot of user requests with a single thread. Usually you combine them with non blocking database drivers or web clients.
It's a very powerful model (less thread migrations, warm caches, ...etc). But you must not block the event loop or you can't handle events any more.
In an ideal world you would use only non blocking libraries but reality is many Java libraries are not and we shouldn't just throw this legacy away. As a workaround, Vert.x provides a way to offload blocking code execution to a worker pool.
I hope this clarifies a bit and shows that, beyond the similarities, the use cases are different.
What I understood from Vert.x documentation (and a little bit of coding in it) is that Vert.x is single threaded and executes events in the event pool. It doesn't wait for I/O or any network operation(s) rather than giving time to another event (which was not before in any Java multi-threaded framework).
But I couldn't understand following:
How single thread is better than multi-threaded? What if there are millions of incoming HTTP requests? Won't it be slower than other multi-threaded frameworks?
Verticles depend on CPU cores. As many CPU cores you have, you can have that many verticles running in parallel. How come a language that works on a virtual machine can make use of CPU as needed? As far as I know, the Java VM (JVM) is an application that uses just another OS process for (here my understanding is less about OS and JVM hence my question might be naive).
If a single threaded, non-blocking concept is so effective then why can't we have the same non-blocking concept in a multi-threaded environemnt? Won't it be faster? Or again, is it because CPU can execute one thread at a time?
What I understood from Vert.x documentation (and a little bit of coding in it) is that Vert.x is single threaded and executes events in the event pool.
It is event-driven, callback-based. It isn't single-threaded:
Instead of a single event loop, each Vertx instance maintains several event loops. By default we choose the number based on the number of available cores on the machine, but this can be overridden.
It doesn't wait for I/O or any network operation(s)
It uses non-blocking or asynchronous I/O, it isn't clear which. Use of the Reactor pattern suggests non-blocking, but it may not be.
rather than giving time to another event (which was not before in any Java multi-threaded framework).
This is meaningless.
How single thread is better than multi-threaded?
It isn't.
What if there are millions of incoming HTTP requests? Won't it be slower than other multi-threaded frameworks?
Yes.
Verticles depend on CPU cores. As many CPU cores you have, you can have that many verticles running in parallel. How come a language that works on a virtual machine can make use of CPU as needed? As far as I know, the Java VM (JVM) is an application that uses just another OS process for (here my understanding is less about OS and JVM hence my question might be naive).
It uses a thread per core, as per the quotation above, or whatever you choose by overriding that.
If a single threaded, non-blocking concept is so effective then why can't we have the same non-blocking concept in a multi-threaded environemnt?
You can.
Won't it be faster?
Yes.
Or again, is it because CPU can execute one thread at a time?
A multi-core CPU can execute more than one thread at a time. I don't know what 'it' in 'is it because' refers to.
First of all, Vertx isn't single threaded by any means. It just doesn't spawn more threads that it needs.
Second, and this is not related to Vertx at all, JVM maps threads to native OS threads.
Third, we can have non-blocking behavior in multithreaded environment. It's not one thread per CPU, but one thread per core.
But then the question is: "what are those threads doing?". Because usually, to be useful, they need other resources. Network, DB, filesystem, memory. And here it becomes tricky. When you're single threaded, you don't have race conditions. The only one accessing the memory at any point of time is you. But if you're multi threaded, you need to concern yourself with mutexes, or any other way to keep you data consistent.
Q:
How single thread is better than multi-threaded? What if there are millions of incoming HTTP requests? Won't it be slower than other multi-threaded frameworks?
A:
Vert.x isn't a single threaded framework, it does make sure that a "verticle" which is something you deploy within you application and register with vert.x is mostly single threaded.
The reason for this is that concurrency with multiple threads over complicates concurrency with locks synchronisation and other concept that need to be taken care of with multi threaded communication.
While verticles are single threaded the do use something called an event loop which is the true power behind this paradigm called the reactor pattern or multi reactor pattern in Vert.x's case. Multiple verticles can be registered within one application, communication between these verticles run through an eventbus which empowers verticles to use an event based transfer protocol internally but this can also be distributed using some other technology to manage the clustering.
event loops handle events coming in on one thread but everything is async so computation gets handled by the loop and when it's done a signal notifies that a result can be used.
So all computation is either callback based or uses something like Reactive.X / fibers / coroutines / channels and the lot.
Due to the simpler communication model for concurrency and other nice features of Vert.x it can actually be faster than a lot of the Blocking and pure multi threaded models out there.
the numbers
Q:
If a single threaded, non-blocking concept is so effective then why can't we have the same non-blocking concept in a multi-threaded environemnt? Won't it be faster? Or again, is it because CPU can execute one thread at a time?
A:
Like a said with the first question it's not really single threaded. Actually when you know something is blocking you'll have to register computation with a method called executeBlocking which wil make it run multithreaded on an ExecutorService managed by Vert.x
The reason why Vert.x's model is mostly faster is also here because event loops make better use of cpu computation features and constraints. This is mostly powered by the Netty project.
the overhead of multi threading with it's locks and syncs imposes to much strain to outdo Vert.x with it's multi reactor pattern.
Where can I find asynchronous programming example using Java? I'm interested in finding patterns in asynchronous programming for building applications that present responsiveness (preventing applications that periodically hang and stop responding to user input and server applications that do not respond to client requests in a timely fashion) and scalability.
In particularly it will be helpful to see a sample which performs I/O operations (such as file reads/writes, Web requests, and database queries) and also has a lot of CPU processing involved like a shopping suggester in a webpage.
Which are the Java libraries which can help in determining when an application's responsiveness is unpredictable - because the application's thread performs I/O requests, the application is basically giving up control of the thread's processing to the I/O device (a hard drive, a network, or whatever)
In a GUI, you could use threads to perform background tasks.
Java supports non blocking I/O in the new I/O API (NIO).
If your question is more architecturally oriented, this book offers an in-depth discussion of asynchronous patterns: Patterns of Enterprise Application Architecture, by Martin Fowler.
For examples performing asynchronous operations with emphasis on non-blocking IO on files, you may check some samples here: https://github.com/javasync/idioms (disclaimer I am the author).
I use this samples in introduction to asynchronous programming in Java and we explore callback based, CompletableFuture and finally reactive streams.
Which are the Java libraries which can help in determining when an application's responsiveness is unpredictable - because the application's thread performs I/O requests, the application is basically giving up control of the thread's processing to the I/O device (a hard drive, a network, or whatever)
If I understand you correctly, you are asking for some library that examines other threads to determine if they are blocked in I/O calls.
I don't think that this is possible in a standard JVM. Furthermore, I don't think that this would necessarily be sufficient to guarantee "responsiveness".
If you are using some kind of I/O operation (for example read on InputStream, which can block) you put it into a thread and the simplest solution is to use join on the thread for a given amount:
MyThread myThread = new MyThread();
myThread.start();
myThread.join(10000);
This join will then wait for atmost 10 seconds. After that time you can just ignore the thread, ...
You can also use the Decorator pattern. You can read more here.
in a web environment, you can make use of the new j2ee6 Asynchronous feature.
take a look at
http://docs.oracle.com/javaee/6/tutorial/doc/gkkqg.html