Java ForkJoin pools has been compared to the other "classic" thread pool implementation in Java many times. The question I have is slightly different though:
Can I use a single, shared ForkJoin pool for an application that have BOTH types of thread usage - long running, socket-handling, transactional threads, AND short running tasks (CompletableFuture)? Or do I have to go through the pain of maintaining 2 separate pools for each type of need?
... In other words, is there a significant (performance?) penalty, if ForkJoin is used in place where other Java thread pool implementations suffice?
According to the documentation, it depends:
(3) Unless the ForkJoinPool.ManagedBlocker API is used, or the number of possibly blocked tasks is known to be less than the pool's ForkJoinPool.getParallelism() level, the pool cannot guarantee that enough threads will be available to ensure progress or good performance.
The parallelism is coupled to the amount of available CPU cores. So given enough CPU cores and not to many blocking I/O tasks, you could use the commonPool. It does not mean you should though. For one thing, ForkJoinPool is explicitly not designed for long running (blocking) tasks. For another thing, you probably want to do something with long running (blocking) tasks during shutdown.
Related
I'm using ScheduledThreadPoolExecutor to schedule a large number of tasks to run evenly over an hour.
There will be tens of thousands of tasks, and this may grow during surges in demand to hundreds of thousands or millions.
Normally, with a ThreadPoolExecutor, I can set the corePoolSize to a reasonable figure, such as 10 threads, and then allow the pool to grow as necessary.
However, the documentation for ScheduledThreadPoolExecutor says that:
because it acts as a fixed-sized pool using corePoolSize threads and
an unbounded queue, adjustments to maximumPoolSize have no useful
effect
This means that if I set the corePoolSize to 10, it will be capped at 10 threads. If I set the corePoolSize to 1000, it will instantly allocate 1000 threads, even though it may never reach that many active threads.
Is there an alternative to ScheduledThreadPoolExecutor that will let me set a high maximum thread count, without allocating all of those threads instantly?
FYI, the reason for the need for a huge active thread count is that the threads are I/O bound, and not CPU bound.
Is there an alternative to ScheduledThreadPoolExecutor that will let me set a high maximum thread count
Yes!
Well… maybe. Perhaps in the near future.
Project Loom
Project Loom is an effort in the making for a few years now to bring new capabilities to the concurrency facilities in Java.
Virtual Threads are being previewed now in Java 19. Each conventional thread in Java maps one-to-one to a thread provided and managed by the host operating system. OS threads are expensive in terms of scheduling, CPU utilization, and memory. Virtual threads, in contrast, are mapped many-to-one to a host OS thread. The JVM switches between virtual threads when detecting Java code that blocks. As a result you can have many more threads, millions even, rather than several or dozens.
Structured Concurrency is being incubated in Java 19. Basically this provides an easy way to submit and track a bunch of tasks across a bunch of threads. This “treats multiple tasks running in different threads as a single unit of work“, to quote JEP 428.
Virtual threads promise to be generally appropriate to most tasks in most apps commonly implemented in Java. There are two caveats: (a) Virtual threads are contra-indicated for tasks that are CPU-bound, continually computing with no blocking (no user-interface interaction, no logging, no disk access, no network access, no database access, etc.). An example would be video-encoding; for such work use conventional threads in Java. (b) Tasks that utilize constrained resources will need gate-keeping or throttling.
In your scenario, I expect you will not need to concern yourself with setting thread pool size. Virtual threads handle that automatically in common cases. And you’ll not need to distribute the tasks over time to avoid overloading the machine — again, virtual threads manage that automatically in common cases.
For more info on Project Loom, see the many articles available on the Web, and the many talks and interviews with the Loom team members including Ron Pressler and Alan Bateman. And I recommend multiple recent videos on the YouTube channel JEP Café.
The answer was simple:
I replaced
scheduledThreadPoolExecutor.schedule(task, delay, timeUnit);
with
scheduledThreadPoolExecutor.schedule(()->executorService.submit(task), delay, timeUnit);
where executorService is a ThreadPoolExecutor that is allowed to have a corePoolSize that is different from the maxPoolSize
The spec for this method: https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Executors.html#newCachedThreadPool()
Creates a thread pool that creates new threads as needed, but will reuse previously constructed threads when they are available. These pools will typically improve the performance of programs that execute many short-lived asynchronous tasks. Calls to execute will reuse previously constructed threads if available. If no existing thread is available, a new thread will be created and added to the pool. Threads that have not been used for sixty seconds are terminated and removed from the cache. Thus, a pool that remains idle for long enough will not consume any resources. Note that pools with similar properties but different details (for example, timeout parameters) may be created using ThreadPoolExecutor constructors.
It's not clear to me from this description - is it safe to have several of these pools in a single program? Or would I potentially run into a situation where one pool stalls on many threads and freezes up other pools?
I don't think there is a clear yes / no answer on this.
On the one hand, there is not a finite number of threads that ThreadPoolExecutor instances consume. The JVM architecture itself doesn't limit the number of threads.
On the the second hand, the OS / environment may place some limits:
The OS may have hard limits on the total number of native threads it will support.
The OS may restrict the number of native threads that a given process (in this case the JVM) can create. This could be done using ulimit or cgroup limits, and potentially other ways.
A Java thread stack has a size of 1MB (by default) on a typical 64 bit JVM. If you attempt to start() too many threads, you may run out of memory and get an OOME.
If there are a large enough number of threads and/or too much thread context switching, the thread scheduler (in the OS) may struggle.
(Context switching typically happens when a thread does a blocking syscall or has to wait on a lock or a notification. Each time you switch context there are hardware related overheads: saving and restoring registers, switching virtual memory contexts, flushing memory caches, etc.)
On the third hand, there are other things than just the number and size of thread pools that could cause problems. For example, if the thread tasks interact with each other, you could experience problems due to:
deadlocking when locking shared objects,
too much contention on shared locks leading to resource starvation,
too much work leading to timeouts, or
priority inversion problems ... if you try to use priorities to "manage" the workload.
So ...
Is it safe to have several of these pools in a single program?
Or would I potentially run into a situation where one pool stalls on many threads and freezes up other pools.
It is unlikely you would get a "stall" ... unless the tasks are interacting in some way.
But if you have too many runnable threads competing for CPU, each one will get (on average) a smaller share of the finite number of cores available. And lock contention or too much context switching can slow things down further.
I am using ThreadPoolexecutor by replacing it with legacy Thread.
I have created executor as below:
pool = new ThreadPoolExecutor(coreSize, size, 0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(coreSize),
new CustomThreadFactory(name),
new CustomRejectionExecutionHandler());
pool.prestartAllCoreThreads();
Here core size is maxpoolsize/5. I have pre-started all the core threads on start up of application roughly around 160 threads.
In legacy design we were creating and starting around 670 threads.
But the point is even after using Executor and creating and replacing legacy design we are not getting much better results.
For results memory management we are using top command to see memory usage.
For time we have placed loggers of System.currentTime in millis to check the usage.
Please tell how to optimize this design. Thanks.
But the point is even after using Executor and creating and replacing legacy design we are not getting much better results.
I am assuming that you are looking at the overall throughput from your application and you are not seeing a better performance as opposed to running each task in its own thread -- i.e. not with a pool?
This sounds like you were not being blocked because of context switching. Maybe your application is IO bound or otherwise waiting on some other system resource. 670 threads sounds like a lot and you would have been using a lot of thread stack memory but otherwise it may not have been holding back the performance of your application.
Typically we use the ExecutorService classes not necessarily because they are faster than raw threads but because the code is easier to manage. The concurrent classes take care of a lot of locking, queueing, etc. out of your hands.
Couple code comments:
I'm not sure you want the LinkedBlockingQueue to be limited by core-size. Those are two different numbers. core-size is the minimum number of threads in the pool. The size of the BlockingQueue is how many jobs can be queued up waiting for a free thread.
As an aside, the ThreadPoolExecutor will never allocate a thread past the core thread number, unless the BlockingQueue is full. In your case, if all of the core-threads are busy and the queue is full with the core-size number of queued tasks is when the next thread is forked.
I've never had to use pool.prestartAllCoreThreads();. The core threads will be started once tasks are submitted to the pool so I don't think it buys you much -- at least not with a long running application.
For time we have placed loggers of System.currentTime in millis to check the usage.
Be careful on this. Too many loggers could affect performance of your application more than re-architecting it. But I assume you added the loggers after you didn't see a performance improvement.
The executor merely wraps the creation/usage of Threads, so it's not doing anything magical.
It sounds like you have a bottleneck elsewhere. Are you locking on a single object ? Do you have a single single-threaded resource that every thread hits ? In such a case you wouldn't see any change in behaviour.
Is your process CPU-bound ? If so your threads should (very roughly speaking) match the number of processing cores available. Note that each thread you create consumes memory for its stack, and if you're memory bound, then creating multiple threads won't help here.
I have mainly CPU intensive operation which is running on a thread pool.
Operation however has certain amount of waiting foe external events which doesn't happen uniformly in time.
Since in Java, as far as I know, there is not a thread pool implementation which automatically sizes its number of threads based on observed task throughput (as in Microsoft's CLR 4), is there at least a way to manually tell to thread pool to increase its size when a blocking operation starts and to decrease when it ends?
For example with 8 cores, pool size is 8.
If operation is 100% CPU bound, just use this fixed pool.
If there is some blocking operation, one should be able to do this:
pool.increase();
waitForSpecialKeyPress();
pool.decrease();
Here is how it is being done in Microsoft's C++ Async library: Use Oversubscription to Offset Latency
You could extend ThreadPoolExecutor to add your own increase() and decrease() functions, which do simple setMaximumPoolSize(getMaximumPoolSize() +/- 1).
Make sure to synchronize the methods, to make sure you don't mess up the pool size by accident.
Java 7's ForkJoinPool has a ManagedBlocker, which can be used to keep the pool informed about blocked threads, so that it can schedule more threads if necessary.
EDIT: I forgot to mention, the classes are also available for Java 6 as jsr166y.
In a life without Java Executors, new threads would have to be created for each Runnable tasks. Making new threads requires thread overhead (creation and teardown) that adds complexity and wasted time to a non-Executor program.
Referring to code:
no Java Executor -
new Thread (aRunnableObject).start ();
with Java Executor -
Executor executor = some Executor factory method;
exector.execute (aRunnable);
Bottom line is that Executors abstract the low-level details of how to manage threads.
Is that true?
Thanks.
Bottom line is that Executors abstract the low-level details of how to manage threads. Is that true?
Yes.
They deal with issues such as creating the thread objects, maintaining a pool of threads, controlling the number of threads are running, and graceful / less that graceful shutdown. Doing these things by hand is non-trivial.
EDIT
There may or may not be a performance hit in doing this ... compared with a custom implementation perfectly tuned to the precise needs of your application. But the chances are that:
your custom implementation wouldn't be perfectly tuned, and
the performance difference wouldn't be significant anyway.
Besides, the Executor support classes allow you to simply tune various parameters (e.g. thread pool sizes) if there is an issue that needs to be addressed. I don't see how garbage collection overheads would be significantly be impacted by using Executors, one way or the other.
As a general rule, you should focus on writing your applications simply and robustly (e.g. using the high level concurrency support classes), and only worry about performance if:
your application is running "too slow", and
the profiling tools tell you that you've got a problem in a particular area.
Couple of benefits of executors as against normal threads.
Throttling can be achieved easily by varying the size of ThreadPools. This helps keeping control/check on the number of threads flowing through your application. Particularly helpful when benchmarking your application for load bearing.
Better management of Runnable tasks can be achieved using the RejectionHandlers.
I think all that executors do is that they will do the low level tasks
for you, but you still have to judiciously decide which thread pool do
you want. I mean if your use case needs maximum 5 threads and you go
and use thread pool having 100 threads, then certainly it is going to
have impact on performance. Other than this there is noting extra
being done at low level which is going to halt the system. And last of
all, it is always better to get an idea what is being done at low
level so that it will give us fair idea about the underground things.