I need to make a program with a limited amount of threads (currently using newFixedThreadPool) but I have the problem that all threads get created from start, filling up memory at alarming rate.
I wish to prevent this. Threads should only be created shortly before they are executed.
e.g.: I call the program and instruct it to use 2 threads in the pool. The program should create & launch the first 2 Threads immediately (obviously), create the next 2 to wait for the previous 2, and at that point wait until one or both of the first 2 ended executing.
I thought about extending executor or FixedThreadPool or such. However I have no clue on how to start there and doubt it is the best solution. Easiest would have my main Thread sleeping on intervals, which is not really good either...
Thanks in advance!
Have you tried taking a look at ThreadPoolExecutor ? Using the right constructor parameters, you could easily tweak the number and keep-alive time of the created threads.
Looking at the details in your post...
I call the program and instruct it to use 2 threads in the pool. The program should create & launch the first 2 Threads immediately (obviously), create the next 2 to wait for the previous 2, and at that point wait until one or both of the first 2 ended executing.
Your problem is much more about synchronizing tasks execution than in fact pooling threads. From what you say here, you want to have 2 threads executing any number of tasks; if you don't want to have 100 jobs running at the same time, don't create a 100 threads pool...
I would suggest either using a BlockingQueue to control your Runnables, or create a 2 threads pool using a ThreadPoolExecutor, and feed it all your tasks. It will execute them when threads are available.
Does that make sense with what you try to achieve here?
I don't think you should manipulate the thread pool implementation. If you create the threads shortly before execution, you lose the main benefit of the pool, that recycles your threads.
Maybe you should reduce the maximum number of threads in the pool. If you instruct the pool to create too many of them, the total out-of-heap memory used for their stack spaces will consume all available memory. I assume that this is the kind of OutOfMemoryError you have (?).
If you're looking at this from a performance perspective, then it's best to take the hit in memory when you first start up the application than constantly get bombarded with allocating and deallocating memory while the program is running.
If it's using too much memory when you start the application, then it will still be too much memory later. You should throttle down the size of the thread pool.
There are additional benefits to using a thread pool, such as if you lose a thread along the way, the thread pool will automatically create a new one to replace it, keeping your thread pool at a constant size.
If this isn't the type of benefit that you're looking for, then you may wish to handle the threads in memory manually, and avoid the thread pool.
Related
I used to use ThreadPoolExecutors for years and one of the main reasons - it is designed to 'faster' process many requests because of parallelism and 'ready-to-go' threads (there are other though).
Now I'm stuck on minding inner design well known before.
Here is snippet from java 8 ThreadPoolExecutor:
public void execute(Runnable command) {
...
/*
* Proceed in 3 steps:
*
* 1. If fewer than corePoolSize threads are running, try to
* start a new thread with the given command as its first
* task. The call to addWorker atomically checks runState and
* workerCount, and so prevents false alarms that would add
* threads when it shouldn't, by returning false.
*/
...
int c = ctl.get();
if (workerCountOf(c) < corePoolSize) {
if (addWorker(command, true))
return;
c = ctl.get();
}
...
I'm interested in this very first step as in most cases you do not want thread poll executor to store 'unprocessed requests' in the internal queue, it is better to leave them in external input Kafka topic / JMS queue etc. So I'm usually designing my performance / parallelism oriented executor to have zero internal capacity and 'caller runs rejection policy'. You chose some sane big amount of parallel threads and core pool timeout not scare others and show how big the value is ;). I don't use internal queue and I want tasks to start to be processed the earlier the better, thus it has become 'fixed thread pool executor'. Thus in most cases I'm under this 'first step' of the method logic.
Here is the question: is this really the case that it will not 'reuse' existing threads but will create new one each time it is 'under core size' (most cases)? Would it be not better to 'add new core thread only if all others are busy' and not 'when we have a chance to suck for a while on another thread creation'? Am I missing anything?
The doc describes the relationship between the corePoolSize, maxPoolSize, and the task queue, and what happens when a task is submitted.
...but will create new one [thread] each time it is 'under core size...'
Yes. From the doc:
When a new task is submitted in method execute(Runnable), and fewer
than corePoolSize threads are running, a new thread is created to
handle the request, even if other worker threads are idle.
Would it be not better to add new core thread only if all others are busy...
Since you don't want to use the internal queue this seems reasonable. So set the corePoolSize and maxPoolSize to be the same. Once the ramp up of creating the threads is complete there won't be any more creation.
However, using CallerRunsPolicy would seem to hurt performance if the external queue grows faster than can be processed.
Here is the question: is this really the case that it will not 'reuse' existing threads but will create new one each time it is 'under core size' (most cases)?
Yes that is how the code is documented and written.
Am I missing anything?
Yes, I think you are missing the whole point of "core" threads. Core threads are defined in the Executors docs are:
... threads to keep in the pool, even if they are idle.
That's the definition. Thread startup is a non trivial process and so if you have 10 core threads in a pool, the first 10 requests to the pool each start a thread until all of the core threads are live. This spreads the startup load across the first X requests. This is not about getting the tasks done, this is about initializing the TPE and spreading the thread creation load out. You could call prestartAllCoreThreads() if you don't want this behavior.
The whole purpose of the core threads is to have threads already started and running available to work on tasks immediately. If we had to start a thread each time we needed one, there would be unnecessary resource allocation time and thread start/stop overhead taking compute and OS resources. If you don't want the core threads then you can let them timeout and pay for the startup time.
I used to use ThreadPoolExecutors for years and one of the main reasons - it is designed to 'faster' process many requests because of parallelism and 'ready-to-go' threads (there are other though).
TPE is not necessarily "faster". We use it because to manually manage and communicate with a number of threads is hard and easy to get wrong. That's why the TPE code is so powerful. It is the OS threads that give us parallelism.
I don't use internal queue and I want tasks to start to be processed the earlier the better,
The entire point of a threaded program is the maximize throughput. If you run 100 threads on a 4 core system and the tasks are CPU intensive, you are going to pay for the increased context switching and the overall time to process a large number of requests is going to decrease. Your application is also most likely competing for resources on the server with other programs and you don't want to cause it to slow to a crawl if 100s of jobs try to run in a thread pool at the same time.
The whole point of limiting your core threads (i.e. not making them a "sane big amount") is that there is an optimal number of concurrent threads that will maximize the overall throughput of your application. It can be really hard to find the optimal core thread size but experimentation, if possible, would help.
It depends highly on the degree of CPU versus IO in a task. If the tasks are making remote RPC calls to a slow service then it might make sense to have a large number of core threads in your pool. If they are predominantly CPU tasks, however, you are going to want to be closer to the number of CPU/cores and then queue the rest of the tasks. Again it is all about overall throughput.
To reuse threads one need somehow to transfer task to existing thread.
This pushed me towards synchronous queue and zero core pool size.
return new ThreadPoolExecutor(0, maxThreadsCount,
10L, SECONDS,
new SynchronousQueue<Runnable>(),
new BasicThreadFactory.Builder().namingPattern("processor-%d").build());
I have really reduced amounts of 'peaks' of 500 - 1500 (ms) on my 'main flow'.
But this will work only for zero-sized queue. For non-zero-sized queue question is still open.
When I have this code in an application:
Executors.newFixedThreadPool(4);
but I never use this thread pool. Will the idle threads consume CPU time? If so - why?
No, these threads are created lazily, or 'on-demand'. As stated in the documentation (emphasis mine):
On-demand construction
By default, even core threads are initially created and started only when new tasks arrive
Java provides methods to override this default and allow for eager creation, namely prestartCoreThread and prestartAllCoreThreads.
Once threads are actually created, idle ones (generally) won't take CPU time as there is no reason for them to be scheduled on a core when they have no work to do.
They will still hold on to some memory however, for their stack and whatnot.
The javadoc states:
Creates a thread pool that reuses a fixed number of threads operating off a shared unbounded queue. At any point, at most nThreads threads will be active processing tasks.
This might lead to assume: we don't know exactly. But as the other answer clearly finds - we can know, and the implementation is actually fully lazy. Thus:
ExecutorService service = Executors.newFixedThreadPool(4);
does not even cause a lot of setup. The implementation would be free to wait for any
service.submit(...
to happen.
On the other hand, that thread pool could (theoretically) be created immediately, and also 4 OS threads could be created. If the later is the case, all those threads would be idle'ing, so they should not consume any CPU resources.
But of course, as Elliott is correctly pointing out, that initial step of creating the pool and (potentially) creating 1 to 4 threads requires CPU activity.
And of course: threads are a OS resource. So even when they "only exist" and do nothing; they have that "cost". Where again, it might depend on the OS if that could ever turn into a problem (like reaching some "maximum number of existing threads" limit).
Finally: as this got me curious, I had a look into the current Java8 source code (from newFixedThreadPoo() and ThreadPoolExcecutor() down into DefaultThreadFactory). If I am not mistaken: those constructors only prepare for thread creation.
So: the "current" implementation is "fully" lazy; and if you really only call newFixedThreadPool() without ever using the resulting ExecutorService ... nothing happens (in terms of new threads being created).
I have read many similar questions . However I was not quite satisfied with answers.
I would like to build an algorithm that would adjust the number of threads depending on the average speed.
Let's say as I introduce a new thread, the average speed of task execution increases , it means that the new thread is good. Then the algorithm should try to add another thread ... until the optimal number of threads is achieved .......
Also the algorithm should be keeping track of the average speed. If at some point the average speed goes down significantly, let's say by 10 % (for any reason e.g. i open a different application or whatever) , then the algorithm should terminate one thread and see if the speed goes up ...
Maybe such an API exists. Please, give me any directions or any code example how I could implement such an algorithm
Thank You !
I do not know self-tune system that you are describing but it sounds like not so complicated task once you are using ready thread pool. Take thread pool from concurrency package, implement class TimeConsumptionCallable implements Callable that wraps any other callable and just measures the execution time.
Now you just have to change (increase or decrease) number of working threads when average execution time increases or decreases.
Just do not forget that you need enough statistics before you decide to change number of working threads. Otherwise various random effects that do not depend on your application can cause your thread pool to grow and go down all the time that can itself kill overall performance.
newCachedThreadPool() V/s newFixedThreadPool suggests that perhaps you should be looking at ExecutorService.newCachedThreadPool().
Creates a thread pool that creates new threads as needed, but will reuse previously constructed threads when they are available. These pools will typically improve the performance of programs that execute many short-lived asynchronous tasks. Calls to execute will reuse previously constructed threads if available. If no existing thread is available, a new thread will be created and added to the pool. Threads that have not been used for sixty seconds are terminated and removed from the cache. Thus, a pool that remains idle for long enough will not consume any resources. Note that pools with similar properties but different details (for example, timeout parameters) may be created using ThreadPoolExecutor constructors.
If your threads do not block at any time, then the maximum execution speed is reached when you have as many threads as cores, as simply more than 100% CPU usage is not possible.
In other situations it is very difficult to measure how much a new thread will increase/decrease the execution speed, as you just watch a moment in time and make assumptions based on something that could be entirely different the next second.
One idea would be to use an Executor class in combination with a Queue that you specified. So you can measure the size of the queue and make assumptions based on that. If the queue is empty, threads are idle and you can remove one. If the queue fills up, threads cannot handle the load, you need to add more. If the queue is stable, you are about right.
You can come up with your own algorithm by using existing API of java :
public void setCorePoolSize(int corePoolSize) in ThreadPoolExecutor
Sets the core number of threads. This overrides any value set in the constructor.
If the new value is smaller than the current value, excess existing threads will be terminated when they next become idle.
If larger, new threads will, if needed, be started to execute any queued tasks.
Initialization:
ExecutorService service = Executors.newFixedThreadPool(5); // initializaiton
On your need basis, resize the pool by using below API
((ThreadPoolExecutor)service).setCorePoolSize(newLimit);//newLimit is new size of the pool
And one important point: If the queue is full, and new value of number of threads is greater than or equal to maxPoolSize defined earlier, Task will be rejected.
Be careful when setting maxPoolSize so that setCorePoolSize works properly.
I have a bit of an issue with an application running multiple Java threads.
The application runs a number of working threads that peek continuously at an input queue and if there are messages in the queue they pull them out and process them.
Among those working threads there is another verification thread scheduled to perform at a fixed period a check to see if the host (on which the application runs) is still in "good shape" to run the application. This thread updates an AtomicBoolean value which in turn is verified by the working thread before they start peeking to see if the host is OK.
My problem is that in cases with high CPU load the thread responsible with the verification will take longer because it has to compete with all the other threads. If the AtomicBoolean does not get updated after a certain period it is automatically set to false, causing me a nasty bottleneck.
My initial approach was to increase the priority of the verification thread, but digging into it deeper I found that this is not a guaranteed behavior and an algorithm shouldn't rely on thread priority to function correctly.
Anyone got any alternative ideas? Thanks!
Instead of peeking into a regular queue data structure, use the java.util.concurrent package's LinkedBlockingQueue.
What you can do is, run an pool of threads (you could use executer service's fixed thread pool, i.e., a number of workers of your choice) and do LinkedBlockingQueue.take().
If a message arrives at the queue, it is fed to one of the waiting threads (yeah, take does block the thread until there is something to be fed with).
Java API Reference for Linked Blocking Queue's take method
HTH.
One old school approach to throttling rate of work, that does not use a health check thread at all (and so by-passes these problems) is to block or reject requests to add to the queue if the queue is longer than say 100. This applies dynamic back pressure on to the clients generating the load, slowing them down when the worker threads are over loaded.
This approach was added to the Java 1.5 library, see java.util.concurrent.ArrayBlockingQueue. Its put(o) method blocks if the queue is full.
Are u using Executor framework (from Java's concurrency package)? If not give it a shot. You could try using ScheduledExecutorService for the verification thread.
More threads does not mean better performance. Usually if you have dual core, 2 threads gives best performance, 3 or more starts getting worse. Quad core should handle 4 threads best, etc. So be careful how much threads you use.
You can put the other threads to sleep after they perform their work, and allow other threads to do their part. I believe Thread.yield() will pause the current thread to give time to other threads.
If you want your thread to run continuously, I would suggest creating two main threads, thread A and B. Use A for the verification thread, and from B, create the other threads. Therefore thread A gets more execution time.
Seems you need to utilize Condition variables. Peeking will take cpu cycles.
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/Condition.html
Using Executors.newFixedThreadPool(int nThreads) is a nice way to minimize the overhead of creating too many threads, but it may lead to a deadlock in case that all threads are waiting for another job which itself is waiting for a free thread from the pool. Sometimes the problem can be solved by using multiple thread pools, but sometimes it can't. I'm looking for something behaving similar to newFixedThreadPool except in case that all pooled threads are blocked - in such a case the pool should grow despite its predefined bound. Is there something like this?
Actually, the deadlock is not that important here. The real problem is "how to manage the number of running threads" rather than their total number. This can be also interesting when trying to keep the CPU fully utilized without creating needlessly many threads.
If you have a contention issue, this is a design problem. If you want a quick fix as you described, you will only be curing the symptoms, not the underlying sickness.
You should instead refactor your design to eliminate deadlock using some other means.
It's generally a bad idea to have threads in a pool blocked waiting for other threads in the same thread pool.
I would try to change the design to a non-blocking one. If a thread needs the result of another operation that is being processed by the same executor I would have it submit a task back to the executor to run after the second operation completes. Or place an object into a queue to be picked up later when the other job finishes.
Alternatively you can do what Swing does with modal dialogs and have the thread that is about to block start up a child thread to keep processing requests until the parent thread unblocks. This is tricky to get right though and would require you to manually manage the threads which is a lot less safe than using an Executor.
Executor.newCachedThreadPool(); A cached thread pool will check to see if there are any available threads. If there is, the thread pool will re use the thread. If it isnt, the thread pool will create a new thread. The threads time to live is 60 seconds, so after 60 seconds the extra threads will be terminated.