Is there any thread pool implementation that also allows to use the calling thread for execution?
Some background - I have a service that needs to call lots of dependent services (and do some work with their results). My service is massively parallel and might use up to 1000 threads serving concurrent requests (really, I'm not kidding).
A common pattern for parallel processing is, of course, a shared pool of background threads that is used to farm out the work from the main thread. It also has a fundamental problem of exhaustion, if each of 1000 service threads submits a long-running request then it's extremely easy to completely exhaust all of the pool's capacity.
Another classic solution is to use a private thread pool for each of the service threads. It's not very appealing, since I won't be able to make these private pools large enough.
So my idea is to use a special type of a thread pool executor that runs tasks in the calling thread and opportunistically uses the background thread pool to run tasks if it has free capacity. This way I can guarantee that the calling thread will make some progress in any case, even if the background pool is exhausted.
Does anybody know of such thread pool implementation?
Though it isn't very clear from the question, it sounds like the threads are mostly blocking waiting for responses from other services. This isn't a very productive use of these threads. A large number of threads often causes the scheduler operate inefficiently.
Alternatively, you can think about using asynchronous sockets with completion handlers. This avoids the blocking i/o, and calls handlers in your code where you can respond to i/o events occurring in the channel.
This ultimately means that you can reduce massively the number of threads in your application, and should improve performance.
Another approach is to place a task queue between the calling thread(s) and the thread pool. Every request is placed on the queue, and workers process tasks in the queue in turn. When a task is complete notification is sent back to the calling thread.
Using this mechanism, you can always ensure that tasks will eventually be processed.
Related
First of all, I could not determine what the title should be, so if it's not specific enough, the question itself will be.
We have an application that uses a foreground service and stays alive forever, and in this service, there are frequent database access jobs, network access jobs and some more, that needs to run on background threads. One job itself consumes a small amount of time, but the jobs themselves are frequent. Obviously, they need to run on worker threads, so I'm here to ask which design we should follow.
HandlerThread is a structure that creates a singular thread and uses a queue to execute tasks but always loops and waits for messages which consumes power, while ThreadPoolExecutor creates multiple threads for each job and deletes threads when the jobs are done, but because of too many threads there could be leaks, or out-of-memory even. The job count may be 5, or it may be 20, depending on how the user acts in a certain way. And, between 2 jobs, there can be a 5 second gap, or a day gap, totally depending on user. But, remember, the application stays alive forever and waits for these jobs to execute.
So, for this specific occasion, which one is better to use? A thread pool executor or a handler thread? Any advice is appreciated, thanks.
Caveat: I do not do Android work, so I am no expert there. My opinions here are based a quick reading of Android documentation.
tl;dr
➥ Use Executors rather than HandlerThread.
The Executors framework is more modern, flexible, and powerful than the legacy Thread facility used by HandlerThread. Everything you can do in HandlerThread you can do better with executors.
Differences
One big difference between HandlerThread and ThreadPoolExecutor is that the first comes from Android while the second comes from Java. So if you'll be doing other work with Java, you might not want to get in the habit of using HandlerThread.
Another big difference is age. The android.os.HandlerThread class inherits from java.lang.Thread, and dates back to the original Android API level 1. While nice for its time, the Thread facility in Java is limited in its design. That facility was supplanted by the more modern, flexible, and powerful Executors framework in later Java.
Executors
Your Question is not clear about whether these are recurring jobs or sporadically scheduled. Either can be handled with Executors.
For jobs that run once at a specific time, and for recurring scheduled jobs, use a ScheduledExecutorService. You can schedule a job to run once at a certain time by specifying a delay, a span of time to wait until execution. For repeated jobs, you can specify an amount to wait, then run, then wait, then run, and so on. I'll not address this further, as you seem to be talking about sporadic immediate jobs rather than scheduled or repeating jobs. If interested, search Stack Overflow as ScheduledExecutorService has been covered many times already on Stack Overflow.
Single thread pool
HandlerThread is a structure that creates a singular thread
If you want to recreate that single thread behavior, use a thread pool consisting of only a single thread.
ExecutorService es = Executors.newSingleThreadExecutor() ;
Make your tasks. Implement either Runnable or Callable using (a) a class implementing either interface, (b) without defining a class, via lambda syntax or conventional syntax.
Conventional syntax.
Runnable sayHelloJob = new Runnable()
{
#Override
public void run ( )
{
System.out.println( "Hello. " + Instant.now() );
}
};
Lambda syntax.
Runnable sayBonjourJob = ( ) -> System.out.println( "Bonjour. " + Instant.now() );
Submit as many of these jobs to the executor service as you wish.
es.submit( sayHelloJob ) ;
es.submit( sayBonjourJob ) ;
Notice that the submit method returns a Future. With that Future object, you can check if the computation is complete, wait for its completion, or retrieve the result of the computation. Or you may choose to ignore the Future object as seen in the code above.
Fixed thread pool
If you want multiple thread behavior, just create your executor with a different kind of thread pool.
A fixed thread pool has a maximum number of threads servicing a single queue of submitted jobs (Runnable or Callable objects). The threads continue to live, and are replaced as needed in case of failure.
ExecutorService es = Executors.newFixedThreadPool( 3 ) ; // Specify number of threads.
The rest of the code remains the same. That is the beauty of using the ExecutorService interface: You can change the implementation of the executor service to get difference behavior while not breaking your code that calls upon that executor service.
Cached thread pool
Your needs may be better service by a cached thread pool. Rather than immediately creating and maintaining a certain number of threads as the fixed thread pool does, this pool creates threads only as needed, up to a maximum. When a thread is done, and resting for over a minute, the thread is terminated. As the Javadoc notes, this is ideal for “many short-lived asynchronous tasks” such as yours. But notice that there is no upper limit of threads that may be running simultaneously. If the nature of your app is such that you may see often spikes of many jobs arriving simultaneously, you may want to use a different implementation other than cached thread pool.
ExecutorService es = Executors.newCachedThreadPool() ;
Managing executors and threads
but because of too many threads there could be leaks, or out-of-memory even
It is the job of you the programmer and your sysadmin to not overburden the production server. You need to monitor performance in production. The managagement is easy enough to perform, as you control the number of threads available in the thread pool backing your executor service.
We have an application that uses a foreground service and stays alive forever
Of course your app does eventually come to end, being shutdown. When that happens, be sure to shutdown your executor and its backing thread pool. Otherwise the threads may survive, and continue indefinitely. Be sure to use the life cycle hooks of your app’s execution environment to detect and react to the app shutting down.
The job count may be 5, or it may be 20, depending on how the user acts in a certain way.
Jobs submitted to an executor service are buffered up until they can be scheduled on a thread for execution. So you may have a thread pool of, for example, 3 threads and 20 waiting jobs. No problem. The waiting jobs will be eventually executed when their time comes.
You may want to prioritize certain jobs, to be done ahead of lower priority jobs. One easy way to do this is to have two executor services. Each executor has its own backing thread pool. One executor is for the fewer but higher-priority jobs, while the other executor is for the many lower-priority jobs.
Remember that threads in a thread pool doing no work, on stand-by, have virtually no overhead in Java for either CPU or memory. So there is no downside to having a special higher-priority executor service sitting around and waiting for eventual jobs to arrive. The only concern is that your total number of all background threads and their workload not overwhelm your machine. Also, the implementation of the thread pool may well shut down unused threads after a period of disuse.
Don't really think its a question of the number of threads you are running, more how you want them run. If you want them run one at at time (i.e. you only want to execute on database query at a time) then use a HandlerThread. If you want multi-threading / a pool of threads, then use and Executor.
In my experience, leaks are really more down to how you have coded your threads, not really the chosen implementation.
Personally, I'd use a HandlerThread, here's a nice article on implementing them and how to avoid memory leaks ... Using HandlerThread in Android
I've been working on a file/loading method for my project. I'm currently using a pool of worker-threads with a queue of all the IO tasks. There is a timer that runs every 50MS that checks whether or not the queue is empty and executes all the tasks within the queue. All the tasks are added to the queue from various threads that CANNOT be delayed from IO or waiting for the worker threads to complete a task.
Is there alternative solution for what i'm trying to achieve? Like adding a wait that only applies to the worker threads rather then the threads attempting to put IO tasks within them.
Edit: I'd also like to avoid constantly creating new Thread objects. It seems to have a pretty big impact on my applications performance.
I know the thread pool is a good thing because it can reuse threads and thus save the cost of creating new threads. But my question is, are there any disadvantages of using a thread pool? In which situation is using a thread pool not as good as using just individual threads?
In which situation is using a thread pool not as good as using just individual threads?
The only time I can think of is when you have a single thread that only needs to do a single task for the life of your program. Something like a background thread attached to a permanent cache or something. That's about the only time I fork a thread directly as opposed to using an ExecutorService. Even then, using a Executor.newSingleThreadExecutor() would be fine. The overhead of the thread-pool itself is maybe a bit more logic and some memory but very hard to see a pressing downside.
Certainly anytime you need multiple threads to perform tasks, a thread-pool is warranted. What the ExecutorService code does is reduce the amount of code you need to write to manage the threads. The improvements in readability and code maintainability is a big win.
Threadpool is suitable only when you use it for operations that takes less time to complete. Threadpool threads are not suitable for long running operations, as it can easily lead to thread starvation.
If you require your thread to have a specific priority, then threadpool thread is not suitable.
You have tasks that cause the thread to block for long periods of time. The thread pool has a maximum number of threads, so a large number of blocked thread pool threads might prevent tasks from starting.
You've got a bunch of different answers here. I think one reason for that is the question is incomplete. You are asking for "disadvantages of using a thread pool," but you didn't say, disadvantages compared to what?
A thread pool solves a particular problem. There are other problems where "thread" or "threads" is part of the solution, but "thread pool" is not. "Thread pool" usually is the answer, when the question is, how to achieve parallel execution of many, short-lived, CPU-intensive tasks, on a multi-processor system.
Threads are useful, even on a uni-processor, for other purposes. The first question I ask about any long-running thread, for example, is "what does it wait for." Threads are an excellent tool for organizing a program that has to wait for different kinds of event. You would not use a thread pool for that, though.
In addition to Gray's answer.
Other use-case is if you are using thread local or using thread as a key of some kind of hash table or stateful custom implementation of thread. In this case you have to care about cleaning the state when particular task finished using the thread even if it failed. Otherwise some surprises are possible: next task that uses thread that has some state can start functioning wrong.
Thread pools of limited size are dangerous if the tasks running on it exchange information via blocking queues - this may cause a thread starvation: What is starvation?. Good rule is to never use blocking operation in the tasks running on a thread pool.
Theads are better when you don't plan to stop using the thread. For instance in an infinite loop. Threadpools are best when doing many tasks that don't happen all at the same time. Especially when the tasks are short the overhead and clarity of using the same thread is bigger.
It depends on the situation you are going to utilize the thread pool. For example, if your system does not need to perform tasks in parallel, a threading pool would be in no use. It would keep unnecessary threads ready for a work that will never come. In such cases you can use a SingleThreadExecutor anyway. Check this link if you haven't it may clarify you about it: Thread Pool Pattern
So I have my Android Bluetooth application that has it's host and clients. The problem is, because I am making multiple connections, I need a thread to handle each connection. That's all milk'n'cookies, so I thought I'd stick all the threads in an array. A little research says a better method to doing this is using a Thread Pool, but I can't seem to get my head around how that works. Also, is it actually even possible to hold threads in an array?
A thread pool is built around the idea that, since creating threads over and over again is time-consuming, we should try to recycle them as much as possible. Thus, a thread pool is a collection of threads that execute jobs, but are not destroyed when they finish a job, but instead "return to the pool" and either take another job or sit idle if there is nothing to do.
Usually the underlying implementation is a thread-safe queue in which the programmer puts jobs and a bunch of threads managed by the implementation keep polling (I'm not implying busy-spinning necessarily) the queue for work.
In Java the thread pool is represented by the ExecutorService class which can be:
fixed - create a thread pool with a fixed number of threads
cached - dynamically creates and destroys threads as needed
single - a pool with a single thread
Note that, since thread pool threads operate in the manner described above (i.e. are recycled), in the case of a fixed thread pool it is not recommended to have jobs that do blocking I/O operations, since the threads taking those jobs will be effectively removed from the pool until they finish the job and thus you may have deadlocks.
As for the array of threads, it's as simple as creating any object array:
Thread[] threads = new Thread[10]; // array of 10 threads
Using Executors.newFixedThreadPool(int nThreads) is a nice way to minimize the overhead of creating too many threads, but it may lead to a deadlock in case that all threads are waiting for another job which itself is waiting for a free thread from the pool. Sometimes the problem can be solved by using multiple thread pools, but sometimes it can't. I'm looking for something behaving similar to newFixedThreadPool except in case that all pooled threads are blocked - in such a case the pool should grow despite its predefined bound. Is there something like this?
Actually, the deadlock is not that important here. The real problem is "how to manage the number of running threads" rather than their total number. This can be also interesting when trying to keep the CPU fully utilized without creating needlessly many threads.
If you have a contention issue, this is a design problem. If you want a quick fix as you described, you will only be curing the symptoms, not the underlying sickness.
You should instead refactor your design to eliminate deadlock using some other means.
It's generally a bad idea to have threads in a pool blocked waiting for other threads in the same thread pool.
I would try to change the design to a non-blocking one. If a thread needs the result of another operation that is being processed by the same executor I would have it submit a task back to the executor to run after the second operation completes. Or place an object into a queue to be picked up later when the other job finishes.
Alternatively you can do what Swing does with modal dialogs and have the thread that is about to block start up a child thread to keep processing requests until the parent thread unblocks. This is tricky to get right though and would require you to manually manage the threads which is a lot less safe than using an Executor.
Executor.newCachedThreadPool(); A cached thread pool will check to see if there are any available threads. If there is, the thread pool will re use the thread. If it isnt, the thread pool will create a new thread. The threads time to live is 60 seconds, so after 60 seconds the extra threads will be terminated.