Develop Reference

Optimal number of threads [duplicate]

Let's say I have a 4-core CPU, and I want to run some process in the minimum amount of time. The process is ideally parallelizable, so I can run chunks of it on an infinite number of threads and each thread takes the same amount of time.
Since I have 4 cores, I don't expect any speedup by running more threads than cores, since a single core is only capable of running a single thread at a given moment. I don't know much about hardware, so this is only a guess.
Is there a benefit to running a parallelizable process on more threads than cores? In other words, will my process finish faster, slower, or in about the same amount of time if I run it using 4000 threads rather than 4 threads?

If your threads don't do I/O, synchronization, etc., and there's nothing else running, 1 thread per core will get you the best performance. However that very likely not the case. Adding more threads usually helps, but after some point, they cause some performance degradation.
Not long ago, I was doing performance testing on a 2 quad-core machine running an ASP.NET application on Mono under a pretty decent load. We played with the minimum and maximum number of threads and in the end we found out that for that particular application in that particular configuration the best throughput was somewhere between 36 and 40 threads. Anything outside those boundaries performed worse. Lesson learned? If I were you, I would test with different number of threads until you find the right number for your application.
One thing for sure: 4k threads will take longer. That's a lot of context switches.

I agree with #Gonzalo's answer. I have a process that doesn't do I/O, and here is what I've found:
Note that all threads work on one array but different ranges (two threads do not access the same index), so the results may differ if they've worked on different arrays.
The 1.86 machine is a macbook air with an SSD. The other mac is an iMac with a normal HDD (I think it's 7200 rpm). The windows machine also has a 7200 rpm HDD.
In this test, the optimal number was equal to the number of cores in the machine.

I know this question is rather old, but things have evolved since 2009.
There are two things to take into account now: the number of cores, and the number of threads that can run within each core.
With Intel processors, the number of threads is defined by the Hyperthreading which is just 2 (when available). But Hyperthreading cuts your execution time by two, even when not using 2 threads! (i.e. 1 pipeline shared between two processes -- this is good when you have more processes, not so good otherwise. More cores are definitively better!) Note that modern CPUs generally have more pipelines to divide the workload, so it's no really divided by two anymore. But Hyperthreading still shares a lot of the CPU units between the two threads (some call those logical CPUs).
On other processors you may have 2, 4, or even 8 threads. So if you have 8 cores each of which support 8 threads, you could have 64 processes running in parallel without context switching.
"No context switching" is obviously not true if you run with a standard operating system which will do context switching for all sorts of other things out of your control. But that's the main idea. Some OSes let you allocate processors so only your application has access/usage of said processor!
From my own experience, if you have a lot of I/O, multiple threads is good. If you have very heavy memory intensive work (read source 1, read source 2, fast computation, write) then having more threads doesn't help. Again, this depends on how much data you read/write simultaneously (i.e. if you use SSE 4.2 and read 256 bits values, that stops all threads in their step... in other words, 1 thread is probably a lot easier to implement and probably nearly as speedy if not actually faster. This will depend on your process & memory architecture, some advanced servers manage separate memory ranges for separate cores so separate threads will be faster assuming your data is properly filed... which is why, on some architectures, 4 processes will run faster than 1 process with 4 threads.)

The answer depends on the complexity of the algorithms used in the program. I came up with a method to calculate the optimal number of threads by making two measurements of processing times Tn and Tm for two arbitrary number of threads ‘n’ and ‘m’. For linear algorithms, the optimal number of threads will be N = sqrt ( (mn(Tm*(n-1) – Tn*(m-1)))/(nTn-mTm) ) .
Please read my article regarding calculations of the optimal number for various algorithms: pavelkazenin.wordpress.com

The actual performance will depend on how much voluntary yielding each thread will do. For example, if the threads do NO I/O at all and use no system services (i.e. they're 100% cpu-bound) then 1 thread per core is the optimal. If the threads do anything that requires waiting, then you'll have to experiment to determine the optimal number of threads. 4000 threads would incur significant scheduling overhead, so that's probably not optimal either.

I thought I'd add another perspective here. The answer depends on whether the question is assuming weak scaling or strong scaling.
From Wikipedia:
Weak scaling: how the solution time varies with the number of processors for a fixed problem size per processor.
Strong scaling: how the solution time varies with the number of processors for a fixed total problem size.
If the question is assuming weak scaling then #Gonzalo's answer suffices. However if the question is assuming strong scaling, there's something more to add. In strong scaling you're assuming a fixed workload size so if you increase the number of threads, the size of the data that each thread needs to work on decreases. On modern CPUs memory accesses are expensive and would be preferable to maintain locality by keeping the data in caches. Therefore, the likely optimal number of threads can be found when the dataset of each thread fits in each core's cache (I'm not going into the details of discussing whether it's L1/L2/L3 cache(s) of the system).
This holds true even when the number of threads exceeds the number of cores. For example assume there's 8 arbitrary unit (or AU) of work in the program which will be executed on a 4 core machine.
Case 1: run with four threads where each thread needs to complete 2AU. Each thread takes 10s to complete (with a lot of cache misses). With four cores the total amount of time will be 10s (10s * 4 threads / 4 cores).
Case 2: run with eight threads where each thread needs to complete 1AU. Each thread takes only 2s (instead of 5s because of the reduced amount of cache misses). With four cores the total amount of time will be 4s (2s * 8 threads / 4 cores).
I've simplified the problem and ignored overheads mentioned in other answers (e.g., context switches) but hope you get the point that it might be beneficial to have more number of threads than the available number of cores, depending on the data size you're dealing with.

4000 threads at one time is pretty high.
The answer is yes and no. If you are doing a lot of blocking I/O in each thread, then yes, you could show significant speedups doing up to probably 3 or 4 threads per logical core.
If you are not doing a lot of blocking things however, then the extra overhead with threading will just make it slower. So use a profiler and see where the bottlenecks are in each possibly parallel piece. If you are doing heavy computations, then more than 1 thread per CPU won't help. If you are doing a lot of memory transfer, it won't help either. If you are doing a lot of I/O though such as for disk access or internet access, then yes multiple threads will help up to a certain extent, or at the least make the application more responsive.

Benchmark.
I'd start ramping up the number of threads for an application, starting at 1, and then go to something like 100, run three-five trials for each number of threads, and build yourself a graph of operation speed vs. number of threads.
You should that the four thread case is optimal, with slight rises in runtime after that, but maybe not. It may be that your application is bandwidth limited, ie, the dataset you're loading into memory is huge, you're getting lots of cache misses, etc, such that 2 threads are optimal.
You can't know until you test.

You will find how many threads you can run on your machine by running htop or ps command that returns number of process on your machine.
You can use man page about 'ps' command.
man ps
If you want to calculate number of all users process, you can use one of these commands:
ps -aux| wc -l
ps -eLf | wc -l
Calculating number of an user process:
ps --User root | wc -l
Also, you can use "htop" [Reference]:
Installing on Ubuntu or Debian:
sudo apt-get install htop
Installing on Redhat or CentOS:
yum install htop
dnf install htop [On Fedora 22+ releases]
If you want to compile htop from source code, you will find it here.

The ideal is 1 thread per core, as long as none of the threads will block.
One case where this may not be true: there are other threads running on the core, in which case more threads may give your program a bigger slice of the execution time.

One example of lots of threads ("thread pool") vs one per core is that of implementing a web-server in Linux or in Windows.
Since sockets are polled in Linux a lot of threads may increase the likelihood of one of them polling the right socket at the right time - but the overall processing cost will be very high.
In Windows the server will be implemented using I/O Completion Ports - IOCPs - which will make the application event driven: if an I/O completes the OS launches a stand-by thread to process it. When the processing has completed (usually with another I/O operation as in a request-response pair) the thread returns to the IOCP port (queue) to wait for the next completion.
If no I/O has completed there is no processing to be done and no thread is launched.
Indeed, Microsoft recommends no more than one thread per core in IOCP implementations. Any I/O may be attached to the IOCP mechanism. IOCs may also be posted by the application, if necessary.

speaking from computation and memory bound point of view (scientific computing) 4000 threads will make application run really slow. Part of the problem is a very high overhead of context switching and most likely very poor memory locality.
But it also depends on your architecture. From where I heard Niagara processors are suppose to be able to handle multiple threads on a single core using some kind of advanced pipelining technique. However I have no experience with those processors.

Hope this makes sense, Check the CPU and Memory utilization and put some threshold value. If the threshold value is crossed,don't allow to create new thread else allow...

How do I compensate for not having a "quiet" machine when benchmarking my Java application?

I run numerical simulations all the time. I can tell if my simulations don't work (i.e., they fail to give acceptable answers), but because I typically run a variable number of these on designated cores running in the background (as I work), looking at clock time tells me less than nothing about how quickly they ran.
I don't want clock time; I want CPU time. None of the articles seems to mention this little aspect. In particular, the recommendation to use a "quiet" machine seems to blur what's being measured.
I don't need a great deal of detail, I just want to know that simulation A runs about 15% faster or slower than simulation B or C, despite the fact that A ran by itself for a while, and then I started B, followed by C. And maybe I played for a little while before retiring, which would run a higher-priority application for part of that time. Don't tell me that ideally I should use a "quiet" machine; my question specifically asks how to do benchmarking without a dedicated machine for this. I also do not wish to kill the efficiency of my applications while measuring how long they take to run; it seems that significant overhead would only be required when a great deal of detail is needed. Am I right?
I want to modify my applications so that when I check whether a batch job succeeds, I can also see how long it took to reach these results in CPU time. Can benchmarking give me the answers I'm looking for? Can I simply use Java 9's benchmarking harness, or do I need something else?

You can measure CPU time instead of wall-clock time from outside the JVM easily enough on most OSes. e.g. time java foo.jar on Unix/Linux, or even perf stat java foo.jar on Linux.
The biggest problem with this is that some workloads have more parallelism than others. Consider this simple example. It's unrealistic, but the math works the same for real programs that alternate between more-parallel and less-parallel phases.
version A is purely serial for 9 minutes, and keeps 8 cores saturated for 1 minute. Wall-clock time = 10 minutes, CPU time = 17 minutes
version B is serial for 1 minute, and keeps all 8 cores busy for 5 minutes. Wall time = 6 minutes, CPU time = 5*8 + 1 = 41 minutes
If you were just looking at CPU time, you wouldn't know which version was stuck on an inherently serial portion of its work. (And this is assuming purely CPU-bound, no I/O waiting.)
For two similar implementations that are both mostly serial, though, CPU time and wall time could give you a reasonable guess.
But modern JVMs like HotSpot use multi-threaded garbage-collection, so even if your own code never starts multiple threads, one version that makes the GC do more work can use more CPU time but still be faster. That might be rare, though.
Another confounding factor: contention for memory bandwidth and cache footprint will mean that it takes more CPU time to do the same work, because your code will spend more time waiting for memory.
And with HyperThreading or other SMT cpu architectures (like Ryzen) where one physical core can act as multiple logical cores, having both logical cores active increases total throughput at the cost of lower per-thread performance.
So 1 minute of CPU time on a core where the HT sibling is idle can get more work done that when the other logical core was also active.
With both logical cores active, a modern Skylake or Ryzen might give you somewhere from 50 to 99% of the single-thread performance of having all the execution resources available for a single core, completely dependent on what the code is running on each thread. (If both bottleneck on latency of FP add and multiply with very long loop-carried dependency chains that out-of-order execution can't see past, e.g. both summing very large arrays in order with strict FP, that's the best case for HT. Neither thread will slow the other down, because FP add throughput is 3 to 8x FP add latency.)
But in the worst case, if both tasks slow slow down a lot from L1d cache misses, HT can even lose throughput from running both at once on the same core, vs. running one then the other.

Why is VisualVM running 50 threads for VisualVM itself?

Update
A different way to ask this question is "Is there ever a point to having more threads available to the application than there are cores?".
Original Question
I have a total of 8 cores on my laptop. I just downloaded the latest version of VisualVM and clicked on the VisualVM tab to see how it is performing. One of the things I noticed is that it's using 32 live threads and 22 daemon threads. Why is it using so many? IIUC using more than 8 is less efficient, and IIUC the JVM knows how many cores I have so it could just limit the total number of threads to 8.
Someone marked this as a duplicate, so I'll try to explain what I'm asking a little differently. I'm not asking how to use a particular feature of VisualVM. I'm asking why it is running more than 8 threads, since IIUC more than 8 would be less than optimal and VisualVM comes from Oracle so I figured they would design it to function optimally, so I'm wondering if I'm missing something, or is Oracle just allowing it to run with more threads than it should be running with?

Just because you have 50 threads of VisualVM running, it doesn't necessarily mean that all of those threads are supposed to be working actively at the same time. If you sort on the "Running" column, you will see that only the first 7 or 8 threads are actively running. The rests of the threads are sleeping or waiting for I/O.

How to handle thousands of threads in Java without using the new java.util.concurrent package

I have a situation in which I need to create thousands of instances of a class from third party API. Each new instance creates a new thread. I start getting OutOfMemoryError once threads are more than 1000. But my application requires creating 30,000 instances. Each instance is active all the time. The application is deployed on a 64 bit linux box with 8gb RAM and only 2 gb available to my application.
The way the third party library works, I cannot use the new Executor framework or thread pooling.
So how can I solve this problem?
Note that using thread pool is not an option. All threads are running all the time to capture events.
Sine memory size on the linux box is not in my control but if I had the choice to have 25GB available to my application in a 32GB system, would that solve my problem or JVM would still choke up?
Are there some optimal Java settings for the above scenario ?
The system uses Oracle Java 1.6 64 bit.

I concur with Ryan's Answer. But the problem is worse than his analysis suggests.
Hotspot JVMs have a hard-wired minimum stack size - 128k for Java 6 and 160k for Java 7.
That means that even if you set the stack size to the smallest possible value, you'd need to use roughly twice your allocated space ... just for thread stacks.
In addition, having 30k native threads is liable to cause problems on some operating systems.
I put it to you that your task is impossible. You need to find an alternative design that does not require you to have 30k threads simultaneously. Alternatively, you need a much larger machine to run the application.
Reference: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2012-June/003867.html

I'd say give up now and figure another way to do it. Default stack size is 512K. At 30k threads, that's 15G in stack space alone. To fit into 2G, you'll need to cut it down to less than 64K stacks, and that leaves you with zero memory for the heap, including all the Thread objects, or the JVM itself.
And that's just the most obvious problem you're likely to run into when running that many simultaneous threads in one JVM.

I think we are missing lots of details, but would a distributed plateform would work? Each of individual instances would manage a range of your classes instances. Those plateform could be running on different pcs or virtual machines and communicate with each other?

I had the same problem with an SNMP provider that required a thread for each outstanding get (I wanted to have tens of thousands of outstanding gets going on at once). Now that NIO exists I'd just rewrite the library myself if I had to do this again.
You cannot solve it in "Java Code" or configuration. Windows chokes at around 2-3000 threads in my experience (this may have changed in later versions). When I was doing this I surprisingly found that Linux supported even less threads (around 1000).
When the system stops supplying threads, "Out of Memory" is the exception you should expect to see--so I'm sure that's it--I started getting this exception long before I ran out of memory. Perhaps you could hack linux somehow to support more, but I have no idea how.
Using the concurrent package will not help here. If you could switch over to "Green" threads it might, but that might take recompiling the JVM (it would be nice if it was available as a command line switch, but I really don't think it is).

Thread number and Java application performance

Hi: I have a multi thread Java application. The current thread size is already 100. We are currently using 4 core CPU. But as one see in the near future, CPU core would be doubled, or even to 32 cores. In order to fully utilize cores, we need to increase our thread pool size. But as you may know (Maybe I am wrong), Java is good when there is 100 hundred threads, but there could be performance problem when thread is 200, 500, 1000 threads. Then shall we use other programming language, for example scala. Is my worry reasonable?

With modern JVMs, a Java process can create as many threads as the operating system will permit. Whether or not your application will be able to make good use of those threads depends on the design of your application.
If scalability is a concern, I would recommend that in the first instance you focus on your application's architecture (data structures, synchronization, etc). These issues need to be considered irrespective of the programming language, and there's nothing about Java that makes it inherently unsuitable for heavily multithreaded apps.

I once made experiments with threads, to find out, whether there is significant difference between Linux and Windows, and hit a kind of barrier at about 2000 threads on both platforms. The test is some years old, and I didn't repeat it, but later I found the same number mentioned by others, but I didn't save the link.
Without testing it, I think you're right about scala. The techniques used there - Actors - works with smaller objects, afaik, but I can't give you numbers.

If you have 4 cores, the optimal thread pool size may be 4 as this is the minimum number of threads required to keep all the CPUs busy. However, you can have any number of idle/waiting threads up to about 10K. This is a JVM thread library tipping point so switching to Scala won't make any difference. Note: you can have far more threads, I wouldn't recommend it.
If you have 10K threads and you want more, I suggest you buy another server. You can buy a lot for server for about $1000.
I ran a test creating lots of threads on my machine with Java 6 update 26, 32-bit and 64-bit on Ubuntu 11. The first 1000 threads took 72 ms to create, to go from 31K to 32K, took 3,861 ms to create. At about 32K threads I got this error
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)

#user84592: not sure about my answer, just brainstorming.
How about having installing virtual machine software on this machine, distributing CPU cores to them, it will make many machines instead of having one physical machine, and then you can have java application workload sliced to each of them...