My question is, do JVM's share some kind of resource related to threading or processes that could cause ProcessBuilder performance to spike after a month or more of normal usage? Using java 6 update 21 for all apps.
Over the past several months, we've noticed that a single server in our data center (Sparc M4000 running Solaris 10) can go about 6-8 weeks with no problems. Quickly, however, performance on an application that utilizes the ProcessBuilder class to run scripts takes a huge performance hit - with ProcessBuilder.start taking over a minute to return sometimes. After a reboot, and for several weeks after, normal return time is in the 10s or maybe 100 millisecond range.
I wrote a separate small application that creates 5 threads, and each thread runs the 'ls' command using ProcessBuilder 10 times serially, then I gather stats from that in order to monitor the original problem. This application exits after each run, and is run from cron only once an hour. It usually only takes a second or two.
Last night, ProcessBuilder times spiked again to over a minute for each ProcessBuilder.start call, after 45 days of uptime and normal behavior.
top shows no memory or CPU hogs. I did try to do a jstack on the test app, but got the error 'Can't create thread_db agent'.
Any ideas?
We had a similar problem on our application which runs in Linux. The Linux JVM code uses a fork which means the address space gets mapped and copied each time you exec. We were executing many small short lived processes. It appears a main difference from your app is that we had a relatively large heap (around 240GB) so I'm sure that had an impact. We ended up implementing our own spawning code using JNI and posix spawn. Here is a link to the question/answer: Slowing process creation under java
Related
I have a number of Java processes using OpenJDK 11 running on Windows Server 2019. The server has two physical processors and 36 total cores; it is an HP machine. When I start my processes, I see work allocation in Task Manager across all the cores. This is good. However after the processes run for some period of time, not a consistent amount of time, the machine begins to only utilize only half the cores.
I am working off a few theories:
The JDK has some problem that is preventing it from consistently accessing all the cores.
Something with Windows Server 2019 is causing a problem, limiting Java from accessing all the cores.
There is a thermal management problem and one processor is getting too hot and the OS is directing all the processing to the other processor.
There is some issue with hyper-threading and the 'logical' processors that is causing the process to not be able to utilize all the cores.
I've tried searching for JDK issues and haven't found anything like this mentioned. I went down to the server and while it's running a little warm, it didn't appear excessively hot. I have not yet tried disabling hyper-threading. I have tried a number of parameters to force the JVM to use all the cores and indeed the process initially does use all the cores; I can see the activity in Task Manager.
Anyone have any thoughts? This is a really baffling problem and I'd appreciate any ideas.
UPDATE: I am able to make it use the other processor by using the Task Manager to assign one of the java.exe processes to the other processor. This is also working from the java invocation on the command line as well with an argument for which socket to use.
Now that said, this feels like a hack. I don't see why I should have to manually assign a socket to each of my java processes; that job should be left to the OS. I'm still not sure exactly where the problem is, if it's the OS or what.
So according to most things I've read on the internet, the number of threads you can have in Java caps out around 10,000. However, in practice I can create nearly 500,000, at which point my computer becomes unresponsive. (The task manager goes a little funny - it starts claiming that though 99% of my 16 GB of memory is used, the highest-using program uses only ~300 MB. After everything stops responding, the fan quiets down, and the disk access light flashes only periodically, leading me to believe neither CPU nor disk is under heavy load.) I waited for about 15 minutes one test, and never got an exception (well, as far as I know).
For repeatability, I've (also) used the following code: https://github.com/jheusser/core-java-performance-examples/blob/master/src/test/java/com/google/code/java/core/threads/MaxThreadsMain.java as referenced here: https://dzone.com/articles/java-what-limit-number-threads .
I did, however, increase the upper limit on i from 100 * 1000 to 1000 * 1000, because it was successfully creating all the threads. One of the last messages it gave before the computer froze up was 440,000 threads: Time to create 4,000 threads was 1.002 seconds - it looks like it was averaging around 2 seconds per 4000, though.
I am using Windows 10 Pro, version 1703.
JRE: Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)
The next highest thread count I know of is about 100k, https://stackoverflow.com/a/46697264/513038. Now, a lot of the claimed limits were given many years ago, but they're based on stack size vs memory, and at 500,000 threads in 16 GB RAM (even assuming ALL of it was used), that's 32kb per thread by default, which is supposedly less than the minimum stack size. If that were true, I'd at least expect more StackOverflowErrors during normal operation. Has the threading system changed silently in the past 10 years? (Or even in the past few months: one of the posts I referenced was made just a few months ago, April 2018.)
Has the threading system changed silently in the past 10 years?
Nope. On Linux, MacOS and Windows, Java threads are implemented as native threads ... since a long time ago.
What has changed is the way that various different operating systems schedule native threads. The OS is where Java thread scheduling takes place, and where any hard limits on the number of threads supported will be enforced.
Basically, your tests try to see what happens when you try to use a pathologically large number of threads. The answer on Windows is that it breaks the OS.
And even if it didn't break the OS out-right, the chances are that for a Java application using 100,000's of threads:
the JVM's resource usage (stack memory) would be terrible,
native scheduler performance would be terrible, and
the application performance would be terrible.
Huge numbers of threads is the wrong way to write a practical Java application. Actors may be a better solution, or maybe an ExecutorService (with a bounded thread pool) or a ForkJoin pool. It will depend on the application, and other factors.
In short, those tests you are running are not instructive for a properly designed Java application. The solution for applications that use huge numbers of threads is to rewrite them.
I'm profiling a Java application using Java Mission Control, and it's saying on the main page of the flight recording that "This recording contains few profiling samples even though CPU load is high.
The profiling data is thus likely not relevant."
It seems to be telling the truth. I asked it to sample every 10 ms for 3 minutes which should be 18000 samples, but I only see 996 samples.
It goes on to explain "The profiling data is thus likely not relevant. This might be because the application is running a lot JNI code or that the JVM is spending a lot of time in GC, class loading, JIT compilation etc."
Hmm, I don't have any native methods, and it shouldn't be loading classes or doing any JIT at the stage I recorded (well into the repetitive number crunching part of the code.) It doesn't look like it's spending an inordinate amount of time garbage collecting either.
We used to use hprof to profile this product, with much success. Hprof helped immensely in figuring out where we were relying on the main thread execution, so we could parallelize the hotspots into multiple threads. But that tool got discontinued in Java 9 so we're moving onward to Java Mission Control. It has a lot going for it, but if it can't identify what line numbers the VM threads are sitting on at random sample times, it's not very useful. Is there some other tool to use? Or, is there a way to debug this further from within Java Mission Control? It also looks like JVisualVM is no longer included in Java 9.
If you have many more running threads than cores, the sampling thread could be starved and not able to wake up at the interval you specified.
The answer is probably as simple as you having more threads than cores, and thus most of them not being scheduled on CPU at the time of sampling. The JFR method sampler will only keep samples of threads actually on CPU. The idea is to provide you of a view of where you are actually spending the time executing your Java code.
Now, we know that there are cases where you want to get random samples of all threads, no matter what they are doing. We are adding new profiling capabilities/events in JDK 10.
Need some help from the experts!
We have a project here (still on dev) that needs to run 50 java processes (for now and it will probably doubled or tripled in the future) at the same time every 5 minutes. I set Xmx50m for every process and our server has only 4gb of RAM, I know that would really slow our server. What I have in mind is to upgrade our RAM. My question is that do I have other options to prevent our server from being slow when running that amount of java processes?
Since you have 50 process and as per your assumption your processes need about 2.5 Gb to run .
To prevent your server from being slow you can follow some best practices to set java memory parameters e.g. set -Xmin and -Xmx the same values and determine a proper values based on your process usage, Also you can profile your process on runtime to ensure that everything is ok.
I am making use of java visualvm for profiling purpose. I am facing two issues:
1) As statetd in the documentation
When a new local Java application is launched, a node for that application appears under the Local node. The application node disappears when the application terminates.
This things happens with me when I run my java application, it runs for seconds. The Visualvm shows this PID but when my program stops then this PID also disappears.
How can I solve this problem?
Secondly, I run the main function in which there are calls to other functions present in other classes. I want to compute time for those functions as well . How can I achieve this task?
is not a problem. When java process terminates you no longer can connect to it.
you could run the same code 1000 times and make a snapshot in visualVM to analyze usages of memory, computations, etc. You'll have to divide the results by 1000.
This gives you better results (taking time of 1000 runs and getting average), since single run can take different time at each run.