I'm not a Java developer but I find myself wearing that hat today, so bear with me.
When I stopped working on a file parser last night I was seeing benchmarks of 100k records per second-- which I was more than happy with. When I re-ran the same code against the same files this morning, I'm only seeing 10-12k records per second.
First thoughts might be that I changed something and introduced inefficient code but I've commented out larger swaths of the code than I had running last night and performance is still abysmal. At this point the program does virtually nothing except read files in the main loop and still reads them slowly.
I had a coworker run the fully-functional version of the jar on his own machine and he is seeing the 100k/s performance benchmarks I saw last night, so i can only assume something is wrong with my JVM/workstation.
Any ideas or thoughts on what I should be looking at? I hesitate to get into JVM performance tuning when I already know the stock JVM is more than capable of doing this task. I just don't understand what's changed since last night.
EDIT: I have rebooted the machine.
EDIT 2: It is now the next day. I booted up my laptop and ran the code, it's back to where it was in the first place-- 100k/ps. I checked out Windows' performance monitor yesterday and it didn't show an unusual amount of CPU, RAM or disk I/O so I'm really at a loss as to why this happened.
Perhaps I will look into JVM tuning after all if only to ensure I have a consistent experience.
Problem seems to be resolved.
Since posting, one thing I've done is eliminate the Windows pagefile in case this was a matter of paging. It helped but not as much as I needed.
What seems to have made the biggest impact is dedicating a sizable amount of memory to the JVM upon launch. I'm guessing the JVM might have been struggling to allocate enough memory dynamically. I'm now running it with the following parameters:
-server -Xmx4096m -Xms4096m
I added the server flag since this is intended to be a long-running process and I don't care about start-up time, and the memory arguments should give the JVM a solid, static 4GB of memory to work with.
I've not had any performance issues since.
Related
I've been stuck on my problem for quite some time. To give you a litte context, I have written a bot in Java and was planning to run it on a Raspberry Pi 3 Model A+ 24/7. To my surprise, when I tested the almost finished program, its memory consumption kept on rising indefinitely.
Soon I realised, I had to limit the memory usage which I looked up on several sites over the past couple months. Unfortunately, most of them are outdated (2013 and older) and the very few newer ones didn't cover the important changes which must have taken place because I'm not able to figure out why my issue is still occurring.
I've tried so many things over such a long period of time that I'm not sure if I'll be able to sum up all the things I've tried so far but will update this post if I remember some important details.
Please see the pictures of my last test with the following settings:
java -Xmx4m -Xms4m -Xss64k -XX:MaxMetaspaceSize=8m -jar bot.jar
As you see the memory was not limited and rose to the point where the process was killed shortly after. In some of my previous tests I used an empty while(true) loop because I don't think I have a memory leak in my program. Weirdly enough, the empty loop also increased the memory size very slowly but did also not decrease over time.
At this point I'm not sure if the JVM is even capable of having a specified memory limit. My code uses the Robot class to make screen captures and fire certain buttons in nested while loops which also remind the garbage collector to cue a collection with System.gc(). Do I also have to use the following argument for the JVM?
-XX:MaxDirectMemorySize
I'm really confused with all the changes on Java as well. I've tried a few different JDKs because I thought that might solve the problem. The last test was compiled with the JDK 14 and runs on Java 11. Do I need to configure something on the OS in order to limit the memory consumption?
Maybe you guys could also recommend me a profiler with which I can check what is even allocating the memory in order to figure out what needs to be limited via the arguments but I would definitely need some help because I have never worked with one before.
I appreciate any help on this topic! Please let me know if you need any additional information and I will do my best to follow up during the week.
Maybe you can use the following args : -XX:+PrintGCDetails -Xloggc:/tmp/jvm-gc.log. It will log gc details in /tmp/jvm-gc.log .
Or you can check the size of the runtime heap with the following command：
# get the pid
ps -aux | bot.jar
# show the heap info
jmap -heap <pid>
I am having a very strange problem with Java that I have been unable to make any progress in fixing. I have designed a small application for generating and viewing simple fractals, it is part of a coursework. When I run the code from Eclipse it runs very quickly, generally using <5% of my processor and acting generally very responsive.
However when I run this same program from the windows command line, after compiling it of course, it runs much more slowly. It uses about 20% of the processor and if I was to put a figure on it I would say it runs about 10x slower, and is generally unusable. I have done a good bit of research on this problem and it hasn't been easy to come across relevant information, it certainly doesn't seem to be a common bug. The times it has been asked before differ from my situation in that people are writing an excessive amount to the console, which causes the slowdown. I am not printing anything to the console.
The process started by Eclipse when the code is run uses up to about 700mb of RAM. The code ran from the console uses up to about 70mb. I tried running it with a greater heap size, which did increase the RAM the process uses but did virtually nothing to increase the performance.
I would really appreciate it if anybody could help with this issue, I am tearing my hair out here.
Thanks very much!
We have a server application with java IO operations. When running the application, we have observed CPU usage for CompilerThread0 and CompilerThread1 taking 45% and 41% CPU. The application is serving IO to the clients at this time like connect, receive and send. As far as I explored related to this I found that the compilerthread is for JIT and for increasing performance.
My question is first, why it is taking much CPU for the compilerthreads and how to minimize this so that we can give CPU to other threads.
Thanks in advance!
My question is first, why it is taking much CPU for the compiler threads
Under normal circumstances, the JIT compiler should kick in after your application has been running for a bit to (progressively) compile the classes / methods that are called frequently. Compilation activity should die down ... once all code that needs to be compiled has been compiled.
If the compilation activity doesn't die down, then something strange is happening. It could be one of the following:
Your application is making a lot of use of dynamic proxies, and you keep generating new proxy classes.
Your application is dynamically loading (and unloading) lots of classes.
You've encountered a JVM bug of some sort. (But I couldn't spot a Bug Database entry that matches these symptoms. So I'd call this "unlikely".)
and how to minimize this so that we can give CPU to other threads.
There are potentially JVM options that might help, but I think you'd be better off figuring out what is causing this.
I have been having occasional problems with a server I wrote. It's in Clojure, but I don't think that matters, and we can pretend it's in Java. Anyway, it works fine for hours at a time, but goes into fits where it behaves very badly: all activity stops, for around fifteen seconds, and then it works normally for a few seconds, then stops for fifteen seconds...and so on for (usually) about ten minutes or so, after which it goes back to behaving normally.
I've done a lot of profiling of it with YourKit, and I've ruled out a number of plausible suspects:
It's not a garbage collection issue: I'm running it with -XX:+UseConcMarkSweepGC, and I've verified that the server continues to run just fine during both minor and major collections, due to the concurrent nature of this garbage collector. And we're not thrashing as we run out of total memory or something: the current heap size is well below its max.
I don't think it's a locking/synchronization issue, but I'm not 100% sure on that. The YourKit profiler shows threads waiting sometimes, eg competing over the lock for System.out to produce log messages, but the only long waits are for worker threads in threadpools when there's nothing to do. And of course YourKit says it's never detected any deadlocks.
It's not something caused by having the profiler attached, because it still happens even if I boot the server up and then leave it alone without ever attaching the profiler.
It's not some other process on the system taking up all the CPU time: top shows CPU usage at 100% for my java process, and basically 0% for everything else.
My biggest problem is that I can't see what the server is doing during these strange funks, because the profiler stops receiving samples. Here's a graph of the CPU usage chart:
The left side of the graph is normal operation, during which we get profiler samples every second or so. The right side is "broken", and is very spiky because the profiler is only getting samples every ten seconds or so. In the samples it does get, the server seems to be doing its usual business: responding to requests and so on; and the logs confirm that it is doing normal stuff, but only at the times the profiler has samples for: during the upward-sloping "straight lines" on the graph, for which the profiler has no samples, the server is doing nothing at all.
So, does this graph look familiar to anyone? Have you had this problem before and fixed it? Or can you point me in the direction of a tool that can figure out what my server is doing during the time when YourKit can't? In case it matters, the server machine is running Ubuntu 10.04, and
$ java -version
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.10) (rhel-1.28.1.10.10.el5_8-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)
Okay, from the comments it seems clear to me we are not going to be able to figure this out with the information you've given so far. The best we can do is to give suggestions on how to debug it...
I would try to use jstack during one of the spikes and see if you can use that to figure out where it hangs.
If you have no chance to measure or debug in code try to look form the outside.
I would at first to try to reproduce the problem. In other words is there a external event that produce the behavior. Try to change the load on server. Switch every thing you can to reproduce the problem.
Maybe it's also a good idea to sniff the network traffic (tcpdump) to find something interesting around the time when you server hangs.
You can also run it on another operating system to check if it depends from your installation environment.
If you can't reproduce a situation where the problem occurs, try to find situations where you don't get the problem. For instance remove the server from net. Shutdown all other services.
If you can't find with that any change of behavior of your program try to reduce the complexity of your working code and see if you can find a internal module that seems to be related with the problem.
Have you had this problem before and fixed it? Or can you point me in
the direction of a tool that can figure out what my server is doing
during the time when YourKit can't?
If you have shell access on the server and can see stdout, try taking a thread dump when the server becomes unresponsive. Not sure if this will give you anything different than what jstack (mentioned in the other answer) would give you or not.
On Ubuntu: kill -QUIT <java-pid> (will not actually kill the Java process).
http://www.crazysquirrel.com/computing/java/basics/java-thread-dump.jspx
I have a java application that runs on Tomcat (which runs as a service on Windows), the java process for which continues to eat up CPU before eventually requiring me to restart the Tomcat service.
First my setup:
Windows 2003 server
Tomcat 6, running as service using Wrapper
JDK: 1.6.0_20
I was seeing catch issues here and there leading up to yesterday. I had to restart midday yesterday, then at 2:30 this morning, then today I could barely restart the application and open jconsole to monitor it before it was hitting 99% CPU usage again. Through a combination of things I'm not quite sure of, it seems like I got the JVM to cycle itself and the app was hovering in the 10-30% CPU usage range for a couple hours. However, then it started to creep up again, finally going into its 99% CPU usage breakdown. I was also having trouble with high memory usage, but that has stayed fairly normal and steady since I so-called got the JVM to "cycle" (bad terminology perhaps, but this is really what it seemed to do - and in the wrapper log there was a dump of all the classes it was reloading after).
Then I was digging around some more and found a JRE 6 Update 24 installed on the server (I didn't install it as I do thorough testing with each java update - but maybe my server admin did the update). I attempted, but can't uninstall this. Thus, I get different versions when I do a java -version versus javac -version
java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) Client VM (build 19.1-b02, mixed mode, sharing)
javac -version
javac 1.6.0_20
Could this difference be causing a JVM conflict of sorts? JAVA_HOME and my PATH variables both point to the correct JDK installation.
Hoping for more stability, I decided to change my app to run on the previous JDK that was still installed - JDK 1.6.0_04. I changed the wrapper.conf, set env variables, cleaned and rebuilt, and started. This does seem more stable and has been up for about 4 hours. The CPU usage has climbed to the 90s, then it seems to clear itself out again.
I've done heapdumps then ran them through the Memory Analyzer in Eclipse (nothing new found there), I've used jconsole with jtop to look at threads - nothing jumps out, thus why I continue to be curious if it's a java/jvm issue. So, I know this is a long post - but I don't really know where to go from here. Any ideas?
(I've done exhaustive web searching on this and some articles have pointed to possibly a Quartz issue or Hibernate queries not flushing. Nothing has changed in the app since I started seeing the CPU issues, so I'm not sure where to start troubleshooting if it could indeed be linked to either.)
This isn't an easy problem. You are doing all of the basics to see if it something jumps out. It sounds like there is either a slow leak that builds up over time to the point where it can't operate. That sounds like GC is thrashing and app comes unresponsive. It could also be runaway background job(s) eating on the CPU and just doesn't complete, that might explain the long delay. You could try turning off any quartz to see if it stays up longer that might help lead you in a direction, or crank it up so it shows up sooner.
I know you've done some jconsole watching, but I think you need to revisit and watch your memory usage, the threads run time, how much time you're spending in GC, and watching what portions of memory are being eaten up (is it Eden, Tenure that's running out?).
I'd make sure you are writing out start and end messages for your background jobs running in Quartz. Then you can correlate when they start and finish with when this problem starts. Also will tell you if your jobs are finishing or not.
It's probably time to drop it into a profiler (instead of jconsole) so you can see where in the code it's spending time or what's blowing up memory. A real profiler will let you see all that data mashed up on your code and classes. My favorites is JProfiler, but YourKit is also good. You can get a 7-30 day trial so you'll have plenty of time to profile and figure your issue out without having to buy it.
Start this early in the morning so you'll hopefully see something by early night.