Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
We have apps that are deployed in production [Java / Scala]. We have alerts setup when ever there is a spike in CPU usage or memory usage.
Occasionally we see a huge spike in CPU or memory usage. Some times the application that is running on play stops responding to request.
I usually see the logs for last few API hits before the crash, that way I recently figured out one of API was downloading huge dump of data and memory got exhausted.
Can I get tips for troubleshooting the issues in general [commands / tools to capture stats] when things go wrong in prod?
This requires a lot of experience though. Below are some steps that you could follow:
Prerequisite:
You should understand java Memory Model i.e. what's New Generation(Eden, Survivor-01,Survivor-02), Old Generation, Meta Space, Heap, Stack etc.
Read this to understand it better.
You should understand how Garbage collection works. e.g. you should understand how Mark and Sweep algorithm works. Check the same link as above for same.
Now you could install visual VM. Also, in visual vm install a plugin visual gc it will show you memory used in different space. You will see another tab Visual GC
i) Observe Graphs(Heap one to top right in the snapshot below) in Monitor Tab.
**Trick: ** You could perform manual GC as well to observe how steep the graph line for Used Heap Space is and how quickly it fills up at running some block of code. I used it many times and it really helps (Especially if used with the debugger)!
ii) Also, try to observe the Thread Dump if multithreading is causing some issue.
iii) In any case, you could also do some profiling or sampling via profiler and sampler tab.
Below is a snapshot of sampler. See how clearly it tells how much memory is taken by what data type:
Important: Screenshot is of the heap. You could change to Per Thread Allocation tab to see per thread allocation.
Similarly, you could observe CPU consumption.
Alternatively, use JMeter if you think locally you are not able to reproduce the same. Jmeter can help you extensively load test your application basically.
Also, if you have integrated any server monitoring tool that could also be helpful. You could easily get notified for a problematic code.
At last, you could download the heap dump from the production system and analyze that on local using visual vm.
jmap -J-d64 -dump:format=b,file=<heap_dump_filename> <pid>
this link have more detailed answers from some really cool developers on same.
Use jstat. It comes with java and is very handy sometimes.
jstat -gc 2341 //2341 is the java process id.
These are from my experience. But In this direction, there would never be enough and I believe my knowledge keeps on evolving as I face more such issues. Hence, please practice it and explore further.
Having said that, there are other tools available so also feel free to find other ones that suit your needs well. To get started take a look at Jconsole.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I am currently working on an event driven System with multiple components running. Recently , I have received an urgent requirement to identify the memory consumption of java components running , so that we can give a brief idea of memory requirements before it is getting deployed on UAT/customer production environments.
Do we have any API using which Deep retained size can be calculated or a formula can be provided using which memory requirements can be computed.
Any ideas on this will surely help.
I have seen some API's ( java instrumentation Api) using which Shallow size can be calculated , but this will not suffice my need.
I also found java Assist using which java byte code can be modified at runtime.
To identify the memory consumption of a java aplication, you can use a profiler.
In jdk 6 or greater you can find jvisualvm (https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jvisualvm.html).
With jvisualvm, you can attach to a java process and, in sampler tab, you can see the memory consumed grouped by class type.
There are even other powerful profilers (JProfiler is one of them)
Enable garbage collection logging and analyze the log. As a bonus you will also be able to identify (and fix) aberrant behaviour.
To turn on gc logging, use the following flags:
-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintTenuringDistribution
-XX:+PrintGCCause
-Xloggc:/gc-%t.log
This log file can then be handled in a number of tools like Censum from JClarity or uploaded to https://gceasy.io/ for easy analysis. Note that you will see the memory consumption as a whole for the app, not a breakdown. For that you will have to use something like VisualVM mentioned above.
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 8 years ago.
Improve this question
I have develop a project which has more than 3000 source files; when I want to run my project after few minutes it gives me error like java.lang.OutOfMemoryException: java heap space. I also increased the memory of my project by right clicking and VM option I gave the 1024MB I have two 2gb on my PC.
As you likely know you can increase the JVM's memory using java -Xms<initial heap size> -Xmx<maximum heap size>.
However this just delays the issue as there is likely a place in your application where is memory is leaking causing the heap overflow. I suggest you use a tool like Netbeans Profiler which will help you find out where the memory leak is occurring. Netbeans Profiler will allow you to see where objects are created, where garbage collection occurs, etc.
It may be that the application just allocates a lot of memory or there is an actual leak. My approach would be to use a memory analyzer such as Eclipse MAT to see what objects are taking up the most memory.
If they all seem valid then you probably need to increase the heap size space. Though I had worked with 5000+ class web app projects with 512MB heap, so I wouldn't doubt that it was a memory leak.
You should also look for instances of ByteArrayOutputStream in your code, they tend to take up a large chunk of memory as well.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I have a variety of Rails applications running on Torquebox. Occasionally, Torquebox will stop responding to requests for a short period of time (maybe 2-5 minutes) and the logs will fill up with the following error messages:
java.lang.OutOfMemoryError: Direct buffer memory
The errors happen at unpredictable times (often days between). Load testing doesn't reproduce the problem, and the issue doesn't happen during peak load anyway. And, unlike many other types of memory errors I've seen in the past, the server actually recovers and starts responding normally again without any sort of restarts or intervention.
Are there any common coding mistakes, misconfigurations, or other potential problems that regularly cause this error? Google reveals lots lower level Java/Garbage collection type issues with various libraries (Netty for example), but I'm interested to see if there are other common places to look as well.
JNA/ByteBuffer not getting freed and causing C heap to run out of memory indicates that the direct memory might not be cleared if the Java heap doesn't require garbage collection very often (perhaps, in your case at non-peak times).
If you have a constant function using the direct memory, regardless of load, then the application might not be calling the garbage collection enough during the lighter load times. Using the GC module might help (http://ruby-doc.org/core-2.0/GC.html).
The problem in this case ended up being lack of enough total available memory. Java's memory footprint can end up being significantly larger than just the heap allocation. Direct Buffer Memory is only one of the types of memory use that falls outside the heap.
It's still not clear to me why the fluctuations occur at unpredictable times, but ensuring there is enough excess memory on the system to handle some fluctuation is critical to stability.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
This is more of a general programming questions about a GIANT performance issue I have seen..
Basically I'll use two common programs for an example - Eclipse IDE & Newsbin (usenet client)
On my Windows 7 Machine, Eclipse is so sluggish it's almost painful to use and this is built on the java platform right?
Whilst Newsbin, on the same machine, can handle hundreds of thousands of header information and literally NEVER lags.. It's one of the most responsive programmed I have ever used..
So, is there any chance someone could shed some light on which language/platform Newsbin is built upon? I'm curious because I want to expand my skills into desktop applications and there seems to be such a massive difference in performance.
Apologies if this type of question shouldn't be posted here, but it is 'linked' with programming and I would very much like some feedback/answers.
Thanks.
There are many reason the performance could be different. It is most likely a tuning problem or you hardware doesn't suit the application. I use IntelliJ CE (another IDE like Eclipse) and it caches a lot of information about the Java classes it uses. It does this to provide rich refactoring/search capabilities. This can result in enormous amounts of disk activity if you don't have lots of free memory (to cache the disk data). I use a machine with 48 GB of memory and it almost never lags (at least not when I am the only one using it)
My guess is that newsbin of keeps the most essential information about each post and avoids having to cache lots of information about each article. i.e. its has a completely different use and usage pattern.
The performance between the two is most likely not the platform but the fact that the two are very different application.
Second, two version of the same program can be vastly different. You can create a slower Newsbin type application in the same platform that your Newsbin's application uses.
You're comparing apples to oranges: these two programs do completely different things and the performance difference probably has nothing to do with the underlying platform or language.
Also, keep in mind that Eclipse can be fast by itself, but you can ad plugins to it, and poorly written plugins can slow it down horrendously.
Remember: no matter what language you're using, you can always find a way to write code that is poor enough to make a program feel slow and unresponsive.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
In response to a question about examples of Java usages, I bumped across some articles where NASA used Java for ground control in a mission to Mars but I couldn't find out if it has ever been used outside of Earth. Do you know of any such instances?
Here you will find a paper discussing the current state of using Java in space applications:
REAL-TIME JAVA IN SPACE: POTENTIAL BENEFITS AND OPEN CHALLENGES
They say:
In short time, it is unlikely that
with current Java implementations the
whole spacecraft on-board software
could be written in Java. However,
Java could be used as an isolation
platform for software that has not
been assigned the highest criticality,
while the critical code would still be
written in Ada or C.
Here one of the same authors reports on a successful test flight of an unmanned aerial vehicle (UAV) that uses Ovm (open source RTSJ (Real Time Specification for Java) implementation):
A Real-time Java Virtual Machine for Avionics
One fact often overlooked is that most space probes use older processors. They simply cannot run current commercial software. They are not fast enough or powerful enough. IC chips in space are exposed to high levels of cosmic rays. They energetic particles can destroy the small nanometer components on modern CPUs and other IC chips. To work reliably in space you either need specially designed ICs with redundant components, spend a lot on heavy shielding, or use older IC chips which have larger components that can take a hit from a cosmic ray without being destroyed.
Most of these older CPUs will run with a RTOS such as VxWorks and are either programed in assembler or using a language like C and still get decent performance.
Probably on the documentation laptops they take up there on the Shuttle, at the very least.
Java is also mentioned here, which implies that there is at least Java code that can talk to some spacecraft, if not actually run on them: http://www.dtnrg.org/wiki/Code
According to this article, the Ground Operations Center uses it for 3D Mapping and Planning. A comment on this forum claims the Mars Lander runs VxWorks.
Edit: Confirmed by Wind River, the Spirit and Opportunity run VxWorks RTOS. No other references where Java has been explicitly used in Space.
I think I read a few years ago that some probe or satellite used Java as part of their analyzation equipment but I doubt that mission critical software will be developed in Java today.
However: With all the applications used by astronauts on their Laptops I wouldn't be surprised if there were applications written in Java. Maybe some locally installed webapp that is used as a bugtracker.
I cannot speak with any authority with respect to the software onboard spacecraft, but I can say definitively that Java is used by NASA (or its affiliates) to handle the data from the various missions. I recently worked at a laboratory for space physics, which handled data and telemetry for in-orbit spacecraft, and the development team (of which I was a part) worked almost exclusively in Java.