Develop Reference

Tomcat garbage collection not thoroughly

I'm really out of my comfortzone when I have to monitor memory. But I'm the only one here, and I'm just left clueless: I have a Java8 application (CMS) on a Tomcat application-server running, and where in some trouble. After a while the server crashes.
After some research, I found out it is memory related, so I attached visualVM on my environment, and started monitoring.
I see that the memory is slowly filling up. Garbage Collection does it's job, but not thoroughly. It always leaves some more memory in the used heap. When I do a manual 'Perform Garbage Collection' in visual VM, the garbage collection is performed much better. (See screenshots)
It'll take several hours, but the used heap will grow larger and larger again after each garbage collection. The moment that I will manually preform GC again, the minimal used heap will be 'normal' again.
I have noticed that the heap fills with byte[]. Those will fill the most of the space. Someone could help me out on this?

I see that the memory is slowly filling up. Garbage Collection does
it's job, but not thoroughly. It always leaves some more memory in the
used heap. When I do a manual 'Perform Garbage Collection' in visual
VM, the garbage collection is performed much better.
Full GC gets triggered when JVM feel its necessary (as its costly . For example Its stop the world for parallel GC . Similarly stop the world for two sub phases for concurrent mark sweep collector) . It depends on various factors like Xms and Xmx parameters see JVM heap parameters. So you should not be worried about until and unless you get out of mem exception as JVM will trigger when necessary
For server crash :- I can think of two problems
Memory leak. In that case memory footprints will be increasing even after each GC
May be you are constructing some cache without eviction algorithm if its near to full
If both does not apply, i see a usecase of increasing heap and give it a try

I've had a few problems like this before. One was our app's fault, one was the app server's fault, and one I wasn't able to figure out but was able to mitigate.
In each case I used JProfiler to watch memory usage on a local server and ran a variety of happy-path and exception tests to try to figure out what was causing the problem. Doing this testing wasn't a quick and easy process - on average I spent about a week each time.
In the first case (our app's fault), I found that we were not closing SQL connections for a web service when exceptions were thrown. Testing the happy paths showed no problems, but when I started testing exceptions I could exhaust the server's memory with about 100 consecutive exceptions. Adding code to manually clean up resources in the exception handler solved the problem.
In the second case (WebSphere's fault), I verified that our app was closing all resources correctly, but the problem persisted. So I started reading through WebSphere documentation and found that it was a known issue with JAX-WS clients. Luckily there was a patch to WebSphere which fixed the problem.
In the third case (couldn't determine the cause), I was unable to find any reason why it was happening. So the problem was mitigated by increasing JVM memory allocation to an amount where the OOM exceptions would take greater than 1 week to happen, and configuring the servers to restart every weekend.

There might be some simple technical workarounds to mitigate the problem; like: simply adding more memory to the JVM and/or the underlying machine.
Or if you really can prove that running System.gc() manually helps (as the comments indicates that most people think: it will not) you could automate that step.
If any of that is good enough to prevent those crashes; you are buying yourself more time to work on a real solution.
Beyond that, a meta, non-technical perspective. Actually there are two options:
You turn to your manager; and you tell him that you will need anything beetween 2, 4, 8 weeks in order to learn this stuff; so that you can identify the root cause and get it fixed.
You turn to your manager; and you tell him that he should look for some external consulting service to come, identify the root cause and help fixing it.
In other words: your team / product is in need for some "expert" knowledge. Either you invest in building that knowledge internally; or you have to buy it from somewhere.

Java web application really slow

I am using Rackspace as a hosting provider, using their Cloud server hosting, with 256mb plan.
I am using Geronimo 2.2 to run my java application.
The server starts up no problem, loads Geronimo quite fast, however, when I started to deploy my web application, it is taking forever, and once it is deployed, it takes forever to navigate through pages.
I've been monitoring the server activity, the CPU is not so busy, however, 60% of the memory is being used up. Could this be the problem?
If so, what are my options? Should I consider upgrading this cloud server to something with more RAM, or changing a host provider to better suit my needs?
Edit:
I should note that, even if I don't deploy my application, just having Geronimo loaded, sometimes I would get a connection time when I try to shut down Geronimo.
Also the database is on the same server as the application. (however I wouldn't say its query intensive)
Update:
After what #matiu suggested, I tried running free -m, and this is the output that I get:
total used free shared buffers cached
Mem: 239 232 6 0 0 2
-/+ buffers/cache: 229 9
Swap: 509 403 106
This was totally different result than running ps ux, which is how I got my previous 60%.
And I did an iostat check, and about 25% iowait time, and device is constantly writing and reading.
update:
Upgraded my hosting to 512MB, now it is up to speed! Something I should note is, I forgot about the Java's Permanent Generation memory, which is also used by Geronimo. So it turns out, I do need more RAM, and more RAM did solve my problem. (As expected) yay.

I'm guessing you're running into 'swapping'.
As you'll know Linux swaps out some memory to disk. This is great for memory that's not accessed very much.
When Java starts eating heaps and heaps, linux starts:
Swapping memory block A out to disk to make space to read in block B
Reading/writing block B
Swapping block B to disk to make space for some other block.
As disk is 1000s of times slower than RAM, as the memory usage increases your machine grinds more and more closer to a halt.
With 256 MB Cloud Servers you get 512 MB of Swap space.
Checking:
You can check if this is the case with free -m .. this page shows how to read the output:
Next I'd check with 'iostat 5' to see what the disk IO rate on the swap partition is. I would say a write rate of 300 or more means you're almost dead in the water. I'd say you'd want to keep the write rate of the swap partition down below 50 blocks a second and the read rate down below 500 blocks a second .. if possible both should be zero most of the time. Remember disk is 1000s of times slower than RAM.
You can check if it's Java eating the ram by running top and hitting shift+m to order the processes by memory consumption.
If you want .. you can disable the swap partition with swapoff -a .. then open up the web console, and hit the site a bit .. you'll soon see error messages in the console like 'OOM Killed process xxx' (OOM is for Out of Memory I think). If you see those that's linux trying to satisfy memory requests by killing processes. Once that happens, it's best to hard reboot.
Fixing:
If it's Java using the RAM .. this link might help.
I think the easy fix would be just to upgrade the size of the Cloud Server.
You may find a different Java RTE may be better.
If you run it in a 32 bit chroot it may use less RAM.

You should consider running a virtual dedicated Linux server, from something like linode.
You'd have to worry about how to start a Java service and things like firewalls, etc, but once you get it right, you are in effect you're own hosting provider, allowing you to do anything a standalone actual Linux box can do.
As for memory, I wouldn't upgrade until you have evidence that you do not have enough. 60% being used up is less than 100% used up...
Java normally assumes that it can take whatever it is assigned to it. Meaning, if you give it a max of 200MB, it thins that it's ok to take 200MB even though it's using much less.
There is a way to make Java use less memory, by using the -Xincgc incremental garbage collector. It actually ends up giving chunks of memory back to the system when it no longer needs it. This is a bit of a kept secret really. You won't see anyone point this out...

Based on my experience, memory and CPU load on VPSes are quite related. Meaning, when application server will take up all available memory, CPU usage starts to sky rock, finally making application inaccessible.
This is just a side effect though - you should really need to investigate where your problems origin!
If the memory consumption is very high, then you can have multiple causes:
It's normal - maybe you have reached a point, where all processes (application server, applications within it, background processes, daemons, Operating System, etc.) put together need that huge amount of memory. This is least probably scenario.
Memory leak - can happen due to bug in framework or some library (not likely) or your own code (possible). This can be monitored and solved.
Huge amount of requests - each request will take both CPU and memory to be processed. You can have a look at the correlation between requests per second and memory consumption, meaning, it can be monitored and resolved.
If you are interested in CPU usage:
Again, monitor requests to your application. For constant count of requests - nothing extraordinary should happen.
One component is exhausting most resources (maybe your database is installed on the same server and it uses all CPU power due to inefficient queries? Slow log would help.)
As you can see, it's not trivial task, but you have tools support which will can help you out. I personally use java melody and probe.

Debugging a strange memory leak - Java/Tomcat

I'm experiencing a very odd problem with a Java application running under Tomcat.
We tried to update the production code from a fresh newly produced in a 1-week sprint, the application has been running over months without hiccups and then this new code makes our Linux servers start swapping after some time.
The very strange thing is that when looking at VisualVM for memory usage it never exceeds the maximum heap size, the JVM does not throw an OutOfMemory, the machine only starts swapping and the JVM keeps running even after that.
So, it seems that's leaking memory from somewhere, it seems like it's from the new code but it's odd that it's not inside the JVM, any ideas in how to debug that?
Thanks!

Swapping is not a conclusive indicator of leakage. It results from low physical memory. Use vmstat on Linux to get swap usage. Try using a different machine, experiment with configurations --swap size, physical memory size, address space.
If you are confident that the problem is in your program try this:
Estimate the median and peak memory that your program should use. You must be able to account for all deviations from these metrics. If you cannot, proceed to step 3.
Assuming you did step 1 correctly and were able to account for all deviations, you can rule out the leak (sorry about such vague suggestions but debugging is only as good as the detective). You should now focus on GC tuning. First, enable GC logging. See if your heap is actually full and where the GC is spending most of its time collecting. This may be a good starting point to start optimizations. Try to see if adjusting GC options helps. Try experimenting with collection algorithms, max/min heap sizes, gen ratios etc. Only experiment when you have ruled out a leak (step 1).
Assuming you did step 1 correctly and were not able to account for all deviations, you can assume that you have a leak somwhere. Use a memory profiler to see what objects contribute to the heap size growth most. Leave a profiler running for an extended period of time --have your program handle some requests it routinely expects to get and then leave it relatively isolated after that. If the memory level keeps on growing you may have a leak somewhere. If not, then it is probably not a memory leak. Can you pin point the part of your program that may be creating them? If yes, try sending several requests that only target that part of your program. Does it replicate the problem deterministically? If no, repeat step 3. If yes, use divide and conquer and reapply step 3 till you can find the class/method that are the culprits. It can be a certain combination of multiple portions as well (meaning that individually they may look innocent but together they may form a brilliant crime syndicate).
Hope this helps, if not then please leave a comment to my post.
All the very best on your exercise!

I would suggest you look into creating heap dumps without using jvisualvm. For Unix-based Oracle JVM's this is normally done by sending a signal 3 to the JVM using kill.
For full details see http://www.startux.de/index.php/java/45-java-heap-dumpyvComment45
You can then see if the patterns changes.
If you do not get an idea from this, then this might be because you are storing a sub-string from a very large original string (which carries the underlying string array around), or because you hold on to operating system resources like open database connections etc.
You have checked your connection pool looks good?

If you aren't using it, I'd recommend using visual VM version 1.3.2 and all the plug-ins. It's a big jump up from earlier versions.
What happens to the perm gen space?
What are the memory settings you're using? Min and max, of course, but what about perm space size?

Why does System. gc () seem to have no effect on some JVMs

I have been developing a small Java utility that uses two frameworks: Encog and Jetty to provide neural network functionality for a website.
The code is 'finished' in that it does everything it needs to do, but I have some problems with memory usage. When running on my development machine the memory usage seems to fluctuate between about 4MB and 13MB when the application is doing things (training neural networks) and at most it uses about 18MB. This is very good usage and I think it is due to the fact that I call System.GC() fairly regularly. I do this because the processing time doesn't matter for me, but the memory usage does.
So it all works fine on my machine, but as soon as I put it online on our server (shared unix hosting with memory limits) it uses about 19MB to start with and rises to hundreds of MB of memory usage when doing things. These are the same things that I have been doing in testing. The only way, I believe, to reduce the memory usage, is to quit the application and restart it.
The only difference that I can tell is the Java Virtual Machine that it is being run on. I do not know about this and I have tried to find the reason why it is acting this way, but a lot of the documentation assumes a great knowledge of Java and Virtual Machines. Could someone please help m with some reasons why this may be happening and perhaps some things to try to stop it.
I have looked at using GCJ to compile the application, but I don't know if this is something I should be putting a lot of time in to and whether it will actually help.
Thanks for the help!
UPDATE: Developing on Mac OS 10.6.3 and server is on a unix OS but I don't know what. (Server is from WebFaction)

I think it is due to the fact that I
call System.GC() fairly regularly
You should not do that, it's almost never useful.
A garbage collector works most efficiently when it has lots of memory to play with, so it will tend to use a large part of what it can get. I think all you need to do is to set the max heap size to something like 32MB with an -Xmx32m command line parameter - the default depends on whether the JVM believes it's running on a "server class" system, in which case it assumes that you want the application to use as much memory as it can in order to give better throughput.
BTW, if you're running on a 64 bit JVM on the server, it will legitimately need more memory (usually about 30%) than on a 32bit JVM due to larger references.

Two points you might consider:
Calls of System.gc can be disabled by a commandline parameter (-XX:-DisableExplicitGC), I think the behaviour also depends on the gc algorithm the vm uses. Normally invoking the gc should be left to the jvm
As long as there is enough memory available for the jvm I don't see anything wrong in using this memory to increase application and gc performance. As Michael Borgwardt said you can restrict the amount of memory the vm uses at the command line.

Also you may want to look at what mode the JVM has been started when you deploy it online. My guess its a server VM.
Take a look at the differences between the two right here on stackoverflow. Also, see what garbage collector is actually running on the actual deployment. See if you can tweek the GC behaviour, or change the GC algorithm.See the -X options if its a Sun JVM.

Basically the JVM takes the amount of memory it is allowed to as needed, in order to make the "new" operation as fast as possible (this is a science in itself).
So if you have a lot of objects being used, and then discarded, you will slowly and surely fill up the available memory. Then you can ask for garbage collection, but it is just a hint, and the JVM may choose not to listen.
So, you need another mechanism to keep memory usage down. The typical approach is to limit the amount of memory with -Xoptions, but be careful since the JVM you use on your pc may be very different from the one you deploy on, and the memory need may therefore be different.
Is there a deliberate requirement for low memory usage? If not, then just let it run and see how the JVM behaves. Use jvisualvm to attach and monitor.

Perhaps the server uses more memory because there is a higher load on your app and so more threads are in use? Jetty will use a number of threads to spread out the load if there are a lot of requests. Its worth a look at the thread count on the server versus on your test machine.

Java very limited on max number of threads?

We have a small text box with 512Mb of ram. We wanted to see how many threads we can create in Java in this box. To our surprise, we can't create many. Essentially the minimum stack size you can set with -Xss is 64k. Simple math will tell you that 64*7000 will consume 430Mb so we were only able to get it up to around 7000 threads or so and then we encountered this error:
java.lang.OutOfMemoryError: unable to create new native thread.
Is this the true limit with Java? Per 512Mb of ram we can only squeeze in 7k number of threads or so?

Use asynchronous IO (java nio) and you'll don't need 7k threads to support 7k clients, a few threads for handling io (5?) will be enough.
Take a look at Netty ;)
One thread for each client is a really bad design.

Once you create your 7k threads, you're not going to have any memory to do anything useful. Perhaps you should have a rethink about the design of your application?
Anyway, isn't 512Mb quite small? Perhaps you could provide a bit more information about your application or perhaps the domain?

Keep in mind that you will never be able to dedicate 100% of the RAM to running Java threads. Some RAM is used by the OS and other running applications, meaning you will never have the full 512 Mb available.

It's not the programming language, it's on the operating system level.
More reading about it, for Windows:
Does Windows have a limit of 2000 threads per process?
Pushing the Limits of Windows: Processes and Threads (by Mark Russinovich)

You don't necessarily need one thread per client session. If you look at the way that a J2EE (or JavaEE) server handles multiple connections it uses a mixture of strategies including concurrency, queuing and swapping. Usually you can configure the maximum number of live concurrent instances and idle time-out values at deployment time to tune the performance of your application.

Try setting the maximum memory allowed -Xmx to a lower value and see whether the thread count can be increased. In a project at work I could allocate around 2,5k threads with -Xmx512m and about 4k threads with -Xmx96m.
The bigger your heap the smaller your thread stack space (at least to my experience).