I've been having this memory leak issue for days and I think I have some clues now. The memory of my java process keeps growing but yet the heap does not increase. I was told that this is possible if I create many threads, because Java threads uses memory outside of the heap.
My java process is a server type program so there are 1000-2000 threads. Created and deleted ongoing. How do I reclaim the memory used by a java thread? Do I simply erase all references to the thread object and make sure that this is terminated?
Yes. That is the answer. As long as there is an active reference to any Java object, then that object won't be garbage collected when it's done.
If you're creating and destroying threads and not pooling them, I think you have other issues as well.
From the Java API docs threads die when:
All threads that are not daemon threads have died, either by returning from the call to the run method or by throwing an exception that propagates beyond the run method.
Threads die when they return from their run() method. When they die they are candidates for garbage collection. You should make sure that your threads release all references to objects and exit the run() method.
I don't think that nulling references to your threads will really do the trick.
You should also check out the new threading facilities in Java 5 and up. Check the package java.util.concurrent in the API documentation here.
I also recommend you to check the book Concurrency in Practice. It's being priceless for me.
There are two things that will cause a Thread to be not garbage collected.
Any thread that is still alive will not be garbage collected. A thread is alive until the run method called by Thread.start() exits, either normally or by throwing an exception. Once this happens (and the thread's uncaught exception handler has finished), the thread is dead.
Any live reference to the Thread object for a thread will prevent it from being garbage collected. The live reference could be in your code, or if you are using thread pools, they could be part of the pool data structures.
The memory of my java process keeps growing but yet the heap does not increase.
That would be because each thread has a large (e.g. 1Mb) stack segment that is not allocated in the Java heap.
A thread's stack segment is only allocated when the thread is started, and released as soon as the thread terminates. The same also applies (I think) to the thread's thread-local map. A Thread object that is not "alive" doesn't use much memory at all.
So to sum it up. You appear to have lots of live threads. They won't be garbage collected as long as they are alive, and the only way to make them release their memory is to cause them to die ... somehow.
To reduce memory usage, you to need to do one or more of:
Look at the thread code (the run() methods, etc) to figure out why they are still hanging around.
Reduce the size of the thread stacks. (In theory, you can go as low as 64K ...)
Redesign your app so that it doesn't create thousands of threads. (Thread pools and some kind of work queue is one possible approach.)
That is a lot of threads, each of which imposes a memory overhead, and well as other resources for managing them (context switching etc). Use a profiler to view the thread activity - you'll likely find that most of the threads are idle most of the time.
I'd suggest the first step is to look at managing the threads using the thread pools provided by java.util.concurrent. Rather than creating threads, look to create tasks that are handed off to the pools. Tweak the pools until you have a much smaller number of threads that are kept reasonably busy. This may well resolve the memory issue; it will certainly improve performance.
Related
If I have two Java threads executing in native C++ code and one of them new's a native object, is it OK for the other Java thread to do the delete? Naturally, there are plenty of ways to screw up one thread doing a new and another doing the delete. But is there anything 'extra' because they are Java threads? (Like maybe each Java thread gets a separate native heap, or other such nonsense).
I assume there is nothing special about this case, but Valgrind is telling me that I am definitely leaking that memory and I need to get this nonsense out of my head so I can focus on finding the real problem Valgrind is trying to show me.
All Java threads executing native code see the exact same native heap and there is nothing special about them being Java threads vs threads created from native code. Right?
It's impossible to tell without more details.
If your threads are using some sort of publishing protocol like this
thread 'a' allocates some memory and does some work
thread 'b' is waiting on a semaphore
thread 'a' finishes its work and signals the semaphore
thread 'b' consumes the mememory and deletes it
then things should be OK. (Usually you'd have more than just the two threads.)
If you don't use some synchronization mechanism then the memory needs to have strong thread safety guarantees. Most things in C++ only give weak guarantees.
I'm running an async task (defined in a method annotated with Spring's #Async annotation) on multiple parallel threads in a Spring Boot application. I use a ThreadPoolExecutor to manage threads, which is initialised with a ThreadPoolExecutorFactoryBean:
#Bean
ThreadPoolExecutorFactoryBean myThreadPoolExecutor() {
final ThreadPoolExecutorFactoryBean factoryBean = new ThreadPoolExecutorFactoryBean();
factoryBean.setCorePoolSize(...);
factoryBean.setMaxPoolSize(...);
factoryBean.setQueueCapacity(...);
factoryBean.setKeepAliveSeconds(...);
factoryBean.setPrestartAllCoreThreads(true);
factoryBean.setAllowCoreThreadTimeOut(false);
factoryBean.setExposeUnconfigurableExecutor(true);
return factoryBean;
}
The task uses a ByteBuffer inside. This buffer used to be defined as a local variable in the async method before.
As the app is memory-critical, I'd like to decrease the heap usage of the threads as much as possible, so I try to "recycle" the spaces once allocated.
To achieve this, I introduced a thread-local property for the ByteBuffer (among some other previously local variables) in the #Component where the #Async method is defined:
private ThreadLocal<ByteBuffer> buffer;
This thread-local property gets initialised in the #Component's constructor, where I allocate space for the buffer:
public MyComponent(...) {
...
this.buffer = ThreadLocal.withInitial(() -> ByteBuffer.allocate(2048));
...
}
This way we have to allocate buffer space only once, and the successive tasks received by the same thread should use the same space for their calculations, instead of allocating a new buffer as a local variable every time the thread receives a new task from the ThreadPoolExecutor. And as each thread has its own value of the thread-local property (maintained in its thread-local map), this solution should be thread-safe, the multiple parallel-running threads won't mess up the data in each other's buffer. (The task's code is implemented in such a way that the data written into the buffer by a previous run on the same thread, doesn't effect the current run; the async task just needs a buffer space for counting, it's not a problem for it if the buffer space already contains data from a previous run, it will rewrite it anyway. So the successive runs on the same thread don't disturb each other, either.)
I read about that thread-local properties can cause a memory leak if they are not .remove()-d, because the thread-local property holds a strong reference to them. However, here, all threads in the thread pool are working continuously until the moment they die, and even their death happens very rarely (usually only when the app terminates itself). Once the threads finish the execution of one task, they will shortly receive a new one from the executor service. In this app, the threads and their thread-local properties are designed for life-long continuous usage.
From introducing thread-local properties, we hoped decreased heap usage and GC activity. But we experience the opposite, and this is the question of the post: both heap usage & minor GC collection count and time increased after deploying this change.
I've made heap dumps, and they tell that the thread-local space does get deallocated after a thread's potential death. And debugger proved that the same buffer object is indeed kept and shared between successive tasks executed by a certain thread.
I've been digging into the underlying code for a few days now, to find a root cause why it behaves this way, but I've not found any explanation yet. I once supposed "it might be because the thread-local properties might get created on the main thread instead of the worker threads", but this assumption proved wrong - they are rightfully created on each worker thread, as expected -, and in the moment, I have no further idea where else to look, what direction I should dig into even deeper. :(
Does anyone have any idea how to explain this anomaly (why thread-locals work opposite from what we would expect, why they increase heap usage and GC activity instead of decreasing it in comparison to having a local buffer variable defined in the async method)?...
I have been reading about this article:
Creating a memory leak with Java
Since I myself have issues with threads:
Springboot increasing threads even if server.tomcat.max-threads=5
Does this mean that Long running threads in Java will create memory leaks?
What if I need to have a long running thread in the first place.
Isn't most application long running threads also?
Long running thread doesn't create memory leak. It is what you do inside it. Technically memory leaks happens when garbage collector could not collect free space, as the space is marked as being used. i.e. if references are held.
Now in a long running thread, you could have an object reference present for the lifetime of the thread. This object itself could be expensive. This is the case in first link you shared(threadlocal holding transitively all the references)
On your second link, the problem seems to lie somewhere. Here what I generally do if I suspect memory leak
Use jmap to get count of each class instances
Force full GC
Again count the instances of each class, these are the objects GC was not able to clean
Repeat multiple times, you will notice some objects, which should have been cleared. This will give you some idea. Following those references in code you can get some idea.
I know that I Java's Runtime object can report the JVM's memory usage. However, I need the memory usage for a certain thread. Any idea how to get this?
I appreciate your answer!
A Thread shares everything except its stack and the CPU cycles with all other threads in the VM. All objects created by the Thread are pooled with all the other objects.
The problem is to define what the memory usage of a Thread is. Is it only those objects it created? What if these objects subsequently are referenced by other threads? Do they only count half, then? What about objects created somewhere else, but are now referenced by this Thread?
I know of no tool trying to measure the memory consumption of separate Threads.
I'm working on a program right now that is essentially this: there is a 4 way stop with cars arriving on each road at random times. Each road is served FCFS and the intersection is managed round robin style, 1 car crossing at a time. Each waiting car is a thread. I've gotten the thread synchronization and algorithm working no problem. The issue I can't quite figure out is how to prevent the error: OutOfMemoryError: unable to create new native thread. I realize that this is due to the heap (stack? I always get them switched) becoming full. I can't figure out a way to ensure executed threads are properly managed by the garbage collector and not lingering in memory after execution. I've tried setting my queues (each "road" with the car threads) up with soft references and nulling any hard references out to no avail. Anyone on here have experience with this!? THANKS!!!
"OutOfMemoryError: unable to create new native thread" does not refer to heap memory. It won't help you nulling references or using soft/weak references. Furthermore, increasing the heap size can only make things worse.
Java uses native memory for thread stacks. Each time you start a thread, a new stack is allocated, outside of the JVM heap. The stack is not released until the thread terminates. Consider using less concurrent threads (you can control the number by using ThreadPoolExecutor for example), or maybe decrease the stack sizes (using -Xss{size}k)
See also this post, which details many types of out of memory errors.
Did you tried using a ThreadPool?
You can create a ThreadPool since Java 5 in which you decide how many threads the Vm should initialize for you algorithm. Threads are created and reused.
I had a similar problem. Threads are not deleted/removed by the GarbageCollector and somehow live for ever.
This will only happen if you have too many running threads. (Not just references to threads) Like #Markus, I would suggest you switch to a ThreadPool like ExecutionService as it will manage the creation of threads and it works.
BTW: The concurrency library dates back to 1998, but was only included in Java 5.0 (2005) so if you have to have an older version you can use either the backport or the original library.