This is a tricky one and is a little hard to explain but I will give it a shot to see if anyone out there has had a similar issue + fix.
Quick background:
Running a large Java Spring App on Tomcat in a Docker container. Other containers are simple, 1 for a JMS Queue and the other for Mysql. I run on Windows and have given Docker as much CPU as I have (and memory too). I have set JAVA_OPTS for Catalina to max out memory as well as memory limits in my docker-compose, but the issue seems to be CPU related.
When the app is idling it normally is sitting around 103% CPU (8 Cores, 800% max). There is a process we use which (using a Thread Pool) runs some workers to go out and run some code. On my local host (no docker in between) it runs very fast and flies, spitting out logs at a good clip.
Problem:
When running in Docker watching docker stats -a I can see the CPU start to ramp up when this process begins. Meanwhile in the logs, everything is flying by like expected while the CPU grows and grows. It seems to get close to 700% and then it kind of dies, but it doesn't. When it hits this threshold I see the CPU drop drastically down to < 5% where it stays for a little while. At this time logs stop printing, so I assume nothing is happening. Eventually it will kick back in and go back ~120% and continue its process like nothing happened sometimes respiking to ~400%.
What I am trying
I have played around with the memory settings to no success but it seems more like a CPU issue. I know Java in Docker is a bit wonky but I have given it all the room I can on my beefy dev box where locally this process runs without a hitch. I find it odd the CPU spikes then dies, but the container itself doesn't die or reset. Has anyone seen a similar issue or know some ways to further attack this CPU issue with Docker?
Thanks.
There is an issue in resource allocation in JVM containers, which occurs as it is referring to the overall system matrices instead of container matrices. In JAVA 7 and 8, JVM ergonomics are applying the systems’ (instance) matrices such as the number of cores and memory instead of docker allocated resources (cores and memory). As a result of that, the JVM initialized a number of parameters based on core count and Memory as below.
JVM memory footprints
-Perm/metaspace
-JIT Bytecode
-Heap Size (JVM ergonomics ¼ of instance memory)
CPU
-No. JIT compiler threads
-No. Garbage Collection threads
-No. Thread in the common fork-join pool
Therefore, the containers tend to become unresponsive due to high CPU or terminate the container by OOM kill. The reason for this is the container CGGroups and Namespaces are ignored by JVM in order to limit the Memory and CPU cycles. Therefore, JVM tends to get more resources of instance instead of limiting the separate allocation of docker allocated resources.E
Example
Assume two containers are running on 4 cores instance with 8GB memory. When it comes to the docker initialization point, assume that the docker is with 1GB memory and 2048 CPU cycles as a hard limit. Here, each and every container see 4 cores and those JVM allocate Memory, JIT compilers and GC threads separately according to their stats. However, the JVM will see the overall number of cores on that instance (4) and use that value to initialize the number of default threads that we have seen earlier. Accordingly, the JVM matrices of two containers will be as mentioned below.
-4 * 2 Jit Compiler Threads
-4 * 2 Garbage Collection threads
-2 GB Heap size * 2 (¼ of Instance full memory instead of docker allocated memory)
In terms of Memory
As per the above example, the JVM will gradually increase the heap usage as the JVM sees 2GB heap max size, which is a quarter of Instance memory (8GB). Once the memory usage of a container reaches to the Hard limit of 1GB, the container will be terminated by OOM kill.
In terms of CPU
As per the above example, one JVM has initialized with 4 Garbage Collection Threads and 4 JIT compiler. However, the docker allocates only 2048 CPU cycles. Therefore, it leads to high CPU, more context switching and unresponsive container, and finally will terminate the container due to high CPU.
Solution
Basically, there are two processes namely, CGGroups and Namespaces, which handling that kind of situation on the OS level. However, JAVA 7 and 8 do not accept the CGgroup and Namespaces, but the releases after jdk_1.8.131 are able to enable CGroup limit by JVM parameter (-XX:+UseCGroupMemoryLimitForHeap, -XX:+UnlockExperimentalVMOptions). However, it’s only providing solutions for memory issues but no concern on CPU set issue.
With OpenJDK 9, the JVM will automatically detect CPUsets. Especially in orchestration, it further able to manually overwrite the default parameters for CPU set thread counts as per the count of CPU cycles on the container by using JVM flags (XX:ParallelGCThreads, XX:ConcGCThreads).
Related
I have a Spring API with a heavy use of memory deployed on kubernetes cluster.
I configured the auto scale (HPA) to look at the memory consumption as a scale criterion, and running a load test everything works well at the time of scale up, however at the time of scale down the memory does not go down and consequently the pods created are not removed. If I run the tests again, new pods will be created, but never removed.
Doing an local analysis using visual VM, I believe the problem is correlated to the GC. Locally the GC works correctly during the test, but at the end of the requests it stops running leaving garbage behind, and it only runs again after a long time. So I believe that this garbage left behind is preventing the HPA from scale down.
Does anyone have any tips on what may be causing this effect or something that I can try?
PS. In the profiler I have no indications of any memory leak, and when I run the GC manually, the garbage left is removed
Here are some additional details:
Java Version: 11
Spring Version: 2.3
Kubernetes Version: 1.17
Docker Image: openjdk:11-jre-slim
HPA Requests Memory: 1Gi
HPA Limits Memory: 2Gi
HPA Memory Utilization Metrics: 80%
HPA Min Pods: 2
HPA Max Pods: 8
JVM OPS: -Xms256m -Xmx1G
Visual VM After Load test
New Relic Memory Resident After Load Test
There most likely isn't a memory leak.
The JVM requests a memory from the operating system up to the limit set by the -Xmx... command-line option. After each major GC run, the JVM looks at the ratio of heap memory in use to the (current) heap size:
If the ratio is too close to 1 (i.e. the heap is too full), the JVM requests memory from the OS to make the heap larger. It does this "eagerly".
If the ration is too close to 0 (i.e. the heap is too large), the JVM may shrink the heap and return some memory to the OS. It does this "reluctantly". Specifically, it may take a number of full GC runs before the JVM decides to release memory.
I think that what you are seeing is the effects of the JVM's heap sizing policy. If the JVM is idle, there won't be enough full GC to trigger the JVM to shrink the heap, and memory won't be given back to the OS.
You could try to encourage the JVM to give memory back by calling System.gc() a few times. But running a full GC is CPU intensive. And if you do manage to get the JVM to shrink the heap, then expanding the heap again (for the next big request) will entail more full GCs.
So my advice would be: don't try that. Use some other criteria to triggering your autoscaling ... if it makes any sense.
The other thing to note is that a JVM + application may use a significant amount of non-heap memory; e.g. the executable and shared native libraries, the native (C++) heap, Java thread stack, Java metaspace, and so on. None of that usage is constrained by the -Xmx option.
I have a Java application which uses 10 threads. Each thread opens a Matlab session by using the Java Matlabcontrol library. I'm running the application on a cluster running CentOS 6.
The used physical memory (Max Memory) for the whole application is around 5GB (as expected) but the reserved physical memory (Max Swap) is around 80GB which is too high. Here a short description from the cluster wiki:
A note on terminology: in LSF the Max Swap is the memory allocated by
an application and the Max Memory is the memory physically used (i.e.,
it is actually written to). As such, Max Swap > Max Memory. In most
applications Max Swap is about 10–20% higher than Max Memory
I think the problem is Java (or perhaps a mix between Java and Matlab). Java tends to allocate about 50% of the physically available memory on a compute node by default. A java process assumes that it can use the entire resources available on the system that it is running on. That is also the reason why it starts several hundred threads (although my application only uses 11 threads). It sees 24 cores and lots of memory even though the batch system reserves only 11 core for the job.
Is there a workaround for this issue?
Edit: I've just found the following line in the Matlabcontrol documentation:
When running outside MATLAB, the proxy makes use of multiple
internally managed threads. When the proxy becomes disconnected from
MATLAB it notifies its disconnection listeners and then terminates all
threads it was using internally. A proxy may disconnect from MATLAB
without exiting MATLAB by calling disconnect().
This explains why there are a lot of threads created but it does not explain the high amount of reserved memory.
Edit2: Setting MALLOC_ARENA_MAX=4 environment variable brought the amount of reserved memory down to 30GB. What value of MALLOC_ARENA_MAX should I choose and are they other tuning possibilities?
I have a machine with 10GB of RAM. I am running 6 java processes with the -Xmx option set to 2GB. The probability of all 6 processes running simultaneously and consuming the entire 2GB memory is very very low. But I still want to understand this worst case scenario.
What happens when all 6 processes consume a little less than 2GB memory at the same instant such that the JVM does not start garbage collection yet the processes are holding that much memory and the sum of the memory consumed by these 6 processes exceeds the available RAM?
Will this crash the server? OR Will it slow down the processing?
You should expect each JVM could use more than 2 GB. This is because the heap is just one memory region, you also have
shared libraries
thread stacks
direct memory
native memory use by shared libraries
perm gen.
This means that setting a maximum heap of 2 GB doesn't mean your process maximum 2 GB.
Your processes should perform well until they get the point where you have swapped some of the heap and a GC is performed. A GC assumes random access to the whole heap and at this point, your system could start swapping like mad. If you have a SSD for swap your system is likely to stop, or almost stop for very long periods of time. If you have Windows (which I have found is worse than Linux in this regard) and a HDD, you might not get control of the machine back and have to power cycle it.
I would suggest either reducing the heap to say 1.5 GB at most, or buying more memory. You can get 8 GB for about $100.
Your machine will start swapping. As long as each java process uses only a small part of the memory it has allocated, you won't notice the effect, but if they all garbage collect at the same time, accessing all of their memory, your hard disk will have 100% utilization and the machine will "feel" very, very slow.
I'm monitoring a production system with AppDynamics and we just had the system slow to a crawl and almost freeze up. Just prior to this event, AppDynamics is showing all GC activity (minor and major alike) flatline for several minutes...and then come back to life.
Even during periods of ultra low load on the system, we still see our JVMs doing some GC activity. We've never had it totally flatline and drop to 0.
Also - the network I/O flatlined at the same instance of time as the GC/memory flatline.
So I ask: can something at the system level cause a JVM to freeze, or cause its garbage collection to hang/freeze? This is on a CentOS machine.
Does your OS have swapping enabled.
I've noticed HUGE problems with Java once it fills up all the ram on an OS with swapping enabled--it will actually devistate windows systems, effictevly locking them up and causing a reboot.
My theory is this:
The OS ram gets near full.
The OS requests memory back from Java.
This Triggers Java into a full GC to attempt to release memory.
The full GC touches nearly every piece of the VMs memory, even items that have been swapped out.
The system tries to swap data back into memory for the VM (on a system that is already out of ram)
This keeps snowballing.
At first it doesn't effect the system much, but if you try to launch an app that wants a bunch of memory it can take a really long time, and your system just keeps degrading.
Multiple large VMs can make this worse, I run 3 or 4 huge ones and my system now starts to sieze when I get over 60-70% RAM usage.
This is conjecture but it describes the behavior I've seen after days of testing.
The effect is that all the swapping seems to "Prevent" gc. More accurately the OS is spending most of the GC time swapping which makes it look like it's hanging doing nothing during GC.
A fix--set -Xmx to a lower value, drop it until you allow enough room to avoid swapping. This has always fixed my problem, if it doesn't fix yours then I'm wrong about the cause of your problem :)
It is really difficult to find the exact cause of your problem without more information.
But I can try to answer to your question :
Can the OS block the garbage collection ?
It is very unlikely than your OS blocks the thread garbage collector and let the other threads run. You should not investigate that way.
Can the OS block the JVM ?
Yes it perflecty can and it do it a lot, but so fast than you think that the processes are all running at the same time. jvm is a process like the other and his under the control of the OS. You have to check the cpu used by the application when it hangs (with monitoring on the server not in the jvm). If it is very low then I see 2 causes (but there are more) :
Your server doesn't have enough RAM and is swapping (RAM <-> disk), process becomes extremely slow. In this case cpu will be high on the server but low for the jvm
Another process or server grabs the resources and your application or server receive nothing. Check the priority on CentOs.
In theory, YES, it can. But it practice, it never should.
In most Java virtual machines, application threads are not the only threads that are running. Apart from the application threads, there are compilation threads, finalizer threads, garbage collection threads, and some more. Scheduling decisions for allocating CPU cores to these threads and other threads from other programs running on the machine are based on many parameters (thread priorities, their last execution time, etc), which try be fair to all threads. So, in practice no thread in the system should be waiting for CPU allocation for an unreasonably long time and the operating system should not block any thread for an unlimited amount of time.
There is minimal activity that the garbage collection threads (and other VM threads) need to do. They need to check periodically to see if a garbage collection is needed. Even if the application threads are all suspended, there could be other VM threads, such as the JIT compiler thread or the finalizer thread, that do work and ,hence, allocate objects and trigger garbage collection. This is particularly true for meta-circular JVM that implement VM threads in Java and not in a C/C++;
Moreover, most modern JVM use a generational garbage collector (A garbage collector that partitions the heap into separate spaces and puts objects with different ages in different parts of the heap) This means as objects get older and older, they need to be moved to other older spaces. Hence, even if there is no need to collect objects, a generational garbage collector may move objects from one space to another.
Of course the details of each garbage collector in different from JVM to JVM. To put more salt on the injury, some JVMs support more than one type of garbage collector. But seeing a minimal garbage collection activity in an idle application is no surprise.
We have a customer that uses WebSphere 7.0 on RedHat Linux Server 5.6 (Tikanga) with IBM JVM 1.6.
When we look at the OS reports for memory usage, we see very high numbers and OS starts to use SWAP memory in some point due to lack in memory.
On the other hand, JConsole graphs show perfectly normal behavior of memory - Heap size increases until GC is invoked when expected and Heap size drops to ~30% in normal cycles. Non heap is as expected and very constant in size.
Does anyone have an idea what this extra native memory usage can be attributed to?
I would check you are looking at resident memory and not virtual memory (the later can be very high)
If you swap, even slightly this can cause the JVM to halt for very long periods of time on a GC. If your application is not locking up for second or minutes, it probably isn't swapping (another program could be)
If your program really is using native memory, this will most like be due to a native library you have imported. If you have a look at /proc/{id}/mmap this may give you a clue, but more likely to will have to check which native libraries you are loading.
Note: if you have lots of threads, the stack space for all these reads can add up. I would try to keep these to a minimum if you can, but I have seen JVMs with many thousands and this can chew up native memory. GUI components can also use native memory but I assume you don't have any of those.