Develop Reference

Java (prior to JDK8 update 131) applications running in docker container CPU / Memory issues?

JVM's (JDK 8 before Update 131) running in docker containers were ignoring the CGroup limitations set by the container environment.
And, they were querying for host resources and not what was allocated to the container.
The result is catastrophic for the JVM i.e As the JVM was trying to allocate itself more resources (CPU or Memory) than what is permitted through CGroup limits, docker demon would notice this and kill the JVM process or the container itself if the java program was running with pid 1.
Solution for memory issue - (possibly fixed in JDK 8 update 131)
Like described above, JVM was allocating it's self more memory than what's allowed for the container. This could be easily fixed by
explicitly setting the max heap memory limit (using -Xmx ) while starting the JVM. ( prior to 131 update)
or by passing these flags - (after 131 update)
-XX:+UnlockExperimentalVMOptions and
-XX:+UseCGroupMemoryLimitForHeap
Resolving the CPU issue (possibly fixed in JDK update 212 )
Again like described above, JVM running in docker would look at the host hardware directly and obtain the total CPUs available. Then it would try to access or optimize based on this CPU counts.
After JDK 8 update 212, any JVM running in docker container will respect the cpu limits allocated to container and not look into host cpus directly.
If a container with cpu limitation is started as below, JVM will respect this limitation and restrict itself to 1 cpu.
docker run -ti --cpus 1 -m 1G openjdk:8u212-jdk //jvms running in this container are restricted to 1cpu.
HERE IS MY QUESTION: The CPU issue is probabily fixed in JDK8 Update 212, but what if I can not update my JVM and I am running version prior to update 131 , how can I fix the cpu issue.

Linux container support first appeared in JDK 10 and then ported to 8u191, see JDK-8146115.
Earlier versions of the JVM obtained the number of available CPUs as following.
Prior to 8u121, HotSpot JVM relied on sysconf(_SC_NPROCESSORS_ONLN) libc call. In turn, glibc read the system file /sys/devices/system/cpu/online. Therefore, in order to fake the number of available CPUs, one could replace this file using a bind mount:
echo 0-3 > /tmp/online
docker run --cpus 4 -v /tmp/online:/sys/devices/system/cpu/online ...
To set only one CPU, write echo 0 instead of echo 0-3
Since 8u121 the JVM became taskset aware. Instead of sysconf, it started calling sched_getaffinity to find the CPU affinity mask for the process.
This broke bind mount trick. Unfortunately, you can't fake sched_getaffinity the same way as sysconf. However, it is possible to replace libc implementation of sched_getaffinity using LD_PRELOAD.
I wrote a small shared library proccount that replaces both sysconf and sched_getaffinity. So, this library can be used to set the right number of available CPUs in all JDK versions before 8u191.
How it works
First, it reads cpu.cfs_quota_us and cpu.cfs_period_us to find if the container is launched with --cpus option. If both are above zero, the number of CPUs is estimated as
cpu.cfs_quota_us / cpu.cfs_period_us
Otherwise it reads cpu.shares and estimates the number of available CPUs as
cpu.shares / 1024
Such CPU calculation is similar to how it actually works in a modern container-aware JDK.
The library defines (overrides) sysconf and sched_getaffinity functions to return the number of processors obtained in (1) or (2).
How to compile
gcc -O2 -fPIC -shared -olibproccount.so proccount.c -ldl
How to use
LD_PRELOAD=/path/to/libproccount.so java <args>

How do I elegantly and safely maximize the amount of heap space allocated to a Java application in Kubernetes?

I have a Kubernetes deployment that deploys a Java application based on the anapsix/alpine-java image. There is nothing else running in the container expect for the Java application and the container overhead.
I want to maximise the amount of memory the Java process can use inside the docker container and minimise the amount of ram that will be reserved but never used.
For example I have:
Two Kubernetes nodes that have 8 gig of ram each and no swap
A Kubernetes deployment that runs a Java process consuming a maximum of 1 gig of heap to operate optimally
How can I safely maximise the amount of pods running on the two nodes while never having Kubernetes terminate my PODs because of memory limits?
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
spec:
replicas: 1
template:
metadata:
labels:
app: my-deployment
spec:
containers:
- name: my-deployment
image: myreg:5000/my-deployment:0.0.1-SNAPSHOT
ports:
- containerPort: 8080
name: http
resources:
requests:
memory: 1024Mi
limits:
memory: 1024Mi
Java 8 update 131+ has a flag -XX:+UseCGroupMemoryLimitForHeap to use the Docker limits that come from the Kubernetes deployment.
My Docker experiments show me what is happening in Kubernetes
If I run the following in Docker:
docker run -m 1024m anapsix/alpine-java:8_server-jre_unlimited java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm -version
I get:
VM settings:
Max. Heap Size (Estimated): 228.00M
This low value is because Java sets -XX:MaxRAMFraction to 4 by default and I get about 1/4 of the ram allocated...
If I run the same command with -XX:MaxRAMFraction=2 in Docker:
docker run -m 1024m anapsix/alpine-java:8_server-jre_unlimited java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm -XX:MaxRAMFraction=2 -version
I get:
VM settings:
Max. Heap Size (Estimated): 455.50M
Finally setting MaxRAMFraction=1 quickly causes Kubernetes to Kill my container.
docker run -m 1024m anapsix/alpine-java:8_server-jre_unlimited java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm -XX:MaxRAMFraction=1 -version
I get:
VM settings:
Max. Heap Size (Estimated): 910.50M

The reason Kubernetes kills your pods is the resource limit. It is difficult to calculate because of container overhead and the usual mismatches between decimal and binary prefixes in specification of memory usage. My solution is to entirely drop the limit and only keep the requirement(which is what your pod will have available in any case if it is scheduled). Rely on the JVM to limit its heap via static specification and let Kubernetes manage how many pods are scheduled on a single node via resource requirement.
At first you will need to determine the actual memory usage of your container when running with your desired heap size. Run a pod with -Xmx1024m -Xms1024m and connect to the hosts docker daemon it's scheduled on. Run docker ps to find your pod and docker stats <container> to see its current memory usage wich is the sum of JVM heap, other static JVM usage like direct memory and your containers overhead(alpine with glibc). This value should only fluctuate within kibibytes because of some network usage that is handled outside the JVM. Add this value as memory requirement to your pod template.
Calculate or estimate how much memory other components on your nodes need to function properly. There will at least be the Kubernetes kubelet, the Linux kernel, its userland, probably an SSH daemon and in your case a docker daemon running on them. You can choose a generous default like 1 Gibibyte excluding the kubelet if you can spare the extra few bytes. Specify --system-reserved=1Gi and --kube-reserved=100Mi in your kubelets flags and restart it. This will add those reserved resources to the Kubernetes schedulers calculations when determining how many pods can run on a node. See the official Kubernetes documentation for more information.
This way there will probably be five to seven pods scheduled on a node with eight Gigabytes of RAM, depending on the above chosen and measured values. They will be guaranteed the RAM specified in the memory requirement and will not be terminated. Verify the memory usage via kubectl describe node under Allocated resources. As for elegancy/flexibility, you just need to adjust the memory requirement and JVM heap size if you want to increase RAM available to your application.
This approach only works assuming that the pods memory usage will not explode, if it would not be limited by the JVM a rouge pod might cause eviction, see out of resource handling.

What we do in our case is we launch with high memory limit on kubernetes, observe over time under load and either tune memory usage to the level we want to reach with -Xmx or adapt memory limits (and requests) to the real memory consumption. Truth be told, we usually use the mix of both approaches. The key to this method is to have a decent monitoring enabled on your cluster (Prometheus in our case), if you want high level of finetuning you might also want to add something like a JMX prometheus exporter, to have a detailed insight into metrics when tuning your setup.

Important concepts
The memory request is mainly used during (Kubernetes) Pod scheduling.
The memory limit defines a memory limit for that cgroup.
According to the article Containerize your Java applications the best way to configure your JVM is to use the following JVM args:
-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0
Along with that you should always set the JVM to crash if it runs out of memory. There is nothing worse than a health endpoint that thinks it's healthy, but the JVM has run out of memory!
-XX:+CrashOnOutOfMemoryError
Note, there is a bug where you need to specify 75.0 and not 75
To simulate what happens in Kubernetes with limits in the Linux container run:
docker run --memory="300m" eclipse-temurin:17-jdk java -XX:+UseContainerSupport -XX:MinRAMPercentage=50.0 -XX:MaxRAMPercentage=75.0 -XX:+CrashOnOutOfMemoryError -XshowSettings:vm -version
result:
VM settings:
Max. Heap Size (Estimated): 218.50M
Using VM: OpenJDK 64-Bit Server VM
It also works on old school Java 8:
docker run --memory="300m" eclipse-temurin:8-jdk java -XX:+UseContainerSupport -XX:MinRAMPercentage=50.0 -XX:MaxRAMPercentage=75.0 -XX:+CrashOnOutOfMemoryError -XshowSettings:vm -version
This way the container will read your requests from the cgroups (cgroups v1 or cgroups v2). Having a limit is extremely important to prevent evictions and noisy neighbours. I personally set the limit 10% over the request.
Older versions of the Java like Java 8 don't read the cgroups v2 and Docker desktop uses cgroups v2.
To force Docker Desktop to use legacy cgroups1 set {"deprecatedCgroupv1": true} in ~/Library/Group\ Containers/group.com.docker/settings.json

I think the issue here is that the kubernetes memory limits are for the container and MaxRAMFraction is for jvm. So, if jvm heap is the same as kubernetes limits then there wont be enough memory left for the container itself.
One thing you can try is increasing
limits:
memory: 2048Mi
keeping requests limit the same. Fundamental difference between requests and limits is that requests will let you go over the limit if there is memory available at the node level while limits is a hard limit. This may not be the ideal solution and you will have to figure out how much memory is your pod consuming on top of jvm, but as a quick fix increasing limits should work.

unexpected error has been detected by Java Runtime Environment

I am generating report(CSV) through java and i am using hibernate for fetching the data from data base.
Part of my code is as below :
ScrollableResults items = null;
String sql = " from " + topBO.getClass().getName() + " where " + spec;
StringBuffer sqlQuery = new StringBuffer(sql);
Query query = sessionFactory.getCurrentSession().createQuery(sqlQuery.toString());
items = query.setFetchSize( 1000 ).setCacheable(false).scroll(ScrollMode.FORWARD_ONLY);
list = new ArrayList<TopBO>();
// error occurs in while loop. at the time of fetching more data.
while(items.next())
{
TopBO topBO2 =(TopBO) items.get(0);
list.add(topBO2 );
topBO2 = null;
}
sessionFactory.evict(topBO.getClass());
Environment info
JVM config : Xms512M -Xmx1024M -XX:MaxPermSize=512M -XX:MaxHeapSize=1024M
Jboss : JBoss 5.1 Runtime Server
Oracle : 10g
JDK : jdk1.6.0_24(32-bit/x86)
Operating System : Window 7(32-bit/x86)
Ram : 4gb
Error : When i fetch the data up to 50k it works fine. but when i am fetching the data more then it. it gives me the error :
#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 4096000 bytes for GrET in C:\BUILD_AREA\jdk6_11\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
#
# Internal Error (allocation.inline.hpp:42), pid=1408, tid=6060
# Error: GrET in C:\BUILD_AREA\jdk6_11\hotspot\src\share\vm\utilities\growableArray.cpp
#
# Java VM: Java HotSpot(TM) Client VM (11.0-b16 mixed mode windows-x86)
# An error report file with more information is saved as:
# D:\jboss-5.1.0.GA\bin\hs_err_pid1408.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
When i set the Xms512M -Xmx768M -XX:MaxPermSize=512M -XX:MaxHeapSize=768M It throws me another exception :
java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError generally caused due to lacking required Heap space. What you can do is to increase your jvm heap size using flag -Xmx1548M or more MB than 1548.
But you seems like running out of System memory so you should use a better JVM that handles memory more efficiently and I suggest a JVM upgrade. How about upgrading your JVM 1.6 to some more recent versions?

The solution of this problem can be uncommon. Try to use recommendations from article OutOfMemoryError: Out of swap space - Problem Patterns.
There are multiple scenarios which can lead to a native
OutOfMemoryError.
Native Heap (C-Heap) depletion due to too many Java EE applications deployed on a single 32-bit JVM (combined with large Java
Heap e.g. 2 GB) * most common problem *
Native Heap (C-Heap) depletion due to a non-optimal Java Heap size e.g. Java Heap too large for the application(s) needs on a single
32-bit JVM
Native Heap (C-Heap) depletion due to too many created Java Threads e.g. allowing the Java EE container to create too many Threads
on a single 32-bit JVM
OS physical / virtual memory depletion preventing the HotSpot VM to allocate native memory to the C-Heap (32-bit or 64-bit VM)
OS physical / virtual memory depletion preventing the HotSpot VM to expand its Java Heap or PermGen space at runtime (32-bit or
64-bit VM)
C-Heap / native memory leak (third party monitoring agent / library, JVM bug etc.)
I would begin with such recommendation to troubleshooting:
Review your JVM memory settings. For a 32-bit VM, a Java Heap of 2 GB+
can really start to add pressure point on the C-Heap; depending how
many applications you have deployed, Java Threads etc… In that case,
please determine if you can safely reduce your Java Heap by about 256
MB (as a starting point) and see if it helps improve your JVM memory
“balance”.
It is also possible to try (but it is more difficult in respect of labor costs) to upgrade your environment to 64-bit versions of OS and JVM, because 4Gb of physical RAM will be better utilized on x64 OS.

multiple java instances -Xms -Xmx

I am running at the same computer a java game-server and a game-client
the game-client with
java -Xms512m -Xmx1024m -cp etc etc
and the game-server
java -Xmx1024M -Xms1024M -jar etc etc
Computer Properties:
Windows 7 64 bit
8GB RAM
CPU i5-2500 # 3.3GHz
Intel HD Graphics
Problem: The game-client experience serious lags. At the game-server is also connected via LAN another player with no lag issues!
Has the problem of the lag to do anything with java virtual machine? Am I using one instance of machine or two?
Can I setup something different in order to optimize the performance?
I am thinking that the problem has to do with the fact that one machine is running and its max memory is not enough for both instances, but I do not really know how to solve that.
Edit: No app run out of memory.
Solution:
1:
Updated Java version from:
java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
to
java version "1.7.0_15"
Java(TM) SE Runtime Environment (build 1.7.0_15-b03)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
2:
Changed the server properties in order to minimize requirements, this seems to be the main reason.
3:
Increased memory:
game-client with java -Xms1024m -Xmx1024m -cp etc etc
and the game-server java -Xmx2048M -Xms2048M -jar etc etc
Server runs at about 700MB now.

Has the problem of the lag to do anything with java virtual machine?
Possibly. You haven't presented enough evidence to be sure one way or the other.
The puzzling thing is that the your client running on a different machine is not laggy.
Am I using one instance of machine or two?
You are running two copies of java then you will have two JVMs.
Can I setup something different in order to optimize the performance?
The answer is probably yes. But you haven't provided enough information to allow us to make solid suggestions.
Lagginess can be caused by a number of things, including:
A network with high latency.
A JVM that has a heap that is too small.
An application that is generating lots of garbage and triggering an excessive number of garbage collection.
A mix of applications that is competing for resources; e.g. physical memory, CPU time or disc or network I/O time.
If you are going to get to the root cause of your problem, you will need to do some monitoring to figure out which of the above is the likely problem. Use the task manager or whatever to check whether the system is CPU bound, short of memory, doing lots of disk or network stuff, etc. Use VisualVM to see what is going on inside the JVMs.
Alternatively, you could try to fix with some totally unscientific "knob twiddling":
try making the -Xms and -Xmx parameters the same (that may reduce lagginess at the start ...)
try increasing the sizes of the JVMs' heaps; e.g. make them 2gb instead of 1gb
try using a more recent version of Java
try using a 64 bit JVM so that you can increase the heap size further
try enabling the CMS or G1 collectors (depending on what version of JVM you are using).
If I knew more about what you were currently using, I could possibly give more concrete suggestions ...

You are using two java apps in same computer, resulting in 2 JVMs running.
For a 64 bit system with 8GB RAM, its recommended to use max 2GB(25% of Physical memory OR 75% of free physical memory up to 2 GB) for JVM for better performance.
You may have to look on JVM size adjustment. For better performance, Xms and Xmx size can be kept same with a max size bracket.
Assigning memory size to Heap is not the only area to think of. JVM uses more memory than just Heap. Others memory areas like Thread Stack, Method areas, Class Loader Subsystem, Native Method Stack etc.
While both of the apps(game-server, game-client) are running, there is a chance of issue in memory management by OS between both apps, resulting in slowness.
In that case, client app can be deployed in another core, if available.

What is JVM -server parameter?

I saw Java -server in http://shootout.alioth.debian.org/ for programming language benchmark.
I know that -server is a parameter for running JVM. I want to know:
When we use -server parameter and how it work?
Can we use this parameter for java desktop application?
thanks.

It just selects the "Server Hotspot VM". See documentation (Solaris/Linux) for java.
According to Wikipedia:
Sun's JRE features 2 virtual machines,
one called Client and the other
Server. The Client version is tuned
for quick loading. It makes use of
interpretation, compiling only
often-run methods. The Server version
loads more slowly, putting more effort
into producing highly optimized JIT
compilations, that yield higher
performance.
See: http://en.wikipedia.org/wiki/HotSpot

The -server flag will indicate to the launcher that the hw is a server class machine which for java 6 means at least 2 cores and at least 2 GB physical memory (ie most machines these days). On server class machines the deafult selection is
The throughput gc.
initial heap size of 1/64th of phys mem up to 1 GB
maximum heap size of 1/4th of phys mem up to max of 1 GB.
The server run time compiler.
Note that on 32 bit windows there is no server vm so the client vm is the default.
On the other 32 bit machines the server vm is chosen if the hw is server class, otherwise it's client. On 64 bit machines there is no client vm so the server vm is the default.
A link to the hot spot faq: HotSpot

You can check this blog for additional info: http://victorpillac.wordpress.com/2011/09/11/notes-on-the-java-server-flag/
Basically on most recent machines different from 32bits windows the flag will be turned on by default.
For 32bits windows you will need to download the JDK to get the server system.

More info on server vms : http://download.oracle.com/javase/1.3/docs/guide/performance/hotspot.html#server