Develop Reference

Docker - Java container doesn't respect memory limits

I've been reading multiple posts regarding limiting the container's JVM memory, nothing has worked so far, I don't know where I am messing up.
I made a simple "Hello World" in Spring Boot, using a REST controller, it has nothing else.
Such app was exported as a WAR file, running it with the Tomcat JDK 11 image, I can also run it using the JDK image with a FatJar but the problem persists either way.
Expected
Have my Java container not take more than 25 MB memory (for the sake of a number, could be more)
Actual
Such a simple application is taking 200 - 250 MB memory in docker stats
Dockerfile
FROM tomcat:9.0.30-jdk11-corretto
COPY tomcat.war /usr/local/tomcat/webapps/ROOT.war
CMD ["catalina.sh","run"]
docker-compose.yml
version: '3.7'
services:
hello:
build:
context: .
dockerfile: Dockerfile
image: app-test
environment:
- JVM_OPTS=-Xms13m -Xmx25m
ports:
- 8080:8080
I have tried
-Xms13m -Xmx25m
-XX:PermSize=13m -XX:MaxPermSize=25m
Among other flags that I already deleted and forgot about

To restrict memory for docker containers, you need to add following in your docker-compose.yml file
environment:
- <name>=<value>
deploy:
resources:
memory: 256M
Once you add the above code, then containers would be restricted to given memory limit and you can see the result in docker stats output. For more details about the resource restriction refer this docker documentation.
Note: If the container crosses the given memory limit, the kernel will kill the container.

The environment variable that tomcat's catalina.sh script depends on for java options is : JAVA_OPTS.
If you change the compose file to use the following env variable , it should work.
environment:
- JAVA_OPTS=-Xmx128m -Xms128m
Apart from the fact that 25m seems a way too less memory for a JVM with tomcat running.

Regardless of how little memory you set for JVM heap (-Xmx) there are lots of other types of non-heap memory that JVM uses: Java using much more memory than heap size (or size correctly Docker memory limit) - you need to take that into account.
You can also avoid setting -Xmx altogether and instead leverage -XX:MaxRAMPercentage: Clarification of meaning new JVM memory parameters InitialRAMPercentage and MinRAMPercentage

visualvm/jvisualvm: not supported for this JVM

I wanted to monitor the JVM of wildfly running as service with jvisualvm/visualvm but I fail to do this. I tried the following things:
setting the %TMP% and %TEMP% to C:\Windows\Temp (wildfly console
tells me this for java.io.tmpdir)
running a console with sysinternals
pstools as system account: psexec -i -s cmd.exe and started visualvm
from within this new console (checked that the temp folders are
correctly set).
In both cases under local applications the process of wildfly was listed but visualvm only told me "not supported for this jvm".
As soon as I run wildfly from the cli, visualvm has no problems and shows me everything. There is only the jdk from oracle installed (with the corresponding jre).
How can I monitor the process of wildfly running as service (local system account)? Why is it not working with the solutions above?
Thanks a lot (for reading)

Thank you Salah
With your hint (local JMX connection) I've managed to make it work by using the following command for visualvm (no change of TMP/TEMP variables in cmd):
visualvm.exe -cp:a "<path-to-wildfly>\bin\client\jboss-client.jar"
and adding the path to the jmx console (don't forget to set the username/pw for the admin gui)
service:jmx:http-remoting-jmx://localhost:9990

How to set Java heap size (Xms/Xmx) inside Docker container?

As of raising this question, Docker looks to be new enough to not have answers to this question on the net. The only place I found is this article in which the author is saying it is hard, and that's it.

Note that in a docker-compose.yml file - you'll need to leave out the double-quotes:
environment:
- JVM_OPTS=-Xmx12g -Xms12g -XX:MaxPermSize=1024m
or
environment:
- CATALINA_OPTS=-Xmx12g -Xms12g -XX:MaxPermSize=1024m

I agree that it depends on what container you're using. If you are using the official Tomcat image, it looks like it's simple enough, you will need to pass the JAVA_OPTS environment variable with your heap settings:
docker run --rm -e JAVA_OPTS='-Xmx1g' tomcat
See How to set JVM parameters?

Update: Regarding this discussion, Java has upped there game regarding container support. Nowadays (or since JVM version 10 to be more exact), the JVM is smart enough to figure out whether it is running in a container, and if yes, how much memory it is limited to.
So, rather than setting fixed limits when starting your JVM, which you then have to change in line with changes to your container limits (resource limits in the K8s world), simply do nothing and let the JVM work out limits for itself.
Without any extra configuration, the JVM will set the maximum heap size to 25% of the allocated memory. Since this is frugal, you might want to ramp that up a bit by setting the -XX:MaxRAMPercentage attribute. Also, there is -XX:InitialRAMPercentage for initial heap size and -XX:MinRAMPercentage for containers with less than 96MB RAM.
For more information on the topic, here is an excellent overview.

You can also just place those settings in your image so something like the following would exist in your Dockerfile:
ENV JAVA_OPTS="-XX:PermSize=1024m -XX:MaxPermSize=512m"

you can do it by specifying java options environment in docker compose file
env:
- name: _JAVA_OPTIONS
value: "-Xmx1g"
it will change the heap size.

It all depends how your Java application is packaged and how it's configuration files are exposed using Docker.
For example the official tomcat image states that the configuration file is available in the default location: /usr/local/tomcat/conf/
So easy to override entire directory or just one configuration file:
docker run -it --rm -p 8080:8080 -v $PWD/catalina.properties:/usr/local/tomcat/conf/catalina.properties tomcat:8.0

Heap dump on JRE 6 (Windows) without JDK

Is there a way to create a heap dump on a remote machine without JDK installed?
I can't change the installation / settings and it's running on Windows.
So I have pnly access to commandline tools.
Problem is that a Java app on a remote machine freezes (no out of memory exception so -XX:-HeapDumpOnOutOfMemoryError is useless) and we need to create a dump.
-XX:+HeapDumpOnCtrlBreak
is no option too, because it's not supported anymore on JDK6+.
JMX is not allowed due to security reasons.
Any Ideas? Thank you for your help!
Edit:
Windows
No JDK
No JMX

I think I solved the problem.
You have to "patch" your JRE with some files of the JDK (the same version of course - if you are running jre6uXX you need the corresponding files from jdk6uXX )
Copy the following files:
\JDK6uXX\bin\attach.dll --> %JAVAJRE_HOME%\bin\
\JDK6uXX\bin\jmap.exe --> %JAVAJRE_HOME%\bin\
\JDK6uXX\lib\tools.jar --> %JAVAJRE_HOME%\lib\
No files are overwritten, JRE shouldn't be affected by this.
Now you can use jmap just fine to take dumps ;-)
I appreciate your help! Bye

The simplest solution is to use jmap -dump:liv,format=b,file=app.dump on the command line. You can use jps -lvm to find the process id.
An alternative is to connect to it to jvisualvm This will take the dump and analyse it for you. You can also use this tool to read a dump written by jmap so you may end up using it anyway.
Where jvisualvm struggles is for large heap dumps i.e. more than about half you main memory size. I have found using YourKit to handle larger dumps and also give more useful information. An evaluation license might be all you need to diagnose this.
jmx is not allowed due to security reasons
In that case, you can't do this remotely, unless you use YourKit or some other commercial profiler.

You have start your application with jmx console enabled in a port to debug your application. Execute jconsole and connect to the port which you have enabled for debugging. You can also use of jmap to collect heapdump.

JProfiler has a command line utility bin/jpdump that can take an HPROF heap dump. There is no need to install JDK. There is also no need to run the GUI installer of JProfiler, just extract the ZIP distribution and execute jpdump on the command line.
Disclaimer: My company develops JProfiler.
Update 2016-06-23
As of JProfiler 9.2, jpdump and jpenable run with Java 6 as well.

You could use jvisualvm, just enable jmx port and connect to your application, then you will be able to generate a heap file.
You can do that by adding the following parameters:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=8484
-Dcom.sun.management.jmxremote.ssl=false
Then You need to add your tomcat process manually, So right click on you localhost node -> Add JMX Connection -> type your port -> OK.
Your tomcat process will be listed in under localhost node.

jmap -dump:format=b,file=snapshot.jmap
process-pid
Regardless of how the Java VM was started, the jmap tool will produce a head dump snapshot, in the above example in a file called snapshot.jmap. The jmap output files should contain all the primitive data, but will not include any stack traces showing where the objects have been created.

Can I get Tomcat running as a service to dump heap?

I am attempting to have Tomcat, which is currently running as a service on a Windows 2003 box, dump heap on an OutOfMemoryError.
(Tomcat is running Hudson, which is reporting a heap space problem at the tail end of my build. Running the build manually produces no such error. The Hudson guys need a heap dump to get started.)
As instructed elsewhere, I've told the Apache Service Monitor to configure the JVM it uses to run Tomcat to dump heap when an OutOfMemoryError is encountered by adding the following to the JVM options:
-XX:+HeapDumpOnOutOfMemoryError
Then I run the build again. Sure enough, it reports there was a heap error. I scan the entire disk looking for the default java_pid123.hprof file (where obviously 123 is replaced by the PID of the JVM). No .hprof files exist anywhere.
I am caught in a catch 22: I need the heap dump for the Hudson guys to fix their memory leak, but I can't get the heap dump if I run Hudson under Tomcat.
Is there some special way, when Tomcat is running as a Windows service, to get a heap dump from it on an OutOfMemoryError?
The other thing I've tried is to tell it, on the Startup and Shutdown tabs, to use the "Java" option instead of the "jvm" option. I believe this should tell the Service Manager to attempt to start Tomcat with a Java executable command instead of launching the jvm.dll directly. When I do this, the service won't start.
Surely someone else has had a similar problem?

After finally putting this one to bed, I wanted to answer this for others who might have the same problem.
First, if you install Tomcat on Windows, do not use the .exe installer, even though it is promoted by Apache. It will not let you run Tomcat as anything other than the system account, no matter what you do. It appears that the system account does not have privileges to write .hprof files in the current directory, and no amount of Windows security tweaking appears to make this problem go away.
OK, so you've installed Tomcat from the .zip distribution. Install it as a service using the service.bat script. Make sure it is set to run as a specific user that you created specifically for this purpose. Make sure as well that the folder you want Tomcat to write to in the event of a heap dump is writable by that user.
Edit the service.bat file to include the -XX:+HeapDumpOnOutOfMemoryError and the -XX:HeapDumpPath=C:\whatever options in the correct place (where you can put JVM options). That should do the trick.

Have you tried -XX:HeapDumpPath option?
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp

I found the following link, which describes how to configure the tomcat service (includes setting the java parameters). Not sure if it applies to the version you are running.
http://tomcat.apache.org/tomcat-5.5-doc/windows-service-howto.html

When java process running as window service you can generate the heapdump using below steps,
Run the command console as Administrator
version of JDK (for jmap command) and JRE (Java app run environment) should be same.
Get the PID no of running window process for that java application from task manager
Execute below command
jmap -dump:file=d:\heapdump\myHeapDump.hprof -F #PID_No#
If got any exception with JDK/JRE 7 try the same with JDK/JRE 8
Actually I faced some issue in jmap with JDK 7, but when i moved to JDK 8, I were able to successfully generate the heap dump using same command

The .hprof files are dumped in the current directory. Exactly what that means for a windows service is anyone's guess, assuming it means anything.
I suggest posting a new question (on http://superuser.com) asking what "current directory" means for a windows service.

From 20 Tips for Using Tomcat in Production
Add the following to your JAVA_OPTS in catalina.sh (or catalina.bat for Windows): -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/j2ee/heapdumps

if you have installed tomcat with .exe you can configure tomcat service to use account other than local system account and you can assign that user rights on directory "c:\whatever" where you are creating your dump file. one thing here to remember tomcat service don't run with account having administrative privileges. so create a simple user in windows(member of user group) and set tomcat services to user this account. and give that user rights on "c:\whatever" directory. This resolves the user directory rights issue but you have to configure tomcat for Memory dumps errors.