Develop Reference

Ways to dockerize java apps

What are the principles for app deployment in docker?
I see two concepts
Create image per app version
Create app binaries and somehow deploy them to container(utilize i.e. Tomcats hot deploy)
Maybe there are others, I personally like the first, but there must me tremendous amount of data, if you release very often. How would one choose one over the another?
I'd like to know how others deploys their java application so I can make my personal opinion.

Update 2019: See "Docker memory limit causes SLUB unable to allocate with large page cache"
I mentioned in "Docker support in Java 8 — finally!" last May (2019), that new evolutions from Java 10, backported in Java 8, means Docker will report more accurately the memory used.
As mbluke adds in the comments:
The resource issues have been addressed in later versions of Java.
As of Java SE 8u131, and in JDK 9, the JVM is Docker-aware with respect to Docker CPU limits transparently.
Starting with Java JDK 8u131+ and JDK 9, there’s an experimental VM option that allows the JVM ergonomics to read the memory values from CGgroups.
To enable it on, you must explicit set the parameters -XX:+UnlockExperimentalVMOptions and -XX:+UseCGroupMemoryLimitForHeap on the JVM Java 10 has these set by default and there is no need for the flags.
January 2018: original answer
As any trade-off, it depends on your situation/release cycle.
But do consider also Java might be ill-fitted for a docker environment in the first place, depending on its nature.
See "Nobody puts Java in a container"
So we have finished developing our JVM based application, and now package it into a docker image and test it locally on our notebook. All works great, so we deploy 10 instances of that container onto our production cluster. All the sudden the application is throttling and not achieving the same performance as we have seen on our test system. And our test system is even this high-performance system with 64 cores…
What has happened?
In order to allow multiple containers to run isolated side-by-side, we have specified it to be limited to one cpu (or the equivalent ratio in CPU shares). Unfortunately, the JVM will see the overall number of cores on that node (64) and use that value to initialize the number of default threads we have seen earlier. As started 10 instances we end up with:
10 * 64 Jit Compiler Threads
10 * 64 Garbage Collection threads
10 * 64 ….
And our application,being limited in the number of cpu cycles it can use, is mostly dealing with switching between different threads and does cannot get any actual work done.
All the sudden the promise of containers, “Package once, run anywhere’ seem violated…
So to be specific, how to cope with the amount of data generated when you do build image per release? If you build your app everytime on top of tomcat image, the disk space needed for store the images will grow quickly, right?
2 techniques:
multi-stage build to make sure your application does not include anything but what is need at runtime (and not any compilation files). See my answer here;
bind mounts: you could simply copy your wars in a volume mounted by a single Tomcat container.

How to change Java heap size for Tomcat 9 installed as Windows Service from command line?

I have Tomcat 9 installed as service on Windows 7 64 bit. I want to:
see what heap size is currently configured and active
update the service configuration persistently to use a different heap size
verify that Tomcat is actually using the new heap size.
do all that from the command line.
For 1 and 3: I naively tried to use jconsole, but I don't find the process there because Tomcat is running as Local System Service. While I found out how to run jconsole as Local System Account, it seems that JMX is deactivated when Tomcat is installed as service.
So finding out the currently used memory sizes by JMX seems to be at least very complicated (possibly enabling JMX remote, which should be over TLS...).
For 2: I suppose this is the corresponding place of Tomcat's documentation, which reads:
To update the service parameters, you need to use the //US// parameter.
Update the service named 'Tomcat9'
C:\> tomcat9 //US//Tomcat9 --Description="Apache Tomcat Server - http://tomcat.apache.org/ " ^
--Startup=auto --Classpath=%JAVA_HOME%\lib\tools.jar;%CATALINA_HOME%\bin\bootstrap.jar
But I don't understand that text sufficiently to apply it to my problem. Especially, I don't want to change other parameters (like description, startup etc.).
As far as I understand, when run as a service, configuration is stored in the Windows registry, so the usual configuration in tomcat/conf does not or at least does only partly apply.
Please note that this question is not about installing Tomcat, but about modifying an existing installation. Also I am not interested in some hacky way to get the desired result (somehow), but in the best practice to do that; and it would be perfect to have links to reference documentation for that.

Given the documentation that you link to
--JvmMx
Maximum memory pool size in MB. (Not used in exe mode.)
should help to control the heap size.
For fetching the current values, tools such as jvmtop might be easiest answer.

For 1 and 3 there is jmap. You just need to know the process Id of the tomcat instance running.
jmap -heap 7082
Here is the output from a running jvm I have right now(the relevant lines):
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 1073741824 (1024.0MB) // that is -Xmx flag
....
NewSize = 357564416 (341.0MB) // 1
MaxNewSize = 357564416 (341.0MB)
OldSize = 716177408 (683.0MB) // 2
1 + 2 = -Xms flag
Unfortunately I can't answer 2, since I've never started tomcat on windows - as a service (I hardly know what that means for windows). But assuming this is a process that is started by windows as a script...
Shouldn't : tomcat9 -Xms512M -Xmx2G... work? Again, just a hint, not sure. The last thing to notice is that the heap can be changed only at start-up of the jvm, you can't do that at runtime while tomcat is running obviously (just in case...).

Glassfish stop itself

I'm new in glassfish.
I have a vps in digitalocean with Ram 512 mb.
I just running 1 domain and 1 simple web service application.
But glassfish stop without my command.
Any suggest?
I using glassfish 4.1.1
Edit for Trevor. I forget to say about error log.
I checked the error log. No log that say error or why the glassfish stop. It's running okay when i restart, but after few hours it happen again

There is a good chance you are running out of memory.
By default, the server has -Xmx512m set, which means the heap size can increase to 512MB. Since that is all you have available on your DigitalOcean machine, it will start with a lower amount, and increase as you deploy your application to it. Once GlassFish tries to use more memory than your DigitalOcean machine can spare, it will die.
Decreasing this to something like -Xmx256m will probably give you more stability. For simple apps, you should be OK with that.
Note: you may want to also decrease the -XX:MaxPermSize=192m to -XX:MaxPermSize=128m. If you are on Java 8, then this doesn't matter any more and the value isn't used. For Java 7, decreasing this will help.
Either change this value through the admin console at http://[$HOSTNAME]:4848 and go to Configurations -> server-config -> JVM Settings, then click the JVM Options tab (you will need to change the value, then click save, then restart GlassFish).
Or change it in the domain.xml directly (being careful to get it right)
glassfish41/glassfish/domains/domain1/config/domain.xml
You will notice that that value is in that file twice. One is the actual server-config, used for the server itself, and the other is the "default-config" which is a template used for creating new configurations. Make sure you change the correct one! If you're unsure, just change both.

Tomcat runs as a service, how to tweak memory settings without UI

I would like to modify JVM heap min and max values as well as permGenSpace in a Tomcat that runs as a service on Windows machine. Is this possible using setenv.bat or the only way is to pass these values from UI?

Out of Memory on Tomcat Shutdown

Short description of my problem: I start up Tomcat with my deployed Wicket application. When I want to shut down tomcat I get this error message:
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.lang.ref.Reference.<clinit>(Reference.java:145)
I am running the following setup:
Ubuntu Linux: 10.04 (lucid) with a 2.6.18-028stab094.3 kernel
Java Version: "1.6.0_26" Java HotSpot(TM) 64-Bit Server VM
Tomcat Version: 7.0.23
jvm_args: -Xms512m -Xmx512m -XX:MaxPermSize=205m (these are added via CATALINA_OPTS, nothing else)
Wicket 1.5.1
Tomcat is configured with two virtual hosts on subdomains with ModProxy
My application is deployed as ROOT.war in the appbase directory (it makes no difference if I deploy one or both applications)
'''No application deployed does not result in OOM on shutdown''', unless I mess around with the jvm args
The size of the war is about 500k, all libraries are deployed in tomcat/common/lib (directory which I added to common.loader in conf/catalina.properties)
ulimit -u -> unlimited
When I check the Tomcat manager app it says the following about the JVM memory:
Free memory: 470.70 MB Total memory: 490.68 MB Max memory: 490.68 MB
(http connector) Max threads: 200 Current thread count: 6 Current thread busy: 1
'top' or 'free -m' is similar:
Mem: 2097152k total, 1326772k used, 770380k free, 0k buffers
20029 myuser 18 0 805m 240m 11m S 0 11.7 0:19.24 java
I tried to start jmap to get a dump of the heap, it also fails with an OutOfMemoryError. Actually as long as one or both of my applications are deployed any other java process fails with the same OOM Error (see top).
The problem occurs while the application is deployed. So something is seriously wrong with it. However the application is actually running smoothly for quite a while. But I have seen OOMs in the application as well, so I don't trust the calm.
My application is using a custom filter class? Could that be it?
For completeness (hopefully), here's the list of libraries in my common/lib:
activation-1.1.jar
antlr-2.7.6.jar
antlr-runtime-3.3.jar
asm-3.1.jar
asm-commons-3.1.jar
asm-tree-3.1.jar
c3p0-0.9.1.1.jar
commons-collections-3.1.jar
commons-email-1.2.jar
dependencies-provided.tgz
dom4j-1.6.1.jar
ejb3-persistence-1.0.2.GA.jar
geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jta_1.1_spec-1.1.1.jar
hibernate-annotations-3.4.0.GA.jar
hibernate-commons-annotations-3.1.0.GA.jar
hibernate-core-3.3.0.SP1.jar
hibernate-entitymanager-3.4.0.GA.jar
hibernate-search-3.1.0.GA.jar
javassist-3.4.GA.jar
joda-time-1.6.2.jar
jta-1.1.jar
log4j-1.2.16.jar
lombok-0.9.3.jar
lucene-core-2.4.0.jar
mail-1.4.1.jar
mysql-connector-java-5.1.14.jar
persistence-api-1.0.jar
quartz-2.1.1.jar
servlet-api-2.5.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
stringtemplate-4.0.2.jar
wicket-auth-roles-1.5.1.jar
wicket-core-1.5.1.jar
wicket-datetime-1.5.1.jar
wicket-extensions-1.5.1.jar
wicket-request-1.5.1.jar
wicket-util-1.5.1.jar
xml-apis-1.0.b2.jar
I appreciate any hint or even speculation that gives me additional ideas what to try.
Update: I tested some more and found that this behaviour only occurs while one or both of my applications are deployed. The behaviour does not occur on "empty" tomcat (that was a mistake on my part messing with jvm args)
Update2: I am currently experimenting trying to reproduce this behaviour in a virtual box, I want to debug this with a profiler. I am still not convinved that it should be impossible to run my setup on 2GB RAM.
Update3 (10/01/12): I am trying to run jenkins instead of my own application. Same behaviour, so it is definitely server configuration issues. Jenkins jobs fail when maven is called, so I need not even try the shutdown hack suggested below because I need a second java process while running Jenkins. It was suggested to me that because this is a Virtual Server ulimits may be imposed from outside and I would not be able to see them. I think I'll ask a new question regarding this. Thx all.
Update4 (02/05/12): see below for the answer that contains the hint. I'll clarify some more up here: I am now 95% sure that the errors occur because I am reaching my thread limit. However because this is a virtual server the method described below would not work to check this value because it is not visible with ulimit, that was what was confusing me and only today I found out that this is the "numproc" value that I can see in the Parallels Power Panel that I can log into for my virtual server. There were Resource Alerts for numproc but I did not see those either until just now. The value has a hard limit of 96 which I cannot change of course. The current value of numproc corresponds to the number of processes I see with "top" after toggling "H" to see threads. I had a very hard time finding this because this numproc value is hidden deep inside the panel. Sadly 96 is a rather low number if you want to run a tomcat with apache and mysql. I am also very sad that I cannot even find this value in the small print of my hosting contract and it is rather relevant to my application I dare say. So I guess I'll need a server upgrade.
Thanks all for your helpful answers in the end everyone helped me a bit to find out what the problem was.

The tomcat shutdown procedure consits of sending an command/word via a tcp port to the running tomcat VM. This port is configured in the server.xml (if I remember corretly, writting on my phone right now). So far so good.
Unfortunately, the shutdown script does this by starting a 2. VM using the same java options used for the tomcat. Your system simply has not enough memory for this.
As a sollution you could write your own stop script using telnet or something.
I could help with later if needed.
Hope that helps.
Viele grüsse Bert

Seems you have too many threads open.
Use this command :
ulimit -u
What is the result ?
Should be something like :
max user processes (-u) 100
If this is correct, you can edit this file :
/etc/security/limits.conf
and the the following modifications :
#<domain> <type> <item> <value>
user soft nproc 10000
user hard nproc 10000

You can probably survive for a while like this. All you need to do is kill the tomcat process whenever you need to restart it. It is not a nice approach, but the main concern is that your application runs correctly.
It seems to me though, that on the long run, you might need to order a hosting plan with more RAM available.

I was having a similar problem with a tomcat installation just last week. I managed to fix it by giving tomcat a smaller heap. Something like this:
export CATALINA_OPTS=”-Xms256m -Xmx512m”
Before starting Tomcat may help. In the meantime you'll have to kill it the old fashioned way, with a kill -9 ;)
EDIT: you could also take look here, it appears tomcat automatically creates a bunch of "spare" threads, but you can limit those as well as your max thread count in the config. Hope it helps.