I am trying to execute kill command on OnOutOfMemoryError for a SpringBoot application.
Below is the .conf file having command.
JAVA_OPTS="-Xmx512M -XX:OnOutOfMemoryError=\"kill $(lsof -t -i:8080)\""
If I run Spring boot application as "java -jar" with Java Hostspot VM commands it works fine, but while running as Linux systemd service, application is not getting killed.
Exception : "Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space"
In my scenario, I run an instance of GeoServer using tomcat with a container in the docker swarm.
I expect swarm to recreate the container after any problem with this instance, but after an OutOfMemory error, the container will never be restarted because the JVM is still running, although the application no longer responds.
For this case, I use OnOutOfMemoryError and the container is eliminated after this type of error, so that the swarm can recreate it.
Tomcat environment to set the JVM parameters.
CATALINA_OPTS="-XX:OnOutOfMemoryError=\"kill -9 %p\"
-Djava.awt.headless=true \
-Dfile.encoding=UTF-8 -server \
-Xms1024m -Xmx3072m -Xss1024k -XX:NewSize=768m \
-XX:+UseParallelGC -XX:MaxGCPauseMillis=500"
You can try to adapt it to your use.
Importantly, i use these software:
Tomcat 9
OpenJDK-11
GeoServer 2.16.x
Debian GNU/Linux 10 (buster)
Docs for consult JVM openJDK-11 configuration options: https://manpages.debian.org/testing/openjdk-11-jre-headless/java.1.en.html
Related
I was following the steps mentioned here How to get a heap dump from Kubernetes k8s pod?
I'm able to get the the process id using top command inside the pod. However, when I run jmap I get this:
~ $ jmap
sh: jmap: not found
I access the pod with this command: kubectl exec -it -- sh
I also tried this command:
kubectl exec -it <pod> -- jmap -dump:live,format=b,file=heapdump.bin 1
But I was getting:
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"jmap\": executable file not found in $PATH": unknown command terminated with exit code 126
Is there any other way to get the java heap dump from the pod?
Normally containers are limited on the tools available(like you get 'more', but you don't get 'less'), so the tools available for you depends on your container.
The 2 tools that are used to get a heap dump are jmap and jcmd, check if you got jcmd in the container.
https://www.adam-bien.com/roller/abien/entry/taking_a_heap_dump_with
If not, I recommend to put the java app in a container that has either jmap or jcmd and then run it; even if the container is "heavier" that won't affect the java app nor the heap dump so it will be the same.
If that's not an option, maybe this will be
https://techblog.topdesk.com/coding/extracting-a-heap-dump-from-a-running-openj9-java-process-in-kubernetes/
(not mine).
I am trying to monitor native memory of tomcat using jcmd but getting following exception:
Command: jcmd 14533<pid of tomcat> VM.native_memory summary
I have installed 2 jdk's jdk-11 AND jdk-8.
Tomcat is running by jdk-8 and jcmd used jdk-11 then i got following exception:
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /proc/31752/root/tmp/.java_pid31752: target process 31752 doesn't respond within 10500ms or HotSpot VM not loaded
at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:100)
at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
at jdk.jcmd/sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:114)
at jdk.jcmd/sun.tools.jcmd.JCmd.main(JCmd.java:98)
I attched -XX:+StartAttachListener with java process but it didn't work.
It can be a user permission issue: jmap has to be run with the same user as the java process. e.g. if the owner is usera and this is running in a docker contained execute jmap as user usera
docker exec --user usera -it container_id jmap -dump:format=b,file=/home/usera/memorydump.bin 1
If you are running tomcat as a service on linux check following config in unit file i.e. tomcat.service
PrivateTmp=yes, should be removed. As PrivateTmp=yes create private /tmp folder which is not accessible by tool
If you have ProtectSystem=strict, add ReadWritePaths=/tmp/
Go through man page for systemd unit file config
I use DCOS with Spring boot applications inside Docker containers. I noticed that sometimes containers are killed, but there are no errors in the container logs, only:
Killed
W1114 19:27:59.663599 119266 logging.cpp:91] RAW: Received signal SIGTERM
from process 6484 of user 0; exiting
HealthCheck is enabled only for SQL connection and disk space. The disk is ok on all nodes, in case of SQL problems error should appear in the logs. Other reason could be the memory but it also looks fine.
From marathon.production.json:
"cpus": 0.1,
"mem": 1024,
"disk": 0
And docker-entrypoint.sh:
java -Xmx1024m -server -XX:MaxJavaStackTraceDepth=10 -XX:+UseNUMA
-XX:+UseCondCardMark -XX:-UseBiasedLocking -Xms1024M -Xss1M
-XX:MaxPermSize=128m -XX:+UseParallelGC -jar app.jar
What could be the reason for container killing and are there any logs on DCOS regarding it?
Solved with java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
Or just use openjdk:11.0-jre-slim
I have an application that is running as a win-service (using Apache-Procrun). I want to monitor that application with JVisualVM. I've used the commandline argument --openpid <PID> without success, ironically the old JConsole managed to connect to that application.
I also checked all issues with the %TMP%\hsperfdata_username -Directory that are described in the JVisualVM-Trouble Shooting Guide. Its all well.
How can i get the JVisualVM to monitor my app?
The JVisualVM is not able to connect to a win-service using the --openpid <PID> argument! The old JConsole can this only in WinXP because of a Bug. The alternative is to specify a JMX Agent for the service.
This VM Arguments have to be passed to the prunsrv.exe
-Dcom.sun.management.jmxremote.port=3333 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
For more detail take a look at: Connecting to JMX Agents Explicitly
I am running tomcat 6.0.18 as a windows service. In the service applet the jvm is configured default, i.e. it is using jvm.dll of the JRE.
I am trying to monitor this application with JConsole but cannot connect to it locally. I added the parameter -Dcom.sun.management.jmxremote (which works when starting tomcat with the start.bat script). But the jvm does not seem to pick up the parameter.
There's a nice GUI to edit the options, no need to muck around in the registry.
Open up the C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\tomcat6.exe (or just double-click on the monitor icon in the task bar). Go to the Java pane, add the following to the list of arguments, and restart Tomcat.
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8086
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
Then you can connect with JConsole or the newer VisualVM.
Here's the prescribed way for changing jvmoptions & interacting with the service:
http://tomcat.apache.org/tomcat-5.5-doc/windows-service-howto.html
I would try going into your registry at HKLM/Software/Apache Software Foundation/Procrun 2.0//Parameters/Java and editing the "Options" multi-string value directly.
If Tomcat is running as a Windows service, and you want to attach to the JVM locally, you need to run VisualVM or JConsole as the System account. You can use Sysinternals PsExec.exe to accomplish this.
psexec.exe -i -s c:\visualvm\bin\visualvm.exe
I'm posting it mainly to record this information to myself, I haven't validated it - but this is supposed to work as well:
http://mysqlandsqlserver.blogspot.com/2010/02/jconsolejmap-and-tomcat-as-windows.html
There is still a rather simple way to connect JConsole to Java process started as Windows Service using the local mode which I discovered here.
Basically it says that in order to connect to Java process launched as a Windows Service you need to launch JConsole as a Windows Service (you can do it using windows native api or using any wrapper like yajsw.)
By the way, this will free you from restarting the Java Process which was critical for me.
Add the following near the top of your catalina.bat
set JAVA_OPTS=%JAVA_OPTS% -Dcom.sun.management.jmxremote ^
-Dcom.sun.management.jmxremote.port=8086 ^
-Dcom.sun.management.jmxremote.ssl=false ^
-Dcom.sun.management.jmxremote.authenticate=false
Stop and restart tomcat (obviously)
Run jconsole.exe. If your tomcat is running as service, then run jconsole.exe as administrator.
Select Remote Process and enter localhost:8086