Develop Reference

Execute KILL command from JAVA_OPTS using -XX:OnOutOfMemoryError

I am trying to execute kill command on OnOutOfMemoryError for a SpringBoot application.
Below is the .conf file having command.
JAVA_OPTS="-Xmx512M -XX:OnOutOfMemoryError=\"kill $(lsof -t -i:8080)\""
If I run Spring boot application as "java -jar" with Java Hostspot VM commands it works fine, but while running as Linux systemd service, application is not getting killed.
Exception : "Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space"

In my scenario, I run an instance of GeoServer using tomcat with a container in the docker swarm.
I expect swarm to recreate the container after any problem with this instance, but after an OutOfMemory error, the container will never be restarted because the JVM is still running, although the application no longer responds.
For this case, I use OnOutOfMemoryError and the container is eliminated after this type of error, so that the swarm can recreate it.
Tomcat environment to set the JVM parameters.
CATALINA_OPTS="-XX:OnOutOfMemoryError=\"kill -9 %p\"
-Djava.awt.headless=true \
-Dfile.encoding=UTF-8 -server \
-Xms1024m -Xmx3072m -Xss1024k -XX:NewSize=768m \
-XX:+UseParallelGC -XX:MaxGCPauseMillis=500"
You can try to adapt it to your use.
Importantly, i use these software:
Tomcat 9
OpenJDK-11
GeoServer 2.16.x
Debian GNU/Linux 10 (buster)
Docs for consult JVM openJDK-11 configuration options: https://manpages.debian.org/testing/openjdk-11-jre-headless/java.1.en.html

Jconsole cannot connect

I am using VNC to connect to a server. I am able to VNC into the server, and open Jconsole, but cannot get it to connect to my process whether I include the PID or try connecting using the GUI. The weird thing is all the processes show up in the GUI. See below.
I get this when I try connecting:
Followed by
Is there something I need to do in order to get Jconsole working locally on a Linux server over VNC?

Fix found here: You need to pass to the VM:
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false
--
Edit - Fixed the =fals error mentioned in comments.

In my case, I also had to add this option to get Jconsole to connect to the process.
-Djava.rmi.server.hostname=localhost

Remote JConsole
Add the following parameters to your java Application and run it
-Djava.rmi.server.hostname=DOMAIN_PUBLIC_URL -Dcom.sun.management.jmxremote.port=PORTNUMBER -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
DOMAIN_PUBLIC_URL - Like your instance public address
PORTNUMBER - 9955
Launch jconsole in your ubuntu from terminal command and go to remote connection and paste the DOMAIN_PUBLIC_URL and PORT and Connect it without username and password like insecure connection.
Make sure to expose the port number on your AWS Server Instance.

Adding below property as VM argument in Run Configurations --> Arguments --
-Dcom.sun.management.jmxremote=true

In my case, I had to make it false and also add a hostname to localhost. Add this in VM arguments in Run Configurations.
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=localhost

Debugging VisualVM remote connection

I am running Tomcat 6 on a Linux server in Amazon's cloud. I am trying to connect to it with VisualVM from my Mac at my office. I have allowed opened up all TCP ports between my desktop and the server, but I am unable to get VisualVM to do anything.
On the linux box, I have started Tomcat with:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9191
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
On my Mac, I launch VisualVM and choose File -> New JMX Connection...
I enter amazonhostname.com:9191 in the box. It says "Adding amazonhostname.com:9191..." and sits there for several minutes before timing out.
From my Mac, I can "telnet amazonhostname.com 9191" just fine, and I tried some other random ports, and they all worked. I'm pretty convinced it's not a firewall issue, unless I need to open UDP or ICMP or something like that. (Really??)
There is nothing in catalina.log on the server, or the system console on my Mac, related to jmx. netstat on the server shows 9191 and some other ports open on all interfaces.
There is no jstatd on the remote server. Is that a pre-requisite for JMX? (I am totally new to Java and I'm not even sure what those things mean, but I saw it on a blog post somewhere. I'm grasping at straws now.)

Set the hostname property before the VM starts:
java -Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9191 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=the.public.ip \
-jar program.jar
Add the relevant rules to your security group.

What worked well for me in RHEL7 environment is to implement the JmxRemoteLifecycleListener on Tomcat using the example provided in the documentation. I have also opened up ports 10001 and 10002 on the firewall.
JMX Remote Lifecycle Listener

How to activate JMX on my JVM for access with jconsole?

How to activate JMX on a JVM for access with jconsole?

The relevant documentation can be found here:
http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
Start your program with following parameters:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.rmi.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
For instance like this:
java -Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9010 \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-jar Notepad.jar
-Dcom.sun.management.jmxremote.local.only=false is not necessarily required
but without it, it doesn't work on Ubuntu. The error would be something like
this:
01 Oct 2008 2:16:22 PM sun.rmi.transport. customer .TCPTransport$AcceptLoop executeAcceptLoop
WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=37278] throws
java.io.IOException: The server sockets created using the LocalRMIServerSocketFactory only accept connections from clients running on the host where the RMI remote objects have been exported.
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:89)
at sun.rmi.transport. customer .TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport. customer .TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:636)
see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6754672
Also be careful with -Dcom.sun.management.jmxremote.authenticate=false which
makes access available for anyone, but if you only use it to track the JVM on
your local machine it doesn't matter.
Update:
In some cases I was not able to reach the server. This was then fixed if I set this parameter as well: -Djava.rmi.server.hostname=127.0.0.1

Running in a Docker container introduced a whole slew of additional problems for connecting so hopefully this helps someone. I ended up needed to add the following options which I'll explain below:
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=${DOCKER_HOST_IP}
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
DOCKER_HOST_IP
Unlike using jconsole locally, you have to advertise a different IP than you'll probably see from within the container. You'll need to replace ${DOCKER_HOST_IP} with the externally resolvable IP (DNS Name) of your Docker host.
JMX Remote & RMI Ports
It looks like JMX also requires access to a remote management interface (jstat) that uses a different port to transfer some data when arbitrating the connection. I didn't see anywhere immediately obvious in jconsole to set this value. In the linked article the process was:
Try and connect from jconsole with logging enabled
Fail
Figure out which port jconsole attempted to use
Use iptables/firewall rules as necessary to allow that port to connect
While that works, it's certainly not an automatable solution. I opted for an upgrade from jconsole to VisualVM since it let's you to explicitly specify the port on which jstatd is running. In VisualVM, add a New Remote Host and update it with values that correlate to the ones specified above:
Then right-click the new Remote Host Connection and Add JMX Connection...
Don't forget to check the checkbox for Do not require SSL connection. Hopefully, that should allow you to connect.

Note, Java 6 in the latest incarnation allows for jconsole to attach itself to a running process even after it has been started without JMX incantations.
If that is available to you, also consider jvisualvm as it provides a wealth of information on running processes, including a profiler.

I'm using WAS ND 7.0
My JVM need all the following arguments to be monitored in JConsole
-Djavax.management.builder.initial=
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8855
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

On Linux, I used the following params:
-Djavax.management.builder.initial=
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
and also I edited /etc/hosts so that the hostname resolves to the host address (192.168.0.x) rather than the loopback address (127.0.0.1)

along with below command line parameters ,
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Sometimes in the linux servers , imx connection doesn't get succeeded. that is because , in cloud linux host, in /etc/hosts so that the hostname resolves to the host address.
the best way to fix it is, ping the particular linux server from other machine in network and use that host IP address in the
-Djava.rmi.server.hostname=IP address that obtained when you ping that linux server.
But never rely on the ipaddress that you get from linux server using ifconfig.me. the ip that you get there is masked one which is present in the host file.

The below options works for me:
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.rmi.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname={host name}
and remember to open 9010 port in the server
sudo ufw allow 9010/udp
sudo ufw allow 9010/tcp
sudo ufw reload

Run your java application with the following command line parameters:
-Dcom.sun.management.jmxremote.port=8855
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
It is important to use the -Dcom.sun.management.jmxremote.ssl=false parameter if you don't want to setup digital certificates on the jmx host.
If you started your application on a machine having IP address 192.168.0.1, open jconsole, put 192.168.0.1:8855 in the Remote Process field, and click Connect.

Step 1: Run the application using following parameters.
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Above arguments bind the application to the port 9999.
Step 2: Launch jconsole by executing the command jconsole in command prompt or terminal.
Select ‘Remote Process:’ and enter the url as {IP_Address}:9999 and click on Connect button to connect to the remote application.
You can refer this link for complete application.

RUN LOCAL PROCESS JCONSOLE using Remote Process option
To run locally, this worked for me -
I added this in my vm args -
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=6001
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=localhost
-Dcom.sun.management.jmxremote.rmi.port=6001
I opened JConsole via Intellij Terminal
It was showing me all PID's in grey in local
So I selected remote process and logged in using host - localhost:6001
Keep empty username & password
Then click connect
Make sure no other process is running on port 6001. You can also use other ports.

First you need to check if your java process is already running with JMX parameters. Do this:
ps -ef | grep java
Check your java process you need to monitor. If you can see jmx rmi parameter Djmx.rmi.registry.port=xxxx then use the port mentioned here in your java visualvm to connect it remotely under jmx connection.
If it's not running through jmx rmi port then you need to run your java process with below mentioned parameters :
-Djmx.rmi.registry.port=1234 -Djmx.rmi.port=1235 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
Note: port numbers are based on your choice.
Now you can use this port for jmx coneection. Here it is port 1234.

I had this exact issue, and created a GitHub project for testing and figuring out the correct settings.
It contains a working Dockerfile with supporting scripts, and a simple docker-compose.yml for quick testing.

Unable to use JConsole with Tomcat running as windows service

I am running tomcat 6.0.18 as a windows service. In the service applet the jvm is configured default, i.e. it is using jvm.dll of the JRE.
I am trying to monitor this application with JConsole but cannot connect to it locally. I added the parameter -Dcom.sun.management.jmxremote (which works when starting tomcat with the start.bat script). But the jvm does not seem to pick up the parameter.

There's a nice GUI to edit the options, no need to muck around in the registry.
Open up the C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\tomcat6.exe (or just double-click on the monitor icon in the task bar). Go to the Java pane, add the following to the list of arguments, and restart Tomcat.
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8086
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
Then you can connect with JConsole or the newer VisualVM.

Here's the prescribed way for changing jvmoptions & interacting with the service:
http://tomcat.apache.org/tomcat-5.5-doc/windows-service-howto.html
I would try going into your registry at HKLM/Software/Apache Software Foundation/Procrun 2.0//Parameters/Java and editing the "Options" multi-string value directly.

If Tomcat is running as a Windows service, and you want to attach to the JVM locally, you need to run VisualVM or JConsole as the System account. You can use Sysinternals PsExec.exe to accomplish this.
psexec.exe -i -s c:\visualvm\bin\visualvm.exe

I'm posting it mainly to record this information to myself, I haven't validated it - but this is supposed to work as well:
http://mysqlandsqlserver.blogspot.com/2010/02/jconsolejmap-and-tomcat-as-windows.html

There is still a rather simple way to connect JConsole to Java process started as Windows Service using the local mode which I discovered here.
Basically it says that in order to connect to Java process launched as a Windows Service you need to launch JConsole as a Windows Service (you can do it using windows native api or using any wrapper like yajsw.)
By the way, this will free you from restarting the Java Process which was critical for me.

Add the following near the top of your catalina.bat
set JAVA_OPTS=%JAVA_OPTS% -Dcom.sun.management.jmxremote ^
-Dcom.sun.management.jmxremote.port=8086 ^
-Dcom.sun.management.jmxremote.ssl=false ^
-Dcom.sun.management.jmxremote.authenticate=false
Stop and restart tomcat (obviously)
Run jconsole.exe. If your tomcat is running as service, then run jconsole.exe as administrator.
Select Remote Process and enter localhost:8086