Develop Reference

Debugging VisualVM remote connection

I am running Tomcat 6 on a Linux server in Amazon's cloud. I am trying to connect to it with VisualVM from my Mac at my office. I have allowed opened up all TCP ports between my desktop and the server, but I am unable to get VisualVM to do anything.
On the linux box, I have started Tomcat with:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9191
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
On my Mac, I launch VisualVM and choose File -> New JMX Connection...
I enter amazonhostname.com:9191 in the box. It says "Adding amazonhostname.com:9191..." and sits there for several minutes before timing out.
From my Mac, I can "telnet amazonhostname.com 9191" just fine, and I tried some other random ports, and they all worked. I'm pretty convinced it's not a firewall issue, unless I need to open UDP or ICMP or something like that. (Really??)
There is nothing in catalina.log on the server, or the system console on my Mac, related to jmx. netstat on the server shows 9191 and some other ports open on all interfaces.
There is no jstatd on the remote server. Is that a pre-requisite for JMX? (I am totally new to Java and I'm not even sure what those things mean, but I saw it on a blog post somewhere. I'm grasping at straws now.)

Set the hostname property before the VM starts:
java -Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9191 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=the.public.ip \
-jar program.jar
Add the relevant rules to your security group.

What worked well for me in RHEL7 environment is to implement the JmxRemoteLifecycleListener on Tomcat using the example provided in the documentation. I have also opened up ports 10001 and 10002 on the firewall.
JMX Remote Lifecycle Listener

Can't stop tomcat normally when I configure jmxremote

I added a jmxremote configuraiton in the catalina.bat:
set JAVA_OPTS=-Dcom.sun.management.jmxremote.port=9004 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
so that I could start jconsole to monitor the tomcat's performance.
But I got a problem that I couldn't stop tomcat normally through $CATALINA_HOME\catalina.bat stop, neither did $CATALINA_HOME\shutdown.bat
Any suggestions?

In order to monitor the java process, you need to add the following system properties to the command line:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
In the visualvm you just use the connection string host:9999.
However, sometimes the RMI listener listens to the wrong IP address, one which is inaccessible to the visualvm. Thanks to Pavel’s tip, I found a way to overcome this is by adding the following parameters:
-Djava.rmi.server.hostname=$(hostname)
-Djava.rmi.server.useLocalHostname=true
Now it works like a charm!
For completeness, I’d mention you can secure the connection to the JVM, either by requiring user/password or by using SSL. If you are interested, please see this guide.
Make sure that you put the definitions in a place only the start command sees, but not the shutdown. The reason or this is that the jmx remote create a listening socket, making the shutdown to listen to the same port if not configured properly.

Are you sure Tomcat started successfully?
Usually you need to add this property before the other jmx properties:
-Dcom.sun.management.jmxremote
http://tomcat.apache.org/tomcat-6.0-doc/monitoring.html#Enabling_JMX_Remote

Better to add these parameters to the CATALINA_OPTS in the file setenv.sh or setenv.bat
# JMX OPTIONS
CATALINA_OPT="$CATALINA_OPT "-Dcom.sun.management.jmxremote
CATALINA_OPT="$CATALINA_OPT "-Dcom.sun.management.jmxremote.port=9012
CATALINA_OPT="$CATALINA_OPT "-Dcom.sun.management.jmxremote.local.only=false
CATALINA_OPT="$CATALINA_OPT "-Dcom.sun.management.jmxremote.authenticate=false
CATALINA_OPT="$CATALINA_OPT "-Dcom.sun.management.jmxremote.ssl=false

Still cannot connect Tomcat 6 remotely via VisualVM

If checked the other questions regarding this issue but so far no soultion.
Here is my setup:
TC6 runs on a debian vm on my macbook. in /etc/default/tomcat6.0 I have following options set:
CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8086 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
netstat -ntlp gives me:
tcp6 0 0 :::8080 :::* LISTEN 6550/java
tcp6 0 0 :::8086 :::* LISTEN 6550/java
Where 6550:
root 6550 3.3 6.8 224888 52984 pts/0 Sl 10:31 0:43 /usr/lib/jvm/java-6-sun/bin/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/lib/tomcat6.0/conf/logging.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8086 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.endorsed.dirs=/usr/share/tomcat6.0/common/endorsed -classpath :/usr/lib/jvm/java-6-sun/jre//lib/jcert.jar:/usr/lib/jvm/java-6-sun/jre//lib/jnet.jar:/usr/lib/jvm/java-6-sun/jre//lib/jsse.jar:/usr/share/tomcat6.0/bin/bootstrap.jar:/usr/share/tomcat6.0/bin/commons-logging-api.jar -Dcatalina.base=/var/lib/tomcat6.0 -Dcatalina.home=/usr/share/tomcat6.0 -Djava.io.tmpdir=/var/lib/tomcat6.0/temp org.apache.catalina.startup.Bootstrap start
Firewall:
Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT all anywhere anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
Chain OUTPUT (policy DROP)
target prot opt source destination
ACCEPT all anywhere anywhere
When I try to connect from OSX using jvisualm:
- adding a new host with the hostname of my vm (added it to /etc/hosts)
- adding a new JMX-Connection
VisualVM fails with:
Cannot connect using service:jmx:rmi///jndi/rmi://<vmhostname>:8086/jmxrmi
When I connect to my VM using ssh -X and start jvisualvm on the same machine that tomcat is running on, I do not initially see TC, but I can add the jmx connection as above.
Could this be an issue with foreign hosts being misteriously prevented to access the rmiregistry?
Why would anyone do such a dumb thing?
What am I doing wrong?
(sorry 4 the lengthy post)

Here is a tutorial on how to connect Visual VM to Tomcat 7. It is a bit on the verbose side, but should help steer you around a number of the problems most people run into.

Check out the jmx listener on this page.
http://tomcat.apache.org/tomcat-6.0-doc/config/listeners.html
It lets you set both the JMX port and the RMI registry port, both of which have to open on your firewall.

Try adding to catalina_opts: -Djava.rmi.server.hostname=vmhostname
and ensure that your mac is able to resolve this hostname, e.g.
nc -v vmhostname 8086
should succeed in opening the connection.

You may try to add actual ip and hostname to /etc/hosts where jstatd is running.

Follow this steps,
Download jvisualvm setup from Download Jvisualvm
Unzip this setup and go to bin directory from command line
Use this command to start the visualvm:
\bin>visualvm.exe -J-Dnetbeans.system_socks_proxy=localhost:9998 -J-Djava.net.useSystemProxies=true
get the putty setup ,go to installation directory of putty use following command to ssh tunnel to remote system
putty -D 9998 -load Name_of_the_saved_putty_session
saved_putty_session should have tunnel to localhost:JMX_port and some_source_port
now in visualvm UI in Left nav ,right click the remote and "Add remote host" , with public ip of remote host
Now on this added remote host right click and say "ADD JMX CONNECTION"
In a connection box say localhost:source_port_provided_in_putty_tunnel
Click on the ok button and you are done.

So the above worked for me:
service:jmx:rmi:///jndi/rmi://machinedomainname:1099/jmxrmi
After spending 1/2 day on this. Following all the usual instructions and getting frustrated the above cracked it.
I have a play application deployed in amazon ec2 with the following flags:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=<myport>
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Djava.rmi.server.hostname=<mymachinedomainname>
I have jvisualvm running locally on my mac.
I added a jvisualvm remote host with host name:
<mymachinedomainname>
Then added added a jmx connection to this host.
I tried the recommended
service:jmx:rmi://<mymachinedomainname>:<myport>
but it would just not connect.
But as soon as i did
service:jmx:rmi:///jndi/rmi://<mymachinedomainname>:<myport>/jmxrmi
instead. it connected and works.
No idea why! but it works.

Make sure that you can
a) ping <vmhostname>
b) telnet <vmhostname> 8086
Note also that you can use IP address instead of <vmhostname>

Make sure the jstatd process is running on the remote machine which you want to monitor, with required permissions.

I would suggest using this option:
-Dcom.sun.management.jmxremote.port=1099
Then, connect with
service:jmx:rmi:///jndi/rmi://machinedomainname:1099/jmxrmi
As long as RMI is listening on all interfaces ( ie. 0.0.0.0) then there should be no problem connecting from a remote server as long as the firewall is off (or carefully configured).

I have actually just tackled this problem myself and figured it out.
I would wager that the problem is the RMI connections - you can't predict which ports it will use and so you can't get it to work with a firewall.
The workaround is to use an SSH proxy:
SSH to the box where your application is running but use the -D option like this:
ssh user#remoteHost -D 9999
This will start a socks proxy on your local machine on port 9999.
Open JVisualVM and in the preferences, under 'network' configure it to use a socks proxy at localhost, on port 9999.
If you do the above, you should then be able to connect to the remote machine as normal and since all the RMI traffic is now going over the SSH proxy, it is punched through the firewall and works nicely.
Good luck :-)

How to activate JMX on my JVM for access with jconsole?

How to activate JMX on a JVM for access with jconsole?

The relevant documentation can be found here:
http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
Start your program with following parameters:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.rmi.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
For instance like this:
java -Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9010 \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-jar Notepad.jar
-Dcom.sun.management.jmxremote.local.only=false is not necessarily required
but without it, it doesn't work on Ubuntu. The error would be something like
this:
01 Oct 2008 2:16:22 PM sun.rmi.transport. customer .TCPTransport$AcceptLoop executeAcceptLoop
WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=37278] throws
java.io.IOException: The server sockets created using the LocalRMIServerSocketFactory only accept connections from clients running on the host where the RMI remote objects have been exported.
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:89)
at sun.rmi.transport. customer .TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport. customer .TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:636)
see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6754672
Also be careful with -Dcom.sun.management.jmxremote.authenticate=false which
makes access available for anyone, but if you only use it to track the JVM on
your local machine it doesn't matter.
Update:
In some cases I was not able to reach the server. This was then fixed if I set this parameter as well: -Djava.rmi.server.hostname=127.0.0.1

Running in a Docker container introduced a whole slew of additional problems for connecting so hopefully this helps someone. I ended up needed to add the following options which I'll explain below:
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=${DOCKER_HOST_IP}
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
DOCKER_HOST_IP
Unlike using jconsole locally, you have to advertise a different IP than you'll probably see from within the container. You'll need to replace ${DOCKER_HOST_IP} with the externally resolvable IP (DNS Name) of your Docker host.
JMX Remote & RMI Ports
It looks like JMX also requires access to a remote management interface (jstat) that uses a different port to transfer some data when arbitrating the connection. I didn't see anywhere immediately obvious in jconsole to set this value. In the linked article the process was:
Try and connect from jconsole with logging enabled
Fail
Figure out which port jconsole attempted to use
Use iptables/firewall rules as necessary to allow that port to connect
While that works, it's certainly not an automatable solution. I opted for an upgrade from jconsole to VisualVM since it let's you to explicitly specify the port on which jstatd is running. In VisualVM, add a New Remote Host and update it with values that correlate to the ones specified above:
Then right-click the new Remote Host Connection and Add JMX Connection...
Don't forget to check the checkbox for Do not require SSL connection. Hopefully, that should allow you to connect.

Note, Java 6 in the latest incarnation allows for jconsole to attach itself to a running process even after it has been started without JMX incantations.
If that is available to you, also consider jvisualvm as it provides a wealth of information on running processes, including a profiler.

I'm using WAS ND 7.0
My JVM need all the following arguments to be monitored in JConsole
-Djavax.management.builder.initial=
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8855
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

On Linux, I used the following params:
-Djavax.management.builder.initial=
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
and also I edited /etc/hosts so that the hostname resolves to the host address (192.168.0.x) rather than the loopback address (127.0.0.1)

along with below command line parameters ,
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Sometimes in the linux servers , imx connection doesn't get succeeded. that is because , in cloud linux host, in /etc/hosts so that the hostname resolves to the host address.
the best way to fix it is, ping the particular linux server from other machine in network and use that host IP address in the
-Djava.rmi.server.hostname=IP address that obtained when you ping that linux server.
But never rely on the ipaddress that you get from linux server using ifconfig.me. the ip that you get there is masked one which is present in the host file.

The below options works for me:
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.rmi.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname={host name}
and remember to open 9010 port in the server
sudo ufw allow 9010/udp
sudo ufw allow 9010/tcp
sudo ufw reload

Run your java application with the following command line parameters:
-Dcom.sun.management.jmxremote.port=8855
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
It is important to use the -Dcom.sun.management.jmxremote.ssl=false parameter if you don't want to setup digital certificates on the jmx host.
If you started your application on a machine having IP address 192.168.0.1, open jconsole, put 192.168.0.1:8855 in the Remote Process field, and click Connect.

Step 1: Run the application using following parameters.
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Above arguments bind the application to the port 9999.
Step 2: Launch jconsole by executing the command jconsole in command prompt or terminal.
Select ‘Remote Process:’ and enter the url as {IP_Address}:9999 and click on Connect button to connect to the remote application.
You can refer this link for complete application.

RUN LOCAL PROCESS JCONSOLE using Remote Process option
To run locally, this worked for me -
I added this in my vm args -
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=6001
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=localhost
-Dcom.sun.management.jmxremote.rmi.port=6001
I opened JConsole via Intellij Terminal
It was showing me all PID's in grey in local
So I selected remote process and logged in using host - localhost:6001
Keep empty username & password
Then click connect
Make sure no other process is running on port 6001. You can also use other ports.

First you need to check if your java process is already running with JMX parameters. Do this:
ps -ef | grep java
Check your java process you need to monitor. If you can see jmx rmi parameter Djmx.rmi.registry.port=xxxx then use the port mentioned here in your java visualvm to connect it remotely under jmx connection.
If it's not running through jmx rmi port then you need to run your java process with below mentioned parameters :
-Djmx.rmi.registry.port=1234 -Djmx.rmi.port=1235 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
Note: port numbers are based on your choice.
Now you can use this port for jmx coneection. Here it is port 1234.

I had this exact issue, and created a GitHub project for testing and figuring out the correct settings.
It contains a working Dockerfile with supporting scripts, and a simple docker-compose.yml for quick testing.

Unable to use JConsole with Tomcat running as windows service

I am running tomcat 6.0.18 as a windows service. In the service applet the jvm is configured default, i.e. it is using jvm.dll of the JRE.
I am trying to monitor this application with JConsole but cannot connect to it locally. I added the parameter -Dcom.sun.management.jmxremote (which works when starting tomcat with the start.bat script). But the jvm does not seem to pick up the parameter.

There's a nice GUI to edit the options, no need to muck around in the registry.
Open up the C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\tomcat6.exe (or just double-click on the monitor icon in the task bar). Go to the Java pane, add the following to the list of arguments, and restart Tomcat.
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8086
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
Then you can connect with JConsole or the newer VisualVM.

Here's the prescribed way for changing jvmoptions & interacting with the service:
http://tomcat.apache.org/tomcat-5.5-doc/windows-service-howto.html
I would try going into your registry at HKLM/Software/Apache Software Foundation/Procrun 2.0//Parameters/Java and editing the "Options" multi-string value directly.

If Tomcat is running as a Windows service, and you want to attach to the JVM locally, you need to run VisualVM or JConsole as the System account. You can use Sysinternals PsExec.exe to accomplish this.
psexec.exe -i -s c:\visualvm\bin\visualvm.exe

I'm posting it mainly to record this information to myself, I haven't validated it - but this is supposed to work as well:
http://mysqlandsqlserver.blogspot.com/2010/02/jconsolejmap-and-tomcat-as-windows.html

There is still a rather simple way to connect JConsole to Java process started as Windows Service using the local mode which I discovered here.
Basically it says that in order to connect to Java process launched as a Windows Service you need to launch JConsole as a Windows Service (you can do it using windows native api or using any wrapper like yajsw.)
By the way, this will free you from restarting the Java Process which was critical for me.

Add the following near the top of your catalina.bat
set JAVA_OPTS=%JAVA_OPTS% -Dcom.sun.management.jmxremote ^
-Dcom.sun.management.jmxremote.port=8086 ^
-Dcom.sun.management.jmxremote.ssl=false ^
-Dcom.sun.management.jmxremote.authenticate=false
Stop and restart tomcat (obviously)
Run jconsole.exe. If your tomcat is running as service, then run jconsole.exe as administrator.
Select Remote Process and enter localhost:8086