Develop Reference

JVisualVM remote working when destination is behind NAT?

I am in developer network and the JBoss to be monitored using JVisualVM is behind client firewall in separate network. JBoss is exposed to us through a NAT. The exposed JMX port works when using telnet from our developer network, but JVisualVM still doesn't find the remote JMX. Apparently this can be due to two things: 1) one or several dynamic RMI sockets need to be accessed, or 2) the "java.rmi.server.hostname" is not defined. My problem may be both points 1 and 2, but apparently since we have SSH access this can be tackled according to this site:
http://rafaelsteil.com/setting-up-jmx-for-jconsole-visualvm-on-ec2-plus-jetty-configuration/
“java.rmi.server.hostname” is the public hostname (like example.com) of your server, without HTTP or anything else. You cannot use an internal address, otherwise you won’t be able to remotely access the service.
However I have one question about the "java.rmi.server.hostname", should the defined IP address be the server internal IP or NAT IP which is publicly expoed to our developer network?

I have actually just tackled this problem myself and figured it out.
I would wager that the problem is the RMI connections - you can't predict which ports it will use and so you can't get it to work with a firewall.
The workaround is to use an SSH proxy:
SSH to the box where your application is running but use the -D option like this:
ssh user#remoteHost -D 9999
This will start a socks proxy on your local machine on port 9999.
Open JVisualVM and in the preferences, under 'network' configure it to use a socks proxy at localhost, on port 9999.
If you do the above, you should then be able to connect to the remote machine as normal and since all the RMI traffic is now going over the SSH proxy, it is punched through the firewall and works nicely.
Good luck :-)

JConsole over ssh local port forwarding

I'd like to be able to remotely connect to a Java service that has JMX exposed, however it is blocked by a firewall. I have tried to use ssh local port forwarding, however the connection fails. Looking at wireshark, it appears that when you try to connect with jconsole, it wants to connect via some ephemeral ports after connecting to port 9999, which are blocked by the firewall.
Is there any way to make jconsole only connect through 9999 or use a proxy? Is this article still the best solution? Or, am I missing something?

There's an even nicer way to do this using an SSH socks tunnel, since JConsole supports SOCKS:
Create the SSH socks proxy locally on some free port (e.g. 7777):
ssh -fN -D 7777 user#firewalled-host
Run JConsole by specifying the SOCKS proxy (e.g. localhost:7777) and the address for the JMX server (e.g. localhost:2147)
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 service:jmx:rmi:///jndi/rmi://localhost:2147/jmxrmi -J-DsocksNonProxyHosts=
As mentioned in one of the answers below, from JDK 8u60+ you also need to have the -J-DsocksNonProxyHosts= option in order to get it working.

With almost all current JDK versions (7u25 or later) it's now possible to use JConsole and Visual JVM over SSH quite easily (because now you can bind JMX to single port).
I use the following JVM parameters
-Dcom.sun.management.jmxremote.port=8090
-Dcom.sun.management.jmxremote.rmi.port=8090
-Djava.rmi.server.hostname=127.0.0.1
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Then I launch SSH connection
ssh my.javaserver.domain -L 8090:127.0.0.1:8090
After I can connect from JConsole
Remote Process: -> localhost:8090
And Java Visual VM
Right Click on Local -> Add JMX Connection -> localhost:8090

Is there any way to make jconsole only connect through 9999 or use a proxy? Is this article still the best solution? Or, am I missing something?
Yes, that article is about right.
When you specify the JMX port on your server (-Dcom.sun.management.jmxremote.port=####), you are actually specifying just the registry-port for the application. When you connect it provides an additional server-port that the jconsole actually does all of its work with. To get forwarded to work, you need to know both the registry and server ports.
Something like the following should work to run your application with both the registry and server ports set to 8000. See here for more details.
-Dcom.sun.management.jmxremote.port=8000
-Dcom.sun.management.jmxremote.rmi.port=8000
-Djava.rmi.server.hostname=127.0.0.1
As an aside, my SimpleJMX library allows you to set both ports easily and you can set them both to be the same port.
So, once you know both the port(s) you need to forward, you can set up your ssh command. For example, if you configure the registry and server ports as 8000, you would do:
ssh -L 8000:localhost:8000 remote-host
This creates a local port 8000 which forwards to localhost:8000 on the remote-host. You can specify multiple -L arguments if you need to forward multiple ports. Then you can connect your jconsole to localhost:8000 and it will connect to the remote-host appropriately.
Also, if your server has multiple interfaces, you may need to set the java.rmi.server.hostname variable to bind to the right interface.
-Djava.rmi.server.hostname=10.1.2.3

Continuing the SSH socks method, with newer java versions (around 8u66) you also need to set socksNonProxyHosts empty resulting in:
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 -J-DsocksNonProxyHosts=

About the Spring RMI running in Linux system

When I packed my RMI applications and moved to Linux system and ran it.
The log shows that RMI services are running on server 127.0.0.1(Which was printed by method of InetAddress.getLocalhost()).
The configuration in Host file is "127.0.0.1 localhost.localdomain localhost", so I think RMI server was defalutly got the Localhost as servering IP.
After that, my RMI client try to invoke the RMI server method with its real IP(172.16.7.155) which caused a exception "Refused to connect 127.0.0.1".
There're two ways to reslove this problem. The one is modify Host file and reflect the localhost to real IP(172.16.7.155), but I can not modify it because other applications are using localhost domain.
Another way is to reset the method of fecthing IP address at RMI Server, i.e. instead the InetAddress.getLocalhost(), is there any configuration for this method?

You can probably fix this by starting your java rmi server process with the system property "-Djava.rmi.server.hostname=172.16.7.155" (or whatever the public ip of the box is). (details here)

Apache ActiveMQ: What is TCP Port 64119 Used For?

I am running Apache ActiveMQ 5.5 on CentOS 5.5.
I have searched the ActiveMQ website, source code, configuration files, and I cannot find any reference to this port. Yet, when I start the broker, this is one of three TCP ports listened to by the JVM.
Does anyone know what port 64119 is used for by Apache ActiveMQ?

I have tried with ActiveMQ 5.4.2 on Ubuntu. It listens on:
61616 - broker protocol
8161 - web admin
39271 - JMX remote port
Server is not listening on any other ports.
EDIT: I checked with 5.5.0 and also restarted the broker (both 5.4.2 and 5.5.0) several times. JMX port is always different, looks like it's being randomized. I bet your 64119 changes over time as well.
To make it clear, connect to ActiveMQ broker with jvisualvm, you should see an open connection from jvisualvm to ActiveMQ process on this port (I can see it with lsof).

JMX : cant connect from Windows box to monitor JMX port on a Linux system

I have a lot of experience with jconsole.exe and JVisualVM.exe , in the JDK1.6 and have connected thousands of times from a Windows JVM over to a Windows JVM on another machine via JMX.remote but it fails when I try to monitor a java instance that is running on a Linux host (from a Windows host and I tried JMX listener on redhat and SUSE at Amazon EC2). I also tried using jconsole.exe and get a similar error.
Is there any reason anyone can think of , why this kind of JMX connection would have a problem. Any ideas I can try? Has anyone "actually" done this and can say it will work if I persevere?
The error I get from JVisualVM (on the remote connect attempt) is something like this:
"Cannot connect using service:jmx:rmi:///jndi/rmi://<jmx service ip>:8001/jmxrmi"
My remote JMX service config is like this:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8001
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.hostname=<jmx service ip>
Found a similar question at this link but it didn't answer my question.
I verified that "iptables" is not enabled as a service and is not on and so I don't imaging there is any sort of firewall blockage. Also, both the windows and linux machine are on a 10.0.0.0 private internal subnet together. I am able to telnet into the port 8001 to verify its there , and its listening (netstat -ap) on the linux machine.

See the system property java.rmi.server.hostname. On your Linux server JVM, set this system property to be the public IP of the host. Then use the public IP in you client JMX URL.

I think your problem is that the RMI implementation is kind of hard to work with through firewalls and such as it requires more than the port you specify to work. Things you typically wouldn't notice if you connect to your own machine or a machine on the same network.
This post describes the problems you will run into quite well
If I were you, I would try to setup jmxmp as an alternative protocol. To do so you need to add the jmxremote_optional.jar (free from Oracle, download "JMX Remote API 1.0.1_04 Reference Implementation" from here) to both the server and the jvisualvm classpath but it is worth it.
If you google for jmxmp you will find quite a few examples on how to set things up, one of my first hits are http://pub.admc.com/howtos/jmx/distributed-chapt.html#jmxmp-sect which may be a bit too code oriented but I add it here anyway because it explains the things like like most with jmxmp in a few good sentences.
How to define the server side endpoint is depending on what you are running. Most app servers will let you type in a jmxmp enabled jmx service url but if the server is written from scratch you might have to set it up yourself in code instead of using the -D switches to java you are used to.
Give it a try and return with more specific questions about it if you run into problems.
Edit:
After you have added the jar to the classpath, the only thing you have to do in your code (assuming you are not using a server app that already handles it for you) is the following (omitting declarations, exception handling and such as you will figure it out anyway):
url=new JMXServiceURL(jmxurl);
this.server = JMXConnectorServerFactory.newJMXConnectorServer(url, null, ManagementFactory.getPlatformMBeanServer());
this.server.start();

Have you tried creating an SSH tunnel to the Linux box from your Windows machine? http://oldsite.precedence.co.uk/nc/putty.html
Or if you have cygwin, just try ssh -f your-user#remote-server.com -L 7777:remote-server.com:123 -N where 7777 is the port on YOUR windows machine, and 123 is the port on the remote Linux box that listens to JMX commands.
With either of the above, you can use jconsole or visualvm on your Windows box and connect to localhost:7777.
I know iptables is disabled, but just confirm that JMX is working fine at that port by SSHing into the Linux box and trying to use commandline JMX mode of jconsole for localhost (on the Linux box).

I have actually just tackled this problem myself and figured it out.
I would wager that the problem is the RMI connections - you can't predict which ports it will use and so you can't get it to work with a firewall.
The workaround is to use an SSH proxy:
SSH to the box where your application is running but use the -D option like this:
ssh user#remoteHost -D 9999
As you are on Windows, you can use Cygwin to run the command above, or you can do the same thing with Putty through the GUI (guide here: http://blog.ashurex.com/2012/03/15/creating-ssh-proxy-tunnel-putty/)
This will start a socks proxy on your local machine on port 9999.
Open JVisualVM and in the preferences, under 'network' configure it to use a socks proxy at localhost, on port 9999.
If you do the above, you should then be able to connect to the remote machine as normal and since all the RMI traffic is now going over the SSH proxy, it is punched through the firewall and works nicely.
Good luck :-)