I'd like to be able to remotely connect to a Java service that has JMX exposed, however it is blocked by a firewall. I have tried to use ssh local port forwarding, however the connection fails. Looking at wireshark, it appears that when you try to connect with jconsole, it wants to connect via some ephemeral ports after connecting to port 9999, which are blocked by the firewall.
Is there any way to make jconsole only connect through 9999 or use a proxy? Is this article still the best solution? Or, am I missing something?
There's an even nicer way to do this using an SSH socks tunnel, since JConsole supports SOCKS:
Create the SSH socks proxy locally on some free port (e.g. 7777):
ssh -fN -D 7777 user#firewalled-host
Run JConsole by specifying the SOCKS proxy (e.g. localhost:7777) and the address for the JMX server (e.g. localhost:2147)
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 service:jmx:rmi:///jndi/rmi://localhost:2147/jmxrmi -J-DsocksNonProxyHosts=
As mentioned in one of the answers below, from JDK 8u60+ you also need to have the -J-DsocksNonProxyHosts= option in order to get it working.
With almost all current JDK versions (7u25 or later) it's now possible to use JConsole and Visual JVM over SSH quite easily (because now you can bind JMX to single port).
I use the following JVM parameters
-Dcom.sun.management.jmxremote.port=8090
-Dcom.sun.management.jmxremote.rmi.port=8090
-Djava.rmi.server.hostname=127.0.0.1
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Then I launch SSH connection
ssh my.javaserver.domain -L 8090:127.0.0.1:8090
After I can connect from JConsole
Remote Process: -> localhost:8090
And Java Visual VM
Right Click on Local -> Add JMX Connection -> localhost:8090
Is there any way to make jconsole only connect through 9999 or use a proxy? Is this article still the best solution? Or, am I missing something?
Yes, that article is about right.
When you specify the JMX port on your server (-Dcom.sun.management.jmxremote.port=####), you are actually specifying just the registry-port for the application. When you connect it provides an additional server-port that the jconsole actually does all of its work with. To get forwarded to work, you need to know both the registry and server ports.
Something like the following should work to run your application with both the registry and server ports set to 8000. See here for more details.
-Dcom.sun.management.jmxremote.port=8000
-Dcom.sun.management.jmxremote.rmi.port=8000
-Djava.rmi.server.hostname=127.0.0.1
As an aside, my SimpleJMX library allows you to set both ports easily and you can set them both to be the same port.
So, once you know both the port(s) you need to forward, you can set up your ssh command. For example, if you configure the registry and server ports as 8000, you would do:
ssh -L 8000:localhost:8000 remote-host
This creates a local port 8000 which forwards to localhost:8000 on the remote-host. You can specify multiple -L arguments if you need to forward multiple ports. Then you can connect your jconsole to localhost:8000 and it will connect to the remote-host appropriately.
Also, if your server has multiple interfaces, you may need to set the java.rmi.server.hostname variable to bind to the right interface.
-Djava.rmi.server.hostname=10.1.2.3
Continuing the SSH socks method, with newer java versions (around 8u66) you also need to set socksNonProxyHosts empty resulting in:
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 -J-DsocksNonProxyHosts=
Related
I am trying to connect to a JMX port that I have defined to be 1099 (as is the default) on my Java application (Java 8 and with Spring Boot 1.4 and Tomcat) using a client such as JConsole, Java Mission Control, or Java VisualVM, but I am reaching an error:
java.rmi.ConnectException: Connection refused to host: 10.xxx.xxx.xx, nested exception is:
java.net.ConnectException: Connection timed out
...
Note that I hid the exact host IP, but it is the pod IP of the particular Kubernetes-managed Docker container that my service is deployed in. I try to connect to the JMX port using the following service URL:
jconsole service:jmx:rmi://<nodeIP>:<nodePort>/jndi/rmi://<nodeIP>:<nodePort>/jmxrmi
I know that JMX opens a random high port, and I have tried to resolve that by including a custom #Configuration class that forces that high port to also serve on port 1099. I have gone into the actual container and pod and ran
netstat -tulpn
to see the ports that are opened, and I have confirmed that the only ports opened are 8443 (which my application is running on) and 1099 (the JMX port); this indicates that my class works. I also ensured that port 1099 is open on the Kubernetes side, so that is not what is blocking it.
As many answers surrounding a JMX remote connection have suggested, I have tried many variations of the following Java options to no avail:
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.jmi.port=1099 -Djava.rmi.server.hostname=0.0.0.0
The answers have suggested forcing the JMX port and the RMI registry port to be the same, but none of these have worked.
I believe the issue may be because the hostname (as I tried to make dynamic with -Djava.rmi.server.hostname=0.0.0.0) can not resolve to the different hostnames (pod IPs) that are created every time my service is deployed.
Thus, it looks like the connection cannot complete because JMX is unable to see what hostname Kubernetes is assigning after my service is deployed.
Is there a way for JMX to recognize the Kubernetes hostname? Or, is there some other way to connect remotely to a JMX port through a Kubernetes-deployed service?
EDIT 1: I have done some additional research, and maybe an optional JMXMP instead of RMI may work? Has anyone gotten this working with Tomcat?
The, jmx remote connection is a pain to work with, proxying it is impossible from my view. I had similar problems and in the end I just used jolokia to connect.
Jolokia is a JMX-HTTP bridge giving an alternative to JSR-160 connectors. It is an agent based approach with support for many platforms. In addition to basic JMX operations it enhances JMX remoting with unique features like bulk requests and fine grained security policies. -> http://jolokia.org
If you want to start jstatd on a specific port for its RMI registry, you can pass -p parameter for this.
But the problem is that it opens a second random port (called "anonymous port" in java) which causes problem for writing firewall rules or to use JVisualVM to connect to a remote jstatd running in a Docker container.
If you look at jstatd source, you'll see that it is calling UnicastRemoteObject.exportObject(remoteHost, 0) which will open a new "anonymous port" which seems to be random.
Is there a way to force this last port to a fixed one, or a way to predict which one will be chosen?
I found no easy way to predict which concrete port will be opened by using an anonymous port.
But I found a rewrite of jstatd called "jakestatd" which will force the 3 ports (because at last, I discovered that jstatd actually opens 3 ports and not 2 as I first thought) that jstatd uses.
As it was not enough for me because I needed to control those ports, I wrote ejstatd that answer this exact question (as well as others), so now I can control thos ports using (inside ejstatd's folder):
mvn exec:java -Dexec.args="-pr 2222 -ph 2223 -pv 2224"
Here the 3 ports that will be opened will be 2222, 2223 and 2224, and the RMI registry will be available at port 2222.
i have passed following jmx parameters in a java program and deployed it on some remote machine.
-Dcom.sun.management.jmxremote.port=5001
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Still when I am trying to connect to the host using host:port in jconsole, it redirects me to InsecureConnection and then it doesn't connect.
add
-Dcom.sun.management.jmxremote.host=<hostname>
-Dcom.sun.management.jmxremote.rmi.port=<pornNum>
When you only specify -Dcom.sun.management.jmxremote.port it binds rmi registry to that port. This registry contains remote object, and you should add -Dcom.sun.management.jmxremote.rmi.port to specify where to bind that remote object.
Otherwise random port will be used and it will be not good if you have firewall.
You can see sun.management.jmxremote.ConnectorBootstrap#startRemoteConnectorServer for more details
I am in developer network and the JBoss to be monitored using JVisualVM is behind client firewall in separate network. JBoss is exposed to us through a NAT. The exposed JMX port works when using telnet from our developer network, but JVisualVM still doesn't find the remote JMX. Apparently this can be due to two things: 1) one or several dynamic RMI sockets need to be accessed, or 2) the "java.rmi.server.hostname" is not defined. My problem may be both points 1 and 2, but apparently since we have SSH access this can be tackled according to this site:
http://rafaelsteil.com/setting-up-jmx-for-jconsole-visualvm-on-ec2-plus-jetty-configuration/
“java.rmi.server.hostname” is the public hostname (like example.com) of your server, without HTTP or anything else. You cannot use an internal address, otherwise you won’t be able to remotely access the service.
However I have one question about the "java.rmi.server.hostname", should the defined IP address be the server internal IP or NAT IP which is publicly expoed to our developer network?
I have actually just tackled this problem myself and figured it out.
I would wager that the problem is the RMI connections - you can't predict which ports it will use and so you can't get it to work with a firewall.
The workaround is to use an SSH proxy:
SSH to the box where your application is running but use the -D option like this:
ssh user#remoteHost -D 9999
This will start a socks proxy on your local machine on port 9999.
Open JVisualVM and in the preferences, under 'network' configure it to use a socks proxy at localhost, on port 9999.
If you do the above, you should then be able to connect to the remote machine as normal and since all the RMI traffic is now going over the SSH proxy, it is punched through the firewall and works nicely.
Good luck :-)
I have a lot of experience with jconsole.exe and JVisualVM.exe , in the JDK1.6 and have connected thousands of times from a Windows JVM over to a Windows JVM on another machine via JMX.remote but it fails when I try to monitor a java instance that is running on a Linux host (from a Windows host and I tried JMX listener on redhat and SUSE at Amazon EC2). I also tried using jconsole.exe and get a similar error.
Is there any reason anyone can think of , why this kind of JMX connection would have a problem. Any ideas I can try? Has anyone "actually" done this and can say it will work if I persevere?
The error I get from JVisualVM (on the remote connect attempt) is something like this:
"Cannot connect using service:jmx:rmi:///jndi/rmi://<jmx service ip>:8001/jmxrmi"
My remote JMX service config is like this:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8001
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.hostname=<jmx service ip>
Found a similar question at this link but it didn't answer my question.
I verified that "iptables" is not enabled as a service and is not on and so I don't imaging there is any sort of firewall blockage. Also, both the windows and linux machine are on a 10.0.0.0 private internal subnet together. I am able to telnet into the port 8001 to verify its there , and its listening (netstat -ap) on the linux machine.
See the system property java.rmi.server.hostname. On your Linux server JVM, set this system property to be the public IP of the host. Then use the public IP in you client JMX URL.
I think your problem is that the RMI implementation is kind of hard to work with through firewalls and such as it requires more than the port you specify to work. Things you typically wouldn't notice if you connect to your own machine or a machine on the same network.
This post describes the problems you will run into quite well
If I were you, I would try to setup jmxmp as an alternative protocol. To do so you need to add the jmxremote_optional.jar (free from Oracle, download "JMX Remote API 1.0.1_04 Reference Implementation" from here) to both the server and the jvisualvm classpath but it is worth it.
If you google for jmxmp you will find quite a few examples on how to set things up, one of my first hits are http://pub.admc.com/howtos/jmx/distributed-chapt.html#jmxmp-sect which may be a bit too code oriented but I add it here anyway because it explains the things like like most with jmxmp in a few good sentences.
How to define the server side endpoint is depending on what you are running. Most app servers will let you type in a jmxmp enabled jmx service url but if the server is written from scratch you might have to set it up yourself in code instead of using the -D switches to java you are used to.
Give it a try and return with more specific questions about it if you run into problems.
Edit:
After you have added the jar to the classpath, the only thing you have to do in your code (assuming you are not using a server app that already handles it for you) is the following (omitting declarations, exception handling and such as you will figure it out anyway):
url=new JMXServiceURL(jmxurl);
this.server = JMXConnectorServerFactory.newJMXConnectorServer(url, null, ManagementFactory.getPlatformMBeanServer());
this.server.start();
Have you tried creating an SSH tunnel to the Linux box from your Windows machine? http://oldsite.precedence.co.uk/nc/putty.html
Or if you have cygwin, just try ssh -f your-user#remote-server.com -L 7777:remote-server.com:123 -N where 7777 is the port on YOUR windows machine, and 123 is the port on the remote Linux box that listens to JMX commands.
With either of the above, you can use jconsole or visualvm on your Windows box and connect to localhost:7777.
I know iptables is disabled, but just confirm that JMX is working fine at that port by SSHing into the Linux box and trying to use commandline JMX mode of jconsole for localhost (on the Linux box).
I have actually just tackled this problem myself and figured it out.
I would wager that the problem is the RMI connections - you can't predict which ports it will use and so you can't get it to work with a firewall.
The workaround is to use an SSH proxy:
SSH to the box where your application is running but use the -D option like this:
ssh user#remoteHost -D 9999
As you are on Windows, you can use Cygwin to run the command above, or you can do the same thing with Putty through the GUI (guide here: http://blog.ashurex.com/2012/03/15/creating-ssh-proxy-tunnel-putty/)
This will start a socks proxy on your local machine on port 9999.
Open JVisualVM and in the preferences, under 'network' configure it to use a socks proxy at localhost, on port 9999.
If you do the above, you should then be able to connect to the remote machine as normal and since all the RMI traffic is now going over the SSH proxy, it is punched through the firewall and works nicely.
Good luck :-)