Develop Reference

How to use JProfiler over two-hop SSH tunnel

I'm trying to connect JProfiler to a JVM running on a server that I'll call remote. This server is only accessible from my workstation (local) via another server that I'll call middle. My plan for connecting JProfiler to remote was this:
Install the JProfiler instrumentation on remote
Establish SSH tunnel from local, through middle, to remote:
ssh -v -N -L 8849:[remote's private address (192.168... etc)]:8849 [middle]
Establish a new JProfiler session on localhost:8849, choosing "Startup immediately, connect later with JProfiler GUI"
However, I end up getting an error:
Connection error
Either an old version of the native library is used or another
application is listening on port 8849. Please check your
DYLD_LIBRARY_PATH environment variable and your port configuration
I don't have any other programs bound to local port 8849, aside from my SSH tunnel, and I have confirmed that the SSH tunnel itself should be working correctly - I'm able to forward connections for a test HTTP server from remote to local via a similarly configured tunnel.
I found this similar question, but no solution was provided.
What am I missing from my configuration?

A direct tunnel is established with
ssh -t user#remote -L [localPort]:localhost:[remotePort] -N
A 2-hop tunnel is built with chained ssh commands:
ssh -t user#middle -L [localPort]:localhost:[remotePort] \
ssh -t user#remote -L [remotePort]:localhost:[remotePort] -N
where localPort is the port you want to use locally, and remotePort is the port that the profiling agent is listening on. All of this is executed in a single command on your local machine. More hops can be added with additional ssh calls like in the first line. There must by exactly one trailing -N for the entire command.
This approach works with JProfiler.
If the tunnels fails or if the profiling agent is not listening, you will get the message that you mentioned because of the way that the connection fails in the case of an SSH tunnel.

JConsole over ssh local port forwarding

I'd like to be able to remotely connect to a Java service that has JMX exposed, however it is blocked by a firewall. I have tried to use ssh local port forwarding, however the connection fails. Looking at wireshark, it appears that when you try to connect with jconsole, it wants to connect via some ephemeral ports after connecting to port 9999, which are blocked by the firewall.
Is there any way to make jconsole only connect through 9999 or use a proxy? Is this article still the best solution? Or, am I missing something?

There's an even nicer way to do this using an SSH socks tunnel, since JConsole supports SOCKS:
Create the SSH socks proxy locally on some free port (e.g. 7777):
ssh -fN -D 7777 user#firewalled-host
Run JConsole by specifying the SOCKS proxy (e.g. localhost:7777) and the address for the JMX server (e.g. localhost:2147)
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 service:jmx:rmi:///jndi/rmi://localhost:2147/jmxrmi -J-DsocksNonProxyHosts=
As mentioned in one of the answers below, from JDK 8u60+ you also need to have the -J-DsocksNonProxyHosts= option in order to get it working.

With almost all current JDK versions (7u25 or later) it's now possible to use JConsole and Visual JVM over SSH quite easily (because now you can bind JMX to single port).
I use the following JVM parameters
-Dcom.sun.management.jmxremote.port=8090
-Dcom.sun.management.jmxremote.rmi.port=8090
-Djava.rmi.server.hostname=127.0.0.1
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Then I launch SSH connection
ssh my.javaserver.domain -L 8090:127.0.0.1:8090
After I can connect from JConsole
Remote Process: -> localhost:8090
And Java Visual VM
Right Click on Local -> Add JMX Connection -> localhost:8090

Is there any way to make jconsole only connect through 9999 or use a proxy? Is this article still the best solution? Or, am I missing something?
Yes, that article is about right.
When you specify the JMX port on your server (-Dcom.sun.management.jmxremote.port=####), you are actually specifying just the registry-port for the application. When you connect it provides an additional server-port that the jconsole actually does all of its work with. To get forwarded to work, you need to know both the registry and server ports.
Something like the following should work to run your application with both the registry and server ports set to 8000. See here for more details.
-Dcom.sun.management.jmxremote.port=8000
-Dcom.sun.management.jmxremote.rmi.port=8000
-Djava.rmi.server.hostname=127.0.0.1
As an aside, my SimpleJMX library allows you to set both ports easily and you can set them both to be the same port.
So, once you know both the port(s) you need to forward, you can set up your ssh command. For example, if you configure the registry and server ports as 8000, you would do:
ssh -L 8000:localhost:8000 remote-host
This creates a local port 8000 which forwards to localhost:8000 on the remote-host. You can specify multiple -L arguments if you need to forward multiple ports. Then you can connect your jconsole to localhost:8000 and it will connect to the remote-host appropriately.
Also, if your server has multiple interfaces, you may need to set the java.rmi.server.hostname variable to bind to the right interface.
-Djava.rmi.server.hostname=10.1.2.3

Continuing the SSH socks method, with newer java versions (around 8u66) you also need to set socksNonProxyHosts empty resulting in:
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 -J-DsocksNonProxyHosts=

Remote profilling java application

I'd like to ask how I can profile REMOTELY a java application. For debugging I can say which port the JVM must listen etc since the machine I'm trying to access is behind an ssh gateway so I manually create an SSH tunnel but I've been googling about the same thing but when profiling and I couldn't seem to find. Basically I'm looking for the equivalent of this command:
java -agentlib:jdwp=transport=dt_socket,server=y,address=8000 -jar /bla/bla
but for profiling so that I can remotely attach a profiler.

Disclaimer: My company develops JProfiler
With JProfiler, the VM parameter is like this:
-agentpath:/path/to/libjprofilerti.so=port=8849
"/path/to/libjprofilerti.so" is the path to the native agent library, on Linux x86, for a 32-bit JVM it would be [JProfiler installation directory]/bin/linux-x86/libjprofilerti.so. With the port parameter, you can tell the agent to listen on a specific port. You can set this to the port of your SSH tunnel.
You can easily generate this VM parameter by invoking Session->Integration Wizards->New Remote Integration in JProfiler's main menu:
On your local machine, you create a new session of type "Attach to profiled JVM" and choose the local port of your SSH tunnel (10022 in the screen shot):

Java exec or runtime not working with ssh

I'm trying to ssh into ubuntu using exec, but for some reason when I execute from the code I get the error
port 22: Connection refused
In the code I use concat to put the strings together, but I know they're put together properly because I print them out and if I copy and paste them into the command line then it will ssh properly.
My code tries:
p1= Runtime.getRuntime().exec(run1);
p1.waitfor();
where
run1 = "ssh -o StrictHostKeyChecking=no -v -i key " + "ubuntu#"+ DNS + " sudo mke2fs -F -j "+device;
Any ideas?

You are initiating the connection, so for it to be refused it means that the machine you are attempting to ssh into is denying you ssh access.
Log into that machine by whatever means you have and verify that the ssh server is running. If it is, then verify that the firewall is not blocking port 22; because, sometimes the ssh server is running but the firewall won't allow network access to the ssh server due to a port blocking rule.
--- Edited after question in comments ---
Is there a difference between ssh in the command line and using exec? Because I can
connect to the server through the command line, which I assume is still using port 22.
So if I can ssh that way does that mean port 22 is working?
There are a few possibilities. Java comes with a Security Manager which only serves to deny programs access to machine resources. This is why it is possible to safely run applet code, which is downloaded from remote servers, as the Security Manager denies permission to access the hard drive or make connections to other machines. In the applet sandbox, it does allow connections back to the originating web server (to download more code and images).
However, the lack of a security exception directs the suspicion away from the Security Manager. The fact that the message uses the words "Connection refused" is a strong indicator that the SSH server you are connecting to won't accept a connection from you.
Perhaps by operating on the command line, the ssh command is using a different environment or configuration. I would see if the command is aliased, of if the ssh connection makes some assumption about key pairs. If nothing seems to be out of line with the command, I would check that your program is connection with ssh version 2 (version 1 is not allowed by many due to a security hole).
Then I would also hunt around for possible differences in name resolution. You might be resolving the hostname in the command line differently than you are resolving it from the Java program. This could mean that the Java program is attempting connection to a different machine, one which doesn't have a secure shell server running.
Either way, it seems that you'll have to do a bit of debugging to isolate if it is a true coding problem or an environmental issue.

If you are getting Connection refused, the SSHD is not running or you are being blocked by Firewall (or similiar).

JMX : cant connect from Windows box to monitor JMX port on a Linux system

I have a lot of experience with jconsole.exe and JVisualVM.exe , in the JDK1.6 and have connected thousands of times from a Windows JVM over to a Windows JVM on another machine via JMX.remote but it fails when I try to monitor a java instance that is running on a Linux host (from a Windows host and I tried JMX listener on redhat and SUSE at Amazon EC2). I also tried using jconsole.exe and get a similar error.
Is there any reason anyone can think of , why this kind of JMX connection would have a problem. Any ideas I can try? Has anyone "actually" done this and can say it will work if I persevere?
The error I get from JVisualVM (on the remote connect attempt) is something like this:
"Cannot connect using service:jmx:rmi:///jndi/rmi://<jmx service ip>:8001/jmxrmi"
My remote JMX service config is like this:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8001
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.hostname=<jmx service ip>
Found a similar question at this link but it didn't answer my question.
I verified that "iptables" is not enabled as a service and is not on and so I don't imaging there is any sort of firewall blockage. Also, both the windows and linux machine are on a 10.0.0.0 private internal subnet together. I am able to telnet into the port 8001 to verify its there , and its listening (netstat -ap) on the linux machine.

See the system property java.rmi.server.hostname. On your Linux server JVM, set this system property to be the public IP of the host. Then use the public IP in you client JMX URL.

I think your problem is that the RMI implementation is kind of hard to work with through firewalls and such as it requires more than the port you specify to work. Things you typically wouldn't notice if you connect to your own machine or a machine on the same network.
This post describes the problems you will run into quite well
If I were you, I would try to setup jmxmp as an alternative protocol. To do so you need to add the jmxremote_optional.jar (free from Oracle, download "JMX Remote API 1.0.1_04 Reference Implementation" from here) to both the server and the jvisualvm classpath but it is worth it.
If you google for jmxmp you will find quite a few examples on how to set things up, one of my first hits are http://pub.admc.com/howtos/jmx/distributed-chapt.html#jmxmp-sect which may be a bit too code oriented but I add it here anyway because it explains the things like like most with jmxmp in a few good sentences.
How to define the server side endpoint is depending on what you are running. Most app servers will let you type in a jmxmp enabled jmx service url but if the server is written from scratch you might have to set it up yourself in code instead of using the -D switches to java you are used to.
Give it a try and return with more specific questions about it if you run into problems.
Edit:
After you have added the jar to the classpath, the only thing you have to do in your code (assuming you are not using a server app that already handles it for you) is the following (omitting declarations, exception handling and such as you will figure it out anyway):
url=new JMXServiceURL(jmxurl);
this.server = JMXConnectorServerFactory.newJMXConnectorServer(url, null, ManagementFactory.getPlatformMBeanServer());
this.server.start();

Have you tried creating an SSH tunnel to the Linux box from your Windows machine? http://oldsite.precedence.co.uk/nc/putty.html
Or if you have cygwin, just try ssh -f your-user#remote-server.com -L 7777:remote-server.com:123 -N where 7777 is the port on YOUR windows machine, and 123 is the port on the remote Linux box that listens to JMX commands.
With either of the above, you can use jconsole or visualvm on your Windows box and connect to localhost:7777.
I know iptables is disabled, but just confirm that JMX is working fine at that port by SSHing into the Linux box and trying to use commandline JMX mode of jconsole for localhost (on the Linux box).

I have actually just tackled this problem myself and figured it out.
I would wager that the problem is the RMI connections - you can't predict which ports it will use and so you can't get it to work with a firewall.
The workaround is to use an SSH proxy:
SSH to the box where your application is running but use the -D option like this:
ssh user#remoteHost -D 9999
As you are on Windows, you can use Cygwin to run the command above, or you can do the same thing with Putty through the GUI (guide here: http://blog.ashurex.com/2012/03/15/creating-ssh-proxy-tunnel-putty/)
This will start a socks proxy on your local machine on port 9999.
Open JVisualVM and in the preferences, under 'network' configure it to use a socks proxy at localhost, on port 9999.
If you do the above, you should then be able to connect to the remote machine as normal and since all the RMI traffic is now going over the SSH proxy, it is punched through the firewall and works nicely.
Good luck :-)