I'd like to ask how I can profile REMOTELY a java application. For debugging I can say which port the JVM must listen etc since the machine I'm trying to access is behind an ssh gateway so I manually create an SSH tunnel but I've been googling about the same thing but when profiling and I couldn't seem to find. Basically I'm looking for the equivalent of this command:
java -agentlib:jdwp=transport=dt_socket,server=y,address=8000 -jar /bla/bla
but for profiling so that I can remotely attach a profiler.
Disclaimer: My company develops JProfiler
With JProfiler, the VM parameter is like this:
-agentpath:/path/to/libjprofilerti.so=port=8849
"/path/to/libjprofilerti.so" is the path to the native agent library, on Linux x86, for a 32-bit JVM it would be [JProfiler installation directory]/bin/linux-x86/libjprofilerti.so. With the port parameter, you can tell the agent to listen on a specific port. You can set this to the port of your SSH tunnel.
You can easily generate this VM parameter by invoking Session->Integration Wizards->New Remote Integration in JProfiler's main menu:
On your local machine, you create a new session of type "Attach to profiled JVM" and choose the local port of your SSH tunnel (10022 in the screen shot):
Related
Memory Analysers (Instrumentation and Monitoring tools) like VisualVM and jProfiler connect to Java Application's JVM though JMX extensions (though there might be other means to connect - like jstatd etc, i have seen JMX is quite common)
My Understanding About JMX:
By default JMX must expose its default port(not sure if there is a default port number) so that Memory Analysers can connect. So, I assume that when more than one java apps are running with default JMX config, on the same machine, there must be a JMX port conflict.
But I have never noticed that. I have seen java apps running happily with default configs and Mem Analysers could happily connect with each of these java apps at the same time. So my understanding about JMX ports is not entirely correct. Could some one say how more than one java app is able to expose JMX functionality with default configurations at the same time on the same machine. (???? is a random port used by JMX for each java application????)
Tools like VisualVM use JMX together with Dynamic Attach mechanism to monitor local Java Virtual Machines.
First, the tool connects to a local JVM via Attach
API.
Then it executes (also via Attach API) a command to start Management Agent (JMX server) in the target JVM.
The target JVM starts Management Agent on some free port and sets the opened port value in the Agent properties.
The tool uses Attach API again to read Agent properties, and thus discovers the port the Agent listens to.
Then it establishes the JMX connection to the Management Agent on this port.
Obviously, different local JVMs start Management Server on different ports, but VisualVM discovers the port number via Dynamic Attach.
I have my web application running on java 8, which i need to connect through jconsole . I am able to connect jconsole without any change(i am not sure
if i did any change in the past) but my collegue is not able to connect through jconsole
We both are running java application separately on our windows machine with separate jconsole on same box.
So my question in general is do we need to do any configuration/setting to monitor java application/process from jconsole on same machine or remote machine?
I am trying to use JProfiler on my MAC to connect to a machine remotely that is behind a firewall and only accessible via a Linux machine.
I have set up a direct SSH tunnel as follows:
ssh -L 8849:remote:8849 forwardingmachine
And start Jprofiler with
java -agentpath:/path/jprofiler8/bin/linux-x64/libjprofilerti.so=port=8849 ..."
I systematically get the error:
Connection error
Either an old version of the native library is used or another
application is listening on port 8849. Please check your
DYLD_LIBRARY_PATH environment variable and your port configuration
Online there are solutions for using a 2 hop SSH tunnel, but I can't do that due to the second machine only being accessible to forwardingmachine.
Any ideas of how to get around this?
(The remote machine uses Java 1.7, whereas my Mac uses Java 1.8). Both machines are using jprofiler8
A direct tunnel to remote is set up with the command
ssh -t user#remote -L [localPort]:localhost:[remotePort] -N
I'd like to be able to remotely connect to a Java service that has JMX exposed, however it is blocked by a firewall. I have tried to use ssh local port forwarding, however the connection fails. Looking at wireshark, it appears that when you try to connect with jconsole, it wants to connect via some ephemeral ports after connecting to port 9999, which are blocked by the firewall.
Is there any way to make jconsole only connect through 9999 or use a proxy? Is this article still the best solution? Or, am I missing something?
There's an even nicer way to do this using an SSH socks tunnel, since JConsole supports SOCKS:
Create the SSH socks proxy locally on some free port (e.g. 7777):
ssh -fN -D 7777 user#firewalled-host
Run JConsole by specifying the SOCKS proxy (e.g. localhost:7777) and the address for the JMX server (e.g. localhost:2147)
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 service:jmx:rmi:///jndi/rmi://localhost:2147/jmxrmi -J-DsocksNonProxyHosts=
As mentioned in one of the answers below, from JDK 8u60+ you also need to have the -J-DsocksNonProxyHosts= option in order to get it working.
With almost all current JDK versions (7u25 or later) it's now possible to use JConsole and Visual JVM over SSH quite easily (because now you can bind JMX to single port).
I use the following JVM parameters
-Dcom.sun.management.jmxremote.port=8090
-Dcom.sun.management.jmxremote.rmi.port=8090
-Djava.rmi.server.hostname=127.0.0.1
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Then I launch SSH connection
ssh my.javaserver.domain -L 8090:127.0.0.1:8090
After I can connect from JConsole
Remote Process: -> localhost:8090
And Java Visual VM
Right Click on Local -> Add JMX Connection -> localhost:8090
Is there any way to make jconsole only connect through 9999 or use a proxy? Is this article still the best solution? Or, am I missing something?
Yes, that article is about right.
When you specify the JMX port on your server (-Dcom.sun.management.jmxremote.port=####), you are actually specifying just the registry-port for the application. When you connect it provides an additional server-port that the jconsole actually does all of its work with. To get forwarded to work, you need to know both the registry and server ports.
Something like the following should work to run your application with both the registry and server ports set to 8000. See here for more details.
-Dcom.sun.management.jmxremote.port=8000
-Dcom.sun.management.jmxremote.rmi.port=8000
-Djava.rmi.server.hostname=127.0.0.1
As an aside, my SimpleJMX library allows you to set both ports easily and you can set them both to be the same port.
So, once you know both the port(s) you need to forward, you can set up your ssh command. For example, if you configure the registry and server ports as 8000, you would do:
ssh -L 8000:localhost:8000 remote-host
This creates a local port 8000 which forwards to localhost:8000 on the remote-host. You can specify multiple -L arguments if you need to forward multiple ports. Then you can connect your jconsole to localhost:8000 and it will connect to the remote-host appropriately.
Also, if your server has multiple interfaces, you may need to set the java.rmi.server.hostname variable to bind to the right interface.
-Djava.rmi.server.hostname=10.1.2.3
Continuing the SSH socks method, with newer java versions (around 8u66) you also need to set socksNonProxyHosts empty resulting in:
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 -J-DsocksNonProxyHosts=
I have a lot of experience with jconsole.exe and JVisualVM.exe , in the JDK1.6 and have connected thousands of times from a Windows JVM over to a Windows JVM on another machine via JMX.remote but it fails when I try to monitor a java instance that is running on a Linux host (from a Windows host and I tried JMX listener on redhat and SUSE at Amazon EC2). I also tried using jconsole.exe and get a similar error.
Is there any reason anyone can think of , why this kind of JMX connection would have a problem. Any ideas I can try? Has anyone "actually" done this and can say it will work if I persevere?
The error I get from JVisualVM (on the remote connect attempt) is something like this:
"Cannot connect using service:jmx:rmi:///jndi/rmi://<jmx service ip>:8001/jmxrmi"
My remote JMX service config is like this:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8001
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.hostname=<jmx service ip>
Found a similar question at this link but it didn't answer my question.
I verified that "iptables" is not enabled as a service and is not on and so I don't imaging there is any sort of firewall blockage. Also, both the windows and linux machine are on a 10.0.0.0 private internal subnet together. I am able to telnet into the port 8001 to verify its there , and its listening (netstat -ap) on the linux machine.
See the system property java.rmi.server.hostname. On your Linux server JVM, set this system property to be the public IP of the host. Then use the public IP in you client JMX URL.
I think your problem is that the RMI implementation is kind of hard to work with through firewalls and such as it requires more than the port you specify to work. Things you typically wouldn't notice if you connect to your own machine or a machine on the same network.
This post describes the problems you will run into quite well
If I were you, I would try to setup jmxmp as an alternative protocol. To do so you need to add the jmxremote_optional.jar (free from Oracle, download "JMX Remote API 1.0.1_04 Reference Implementation" from here) to both the server and the jvisualvm classpath but it is worth it.
If you google for jmxmp you will find quite a few examples on how to set things up, one of my first hits are http://pub.admc.com/howtos/jmx/distributed-chapt.html#jmxmp-sect which may be a bit too code oriented but I add it here anyway because it explains the things like like most with jmxmp in a few good sentences.
How to define the server side endpoint is depending on what you are running. Most app servers will let you type in a jmxmp enabled jmx service url but if the server is written from scratch you might have to set it up yourself in code instead of using the -D switches to java you are used to.
Give it a try and return with more specific questions about it if you run into problems.
Edit:
After you have added the jar to the classpath, the only thing you have to do in your code (assuming you are not using a server app that already handles it for you) is the following (omitting declarations, exception handling and such as you will figure it out anyway):
url=new JMXServiceURL(jmxurl);
this.server = JMXConnectorServerFactory.newJMXConnectorServer(url, null, ManagementFactory.getPlatformMBeanServer());
this.server.start();
Have you tried creating an SSH tunnel to the Linux box from your Windows machine? http://oldsite.precedence.co.uk/nc/putty.html
Or if you have cygwin, just try ssh -f your-user#remote-server.com -L 7777:remote-server.com:123 -N where 7777 is the port on YOUR windows machine, and 123 is the port on the remote Linux box that listens to JMX commands.
With either of the above, you can use jconsole or visualvm on your Windows box and connect to localhost:7777.
I know iptables is disabled, but just confirm that JMX is working fine at that port by SSHing into the Linux box and trying to use commandline JMX mode of jconsole for localhost (on the Linux box).
I have actually just tackled this problem myself and figured it out.
I would wager that the problem is the RMI connections - you can't predict which ports it will use and so you can't get it to work with a firewall.
The workaround is to use an SSH proxy:
SSH to the box where your application is running but use the -D option like this:
ssh user#remoteHost -D 9999
As you are on Windows, you can use Cygwin to run the command above, or you can do the same thing with Putty through the GUI (guide here: http://blog.ashurex.com/2012/03/15/creating-ssh-proxy-tunnel-putty/)
This will start a socks proxy on your local machine on port 9999.
Open JVisualVM and in the preferences, under 'network' configure it to use a socks proxy at localhost, on port 9999.
If you do the above, you should then be able to connect to the remote machine as normal and since all the RMI traffic is now going over the SSH proxy, it is punched through the firewall and works nicely.
Good luck :-)