Develop Reference

How to profile remote ubuntu JVM using VisualVM?

I am trying to profile remote JVM using VisualVM. I have a remote production ubuntu machine on which my Java application is running and that's what I need to profile. I was following this tutorial to profile a remote server.
I started jstatd on my ubuntu production machine like this -
root#productionMachineA:/home/david# /usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/jstatd -J-Djava.security.policy=permissions.txt -J-Djava.rmi.server.hostname=100.41.76.19 -J-Djava.rmi.server.logCalls=true -J-Djava.net.preferIPv4Stack=true
Here 100.41.76.19 is the IP Address of my production ubuntu machine. After starting jstatd on the ubunut machine, I did -
netstat -nlp | grep jstatd
And I can see this -
root#productionMachineA:~$ netstat -nlp | grep jstatd
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN 32103/jstatd
tcp 0 0 0.0.0.0:60707 0.0.0.0:* LISTEN 32103/jstatd
which looks to me jstatd is running fine I guess. Now I opened VisualVM on my desktop, right click on Remote and select Add Remote Host, and finally type the IP address of the production machineA. And afterwards I don't see anything happening on VisualVM which makes me think something is wrong for sure.
Can anyone tell me what's wrong and what are the things I should try on? If anyone can provide steps by steps what I am supposed to do then it will be of great help.
Update:-
After adding port 1099 on my remote connection.
I got this error. Cannot connect to 100.41.76.19 using service:jmx:rmi.....
From my local desktop, I tried telnet on remote machine on port 1099 and this is what I got -
david#localDesktop ~
$ telnet 100.41.76.19 1099
david#localDesktop ~
$

How to connect to Java instances running on EC2 using JMX

We are having problem connecting to our Java applications running in Amazon's EC2 cluster. We definitely have allowed both the "JMX port" (which is usually the RMI registry port) and the server port (which does most of the work) to the security-group for the instances in question. Jconsole connects but seems to hang and never show any information.
We are running our java with something like the following:
java -server -jar foo.jar other parameters here > java.log 2>&1
We have tried:
Telnets to the ports connect but no information is displayed.
We can run jconsole on the instance itself using remote-X11 over ssh and it connects and shows information. So the JRE is exporting it locally.
Opening all ports in the security group. Weeee.
Using tcpdump to make sure the traffic is not going to other ports.
Simulating it locally. We can always connect to our local JREs or those running elsewhere on our network using the same application parameters.
java -version outputs:
OpenJDK Runtime Environment (IcedTea6 1.11.5) (amazon-53.1.11.5.47.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
As an aside, we are using my Simple JMX package which allows us to set both the RMI registry and server ports which are typically semi-randomly chosen by the RMI registry. You can also force this with something like the following JMX URI:
service:jmx:rmi://localhost:" + serverPort + "/jndi/rmi://:" + registryPort + "/jmxrmi"
These days we use the same port for both the server and the registry. In the past we have used X as the registry-port and X+1 for the server-port to make the security-group rules easy. You connect to the registry-port in jconsole or whatever JMX client you are using.

We are having problem connecting to our Java applications running in Amazon's EC2 cluster.
It turns out that the problem was a combination of two missing settings. The first forces the JRE to prefer ipv4 and not v6. This was necessary (I guess) since we are trying to connect to it via a v4 address:
-Djava.net.preferIPv4Stack=true
The real blocker was the fact that JMX works by first contacting the RMI port which responds with the hostname and port for the JMX client to connect. With no additional settings it will use the local IP of the box which is a 10.X.X.X virtual address which a remote client cannot route to. We needed to add the following setting which is the external hostname or IP of the server -- in this case it is the elastic hostname of the server.
-Djava.rmi.server.hostname=ec2-107-X-X-X.compute-1.amazonaws.com
The trick, if you are trying to automate your EC2 instances (and why the hell would you not), is how to find this address at runtime. To do that you need to put something like the following in our application boot script:
# get our _external_ hostname
RMI_HOST=`wget -q -O - http://169.254.169.254/latest/meta-data/public-hostname`
...
java -server \
-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=$RMI_HOST \
-jar foo.jar other parameters here > java.log 2>&1
The mysterious 169.254.169.254 IP in the wget command above provides information that the EC2 instance can request about itself. I'm disappointed that this does not include tags which are only available in an authenticated call.
I initially was using the extern ipv4 address but it looks like the JDK tries to make a connection to the server-port when it starts up. If it uses the external IP then this was slowing our application boot time until that timed out. The public-hostname resolves locally to the 10-net address and to the public-ipv4 externally. So the application now is starting fast and JMX clients still work. Woo hoo!
Hope this helps someone else. Cost me 3 hours today.
To force your JMX server to start the server and the RMI registry on designated ports so you can block them in the EC2 Security Groups, see this answer:
How to close rmiregistry running on particular port?
Edit:
We just had this problem re-occur. It seems that the Java JMX code is doing some hostname lookups on the hostname of the box and using them to try to connect and verify the JMX connection.
The issue seems to be a requirement that the local hostname of the box should resolve to the local-ip of the box. For example, if your /etc/sysconfig/network has HOSTNAME=server1.foobar.com then if you do a DNS lookup on server1.foobar.com, you should get to the 10-NET virtual address. We were generating our own /etc/hosts file and the hostname of the local host was missing from the file. This caused our applications to either pause on startup or not startup at all.
Lastly
One way to simplify your JMX creation is to use my SimpleJMX package.

Per the second answer Why does JMX connection to Amazon EC2 fail?, the difficulty here is that by default the RMI port is selected at random, and clients need access to both the JMX and RMI ports. If you're running jdk7u4 or later, the RMI port can be specified via an app property. Starting my server with the following JMX settings worked for me:
Without authentication:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=<public EC2 hostname>
With authentication:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password.file=/path/to/jmxremote.password
-Djava.rmi.server.hostname=<public EC2 hostname>
I also opened ports 9998-9999 in the EC2 security group for my instance.

A bit different approach by using ssh tunnels
(On the Remote machine) Pass the following flags to the JVM
-Dcom.sun.management.jmxremote.port=1099
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=127.0.0.1
(On the Remote machine) Check which ports java started to use
$ netstat -tulpn | grep java
tcp 0 0 0.0.0.0:37484 0.0.0.0:* LISTEN 2904/java
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN 2904/java
tcp 0 0 0.0.0.0:45828 0.0.0.0:* LISTEN 2904/java
(On the local machine) Make ssh tunnels for all the ports
ssh -N -L 1099:127.0.0.1:1099 ubuntu#<ec2_ip>
ssh -N -L 37484:127.0.0.1:37484 ubuntu#<ec2_ip>
ssh -N -L 45828:127.0.0.1:45828 ubuntu#<ec2_ip>`
(On the local machine) Connect by Java Mission Control to localhost:1099

The answer given by Gray worked for me, however I find that I have to open TCP ports 0 to 65535 or I don't get in. I think that you can connect on the main JMX port, and then get another one assigned. I got that from this blog post that has always worked well for me.

We are using AWS Elastic Container Service for running our spring boot services.
The below config allowed us to connect to our docker containers.
Without Authentication:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9090 \
-Dcom.sun.management.jmxremote.rmi.port=9090 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=$(/usr/bin/curl -s --connect-timeout 2 \
http://169.254.169.254/latest/meta-data/public-ipv4)
I found it crisp and also doesn't require any other servicer side init script.

eclipse: remote debugging a tomcat server behind a firewall

After starting tomcat with jpda on, while at my company I can remote debug a bunch of web applications in eclipse. For a number of reasons I am now in need of developing and remote debugging those same webapps from outside the company firewall, and i can only access that server via ssh on port 22.
I tunneled most needed ports (svn, nexus, tomcat itself, from the server or via the server) to localhost and those services work fine, but I cannot start the eclipse debugger in any way; i'm getting "connection timed out while waiting for packet XXX" or "connection refused" from the second time i try on.
Checking with nmap on the server, it reports the port open before the first connection attempt, and it becomes closed after that. I get no interesting output log in catalina.out
The command i use to start the tunnel is:
ssh -L 8000:localhost:8000 user#mycompany.com
iptables was temporarily stopped both on the server and in the local machine for testing.
Am i missing something? Do I need to forward some other port to localhost? Or is it in some way involved name resolution?
EDIT
Open ports before connection attemp from eclipse:
root#lnxulisse:/opt/apache-tomcat-6.0.32/bin# lsof -p 2147 -n |grep TCP
java 2147 root 4u IPv4 640850 0t0 TCP *:8000 (LISTEN)
java 2147 root 38u IPv6 640859 0t0 TCP *:http-alt (LISTEN)
java 2147 root 40u IPv6 640865 0t0 TCP *:https (LISTEN)
java 2147 root 46u IPv6 640908 0t0 TCP 127.0.0.1:18005 (LISTEN)
java 2147 root 48r IPv6 642625 0t0 TCP 172.24.0.82:48347->172.24.0.82:mysql (ESTABLISHED)
java 2147 root 181u IPv6 640891 0t0 TCP 172.24.0.82:60353->172.24.0.82:mysql (ESTABLISHED)
and after:
java 2147 root 4u IPv6 642769 0t0 TCP 172.24.0.82:48956->172.24.0.82:mysql (ESTABLISHED)
java 2147 root 5u IPv4 640851 0t0 TCP 127.0.0.1:8000->127.0.0.1:34193 (ESTABLISHED)
java 2147 root 38u IPv6 640859 0t0 TCP *:http-alt (LISTEN)
java 2147 root 40u IPv6 640865 0t0 TCP *:https (LISTEN)
java 2147 root 46u IPv6 640908 0t0 TCP 127.0.0.1:18005 (LISTEN)
java 2147 root 181u IPv6 640891 0t0 TCP 172.24.0.82:60353->172.24.0.82:mysql (ESTABLISHED)
exact eclipse error returned is:
Exception occurred during launch
Failed to connect to remote JVM. Connection timed out.
Timeout occurred while waiting for packet 204.
(the packet number varies on each attempt).
in workspace/.metadata/.log i get:
!ENTRY org.eclipse.osgi 2 0 2011-07-17 18:43:53.024
!MESSAGE While loading class "org.eclipse.core.net.proxy.IProxyService", thread "Thread[main,6,main]" timed out waiting (5000ms) for thread "Thread[Thread-6,5,main]" to finish starting bundle "org.eclipse.core.net_1.2.1.r35x_20090812-1200 [232]". To avoid deadlock, thread "Thread[main,6,main]" is proceeding but "org.eclipse.core.net.proxy.IProxyService" may not be fully initialized.
!STACK 0
org.osgi.framework.BundleException: State change in progress for bundle "reference:file:plugins/org.eclipse.core.net_1.2.1.r35x_20090812-1200.jar" by thread "Thread-6".
at org.eclipse.osgi.framework.internal.core.AbstractBundle.beginStateChange(AbstractBundle.java:1073)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:278)
[...]
!ENTRY org.eclipse.ui.ide 4 4 2011-07-17 18:43:53.028
!MESSAGE Proxy service could not be found.
eclipse is configured for direct internet connection.
EDIT 2
I think the solution might be here:
http://blog.cantremember.com/debugging-with-jconsole-jmx-ssh-tunnels/
but i have some trouble understanding his JNDI/RMI settings, and to what extent that applies to my configuration.
EDIT 3
This is a clarification for those answering "use <lan|local ip address> instead of <localhost>"
computer A: my workstation in the company
computer B: my workstation at home
computer C: server running tomcat
B and C are in two different sub-networks in the same network infrastructure; only connections to port 22 of C from outside are allowed (and somewhat "proxied", I don't know network internals).
A is "outside" (my dsl connection with dynamic ip address).
Debugging on C from B via ssh tunnel -> works
Debugging on C from A via ssh tunnel -> connection timed out while waiting for packet XXX

This article suggests that the default port on which the remote Java virtual machine (JVM) is listening in debugging mode is 1044. You should tunnel the port on which the remote JVM is running as well.
More generally, you could run wireshark/tcpdump to see to which port connection attempts are made when starting the debugger.
EDIT:
A few more things I would try:
check on the remote host (e.g. with ps auxwww if it's Linux) with which arguments (look for what comes behind -Xrunjdwp or with lsof -p PID_OF_JVM_TO_BE_DEBUGGED on which TCP port it listens (look for lines with TCP and LISTEN in the lsof output)
make sure that the JVM on the remote host listens on the lo interface, not the network interface (that's what you specify with the localhost in the -L option to ssh).
Does starting the debugger by hand on the machine where you start eclipse with jdb -attach localhost:8000 work ? (you could also try this on the remote host to ensure the debugger is running on the port 8000)
make sure that eclipse tries to connect to localhost (when not specifying a bind address before the first 8000 with the -L option ssh listens on the lo interface)

I often had this problem when doing remote debugging. I do not know the exact reason for this problem, but I used the below solution and maybe it works for you, too:
instead of
ssh -L 8000:localhost:8000 user#remotehost
is used
ssh -L 8000:remotehost:8000 user#remotehost
for creating the SSH tunnel (note the remotehost instead of localhost between the port numbers in the second example). Instead of the remote host's name, you can also use the normal IP address of the remote host (not the loopback address 127.0.0.1, but the true local network IP address).
Hope it helps and good luck!

Assuming the remote Tomcat instance has been started with something like -Xrunjdwp:transport=dt_socket,server=y,address=8000,suspend=n, try this command:
ssh -L 8000:0.0.0.0:8000 user#mycompany.com -N
On my Mac, I tried out ssh -L 10701:localhost:10700 user#localhost -N locally, where a Tomcat instance was started with -Xrunjdwp:transport=dt_socket,server=y,address=10700,suspend=n, and attempting to attach on port 10701 within Eclipse, I kept seeing "Failed to connect to remote VM com.sun.jdi.connect.spi.ClosedConnectionException". By changing the tunnel command to ssh -L 10701:0.0.0.0:10700 user#localhost -N, Eclipse was able to attach.

Can you please give the exact parameters of the -Xrunjdwp parameter?
Also do you have tried different methods for debugging (server=y/n, suspend=y/n)?
Perhaps inversing the connection (let the tomcat connect to the debugger instead of letting the debugger connect to tomcat) may help.

Well I am answering myself after a long time; in my specific case, the solution was to put eclipse JVM in listening mode:
Connection Type: "Standard (Socket Listen)"
and reverse the direction of the tunnel:
ssh -L 8001:localhost:8001 user#work (run on server (S), "localhost" is W)
ssh -R 8001:localhost:8001 user#work (run at home (H), "localhost" is W)
Some explanation: as in the question, my situation was:
H -------------------> S not working ( ssh -L 8001:S:8001 user#S from H)
H W -------> S working ( ssh -L 8001:S:8001 user#S from W)
home work server
While reversing like this:
H <------- W S ssh -R 8001:localhost:8001 user#W (from H)
H W <------- S ssh -L 8001:localhost:8001 user#W (from S)
home work server
did the trick. In other words, whatever is written on S:8001, is forwarded to W:8001, and whatever in turn is written to W:8001, is forwarded to H:8001, where my eclipse JVM is listening.
The tomcat JVM on S should be started with server=n, with arguments:
-agentlib:jdwp=transport=dt_socket,server=n,suspend=n,address=8001

Glassfish error upon attempting to use in Netbeans

Good day
I am trying to study JAVA EE so I installed the Glassfish 3. But when I attempted to deploy my project on Netbeans 6.9. I get the following error:
SEVERE: Shutting down v3 due to startup exception : Address already in use: bind: 8080=com.sun.enterprise.v3.services.impl.monitor.MonitorableSelectorHandler#106433d
And the server won't start.
It seems like that the port 8080 is already in used.
I go to Control Panel -> Administrative Tools -> Services but I don't know which application to kill because the port is not indicated there. How can I know which application is currently running at port 8080 so I could kill it?
EDIT: As per your suggestions, I did the netstat -aon
The result is as follows:
TCP 0.0.0.0:3700 0.0.0.0:0 LISTENING 4724
TCP 0.0.0.0:4848 0.0.0.0:0 LISTENING 4724
TCP 0.0.0.0:8080 0.0.0.0:0 LISTENING 4724
TCP 0.0.0.0:8181 0.0.0.0:0 LISTENING 4724
TCP 0.0.0.0:8686 0.0.0.0:0 LISTENING 4724
Can I kill this? I don't know what this means -> "LISTENING".. Sorry I don't understand this result much..
Anyway I killed the application as suggested by #Jigar Joshi and it Worked!
Thank you all for your help!

Go to Command prompt
Type the following command
netstat -aon | findstr "8080"
ex : TCP 10.12.230.222:2049 10.12.240.69:8080 ESTABLISHED 3476
Get the process id from the last column and run the following command
tasklist | findstr "3476"
for example you might get like this
firefox.exe 3476 RDP-Tcp#5 0 168,668 K
go to task manager and kill the firefox or whatever running on 8080 and start the server.

error message tells that the port is already binded with some other process now as you are running on windows machine
&
go to Control Panel -> Administrative Tools -> Services but I don't know which application I want to kill because the port is not indicated there. How can I know which application is currently running at port 8080 so I could kill it?
goto command prompt
netstat -aon
it will show you something like
TCP 192.1.200.48:2053 24.43.246.60:443 ESTABLISHED 248
TCP 192.1.200.48:2055 24.43.246.60:443 ESTABLISHED 248
TCP 192.1.200.48:2126 213.146.189.201:12350 ESTABLISHED 1308
TCP 192.1.200.48:3918 192.1.200.2:8073 ESTABLISHED 1504
TCP 192.1.200.48:3975 192.1.200.11:49892 TIME_WAIT 0
TCP 192.1.200.48:3976 192.1.200.11:49892 TIME_WAIT 0
TCP 192.1.200.48:4039 209.85.153.100:80 ESTABLISHED 248
TCP 192.1.200.48:8080 209.85.153.100:80 ESTABLISHED 248
check which process has binded your port. here in above example its 248 now if you are sure that you need to kill that process fire
Linux:
kill -9 248
Windows:
taskkill /f /pid 248
it will kill that process

Java App servers typically run on port 8080. Look from Apache Tomcat or another java web server that you might have installed and started.

Unable to connect to a remote JVM

I'm working on a Java 10 application that uses an embedded Jetty server to provide control from a local network, and I'm attempting to connect to the JVM and failing. It's running on Ubuntu 18.04 LTS desktop.
My startup script has the following lines:
java -Xdebug -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n -Djava.library_path=${LIB_PATH} -classpath ${CP} -jar ${APP_DIR}/app.jar
I have ufw on the system, and I've verified that the port is open. My output from ufw status includes:
8000 ALLOW Anywhere
8000 (v6) ALLOW Anywhere
In IntelliJ, my debug configuration is
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8000
When I try to connect, I get an error message that the connection is refused.
with the IP of the machine in the config's address box.
Looking at the output of netstat -l, I see the following:
tcp 0 0 localhost:8000 0.0.0.0:* LISTEN
Does this mean that the debugger is only listening for connection on the localhost? Do I need to do something to have it listen on a network?

So I found the answer fairly quick. I needed to modify the line I use for the server so that it reads:
java -Xdebug -agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n -Djava.library_path=${LIB_PATH} -classpath ${CP} -jar ${APP_DIR}/app.jar
So that is listens on all interfaces.