I'm working on a Java 10 application that uses an embedded Jetty server to provide control from a local network, and I'm attempting to connect to the JVM and failing. It's running on Ubuntu 18.04 LTS desktop.
My startup script has the following lines:
java -Xdebug -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n -Djava.library_path=${LIB_PATH} -classpath ${CP} -jar ${APP_DIR}/app.jar
I have ufw on the system, and I've verified that the port is open. My output from ufw status includes:
8000 ALLOW Anywhere
8000 (v6) ALLOW Anywhere
In IntelliJ, my debug configuration is
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8000
When I try to connect, I get an error message that the connection is refused.
with the IP of the machine in the config's address box.
Looking at the output of netstat -l, I see the following:
tcp 0 0 localhost:8000 0.0.0.0:* LISTEN
Does this mean that the debugger is only listening for connection on the localhost? Do I need to do something to have it listen on a network?
So I found the answer fairly quick. I needed to modify the line I use for the server so that it reads:
java -Xdebug -agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n -Djava.library_path=${LIB_PATH} -classpath ${CP} -jar ${APP_DIR}/app.jar
So that is listens on all interfaces.
Related
I try configure jmx according this instruction: http://activemq.apache.org/jmx.html
On localhost all works well. But when i try connect to FreeBSD server over VPN jconsole can't establish connection.
I use such settings for ACTIVEMQ_SUNJMX_START variable:
ACTIVEMQ_SUNJMX_START="
-Dcom.sun.management.jmxremote.port=1616
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_BASE}/conf/jmx.password
-Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_BASE}/conf/jmx.access
-Djava.rmi.server.hostname=vpn_id_address
-Dcom.sun.management.jmxremote.local.only=false"
What i do wrong?
Confirm the port is listening, using netstat or other tool.
netstat -na | grep 1616
... should show an entry as LISTENING
Try to telnet to the port and send garbage. The server will disconnect you
$ telnet localhost 1099
Trying ::1...
Connected to localhost.
Escape character is '^]'.
garbage
Connection closed by foreign host.
There are a million other reasons why it may not be working. Firewall settings on the server, VPN port mapping, etc.. etc.. Until #1 and #2 are confirmed there isn't anything else to go on.
open command prompt/terminal then type: jconsole
then your jconsole will open and connect to activemq from it
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
So i was going to debug my Solr filter plugins on Intellij Community Edition. After i ran the program from comand prompt with this command
java -jar start.jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8983
I started my Intellij debugger with this config:
Transport : socket
Debugger mode : attach
Host : localhost
Port : 8983
But when I ran the debugger I got this error:
Error running Debugger: Unable to open debugger port (localhost:8983):
java.io.IOException "handshake failed - connection prematurally closed"
Any idea how to fix this?
I got that error when trying to access to debug port on a Docker container.
If you are trying to access the debug port inside a Docker container make sure you are specifying the port as *:5005
E.g.
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
This has been changes since Java 9.
See: REGRESSION: Remote debugging does not work on JDK 9
It's not a bug. It's a security.
Before the JDK-8041435
If you have a server with EXT and INT interfaces and start Java process with address=5900 it binds to both interfaces and allow anybody from entire world to connect to your java process unless you block it on firewall.
After JDK-8041435 socket transport try to guess localhost and bind to localhost only. I.e. socket transport by default works only if both client and server are located on the same machine. It's not an easy task to guess proper localhost. so ever same-machine configuration might not work in some situation because of network setup.
You can restore old, insecure behavior using * (asteric)
i.e.
-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5900
should work exactly as it was before JDK-8041435
But it's recommended to explicitly specify ip address to bind when it possible.
And JDWP socket connector accept only local connections by default
The JDWP socket connector has been changed to bind to localhost only if no ip address or hostname is specified on the agent command line. A hostname of asterisk (*) may be used to achieve the old behavior which is to bind the JDWP socket connector to all available interfaces; this is not secure and not recommended.
It should be something like this,
java "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8983" -jar start.jar
it's working now
I had this error with OpenJDK 11 inside Docker container and setting environment variable JAVA_DEBUG_PORT to "*:5005" worked for me.
You forgot to specify -Xdebug on the java command line.
Edit: As in
java -jar start.jar -Xdebug -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8983
It has helped me, at least in Intellij IDEA:
java -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=0.0.0.0:5005
try to add ip 0.0.0.0.
I am trying to profile remote JVM using VisualVM. I have a remote production ubuntu machine on which my Java application is running and that's what I need to profile. I was following this tutorial to profile a remote server.
I started jstatd on my ubuntu production machine like this -
root#productionMachineA:/home/david# /usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/jstatd -J-Djava.security.policy=permissions.txt -J-Djava.rmi.server.hostname=100.41.76.19 -J-Djava.rmi.server.logCalls=true -J-Djava.net.preferIPv4Stack=true
Here 100.41.76.19 is the IP Address of my production ubuntu machine. After starting jstatd on the ubunut machine, I did -
netstat -nlp | grep jstatd
And I can see this -
root#productionMachineA:~$ netstat -nlp | grep jstatd
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN 32103/jstatd
tcp 0 0 0.0.0.0:60707 0.0.0.0:* LISTEN 32103/jstatd
which looks to me jstatd is running fine I guess. Now I opened VisualVM on my desktop, right click on Remote and select Add Remote Host, and finally type the IP address of the production machineA. And afterwards I don't see anything happening on VisualVM which makes me think something is wrong for sure.
Can anyone tell me what's wrong and what are the things I should try on? If anyone can provide steps by steps what I am supposed to do then it will be of great help.
Update:-
After adding port 1099 on my remote connection.
I got this error. Cannot connect to 100.41.76.19 using service:jmx:rmi.....
From my local desktop, I tried telnet on remote machine on port 1099 and this is what I got -
david#localDesktop ~
$ telnet 100.41.76.19 1099
david#localDesktop ~
$
I want to remote debug an application running in Tomcat 7. Tomcat is running as a service on a Win2008 server.
I added the following to the Java options in the Java Configuration Panel of Tomcat:
-Xdebug -Xrunjdwp:transport=dt_socket,address=4711,server=y,suspend=n and opened the firewall on my workstation and the server for this port.
But when I try debugging from IntelliJ 9 on my workstation, I get an error message Unable to open debugger port : java.net.ConnectException "Connection timed out: connect". The jvm is the standatd Sun/Oracle 64 bit JVM version 1.6.0_27.
I verified that the command line parameters are in use by accessing ManagementFactory.getRuntimeMXBean().getInputArguments() within the application deployed to Tomcat and logging the result to the log file.
I verified via Wireshark on my workstation and on the server that the TCP request on port 4711 is sent from my pc and arriving on the server, but there is is no answer. Running netstat -a on the server does not show a process listening on this port. So I assume somehow Tomcat/JVM does not start the remote debugging.
You need to put -Xdebug and -Xrunjdwp... on separate lines in the Java panel of the Tomcat Service Configuration Panel.
So having:
-Xdebug -Xrunjdwp:transport=dt_socket,address=4711,server=y,suspend=n
will not work, but:
-Xdebug
-Xrunjdwp:transport=dt_socket,address=4711,server=y,suspend=n
will.
We are having problem connecting to our Java applications running in Amazon's EC2 cluster. We definitely have allowed both the "JMX port" (which is usually the RMI registry port) and the server port (which does most of the work) to the security-group for the instances in question. Jconsole connects but seems to hang and never show any information.
We are running our java with something like the following:
java -server -jar foo.jar other parameters here > java.log 2>&1
We have tried:
Telnets to the ports connect but no information is displayed.
We can run jconsole on the instance itself using remote-X11 over ssh and it connects and shows information. So the JRE is exporting it locally.
Opening all ports in the security group. Weeee.
Using tcpdump to make sure the traffic is not going to other ports.
Simulating it locally. We can always connect to our local JREs or those running elsewhere on our network using the same application parameters.
java -version outputs:
OpenJDK Runtime Environment (IcedTea6 1.11.5) (amazon-53.1.11.5.47.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
As an aside, we are using my Simple JMX package which allows us to set both the RMI registry and server ports which are typically semi-randomly chosen by the RMI registry. You can also force this with something like the following JMX URI:
service:jmx:rmi://localhost:" + serverPort + "/jndi/rmi://:" + registryPort + "/jmxrmi"
These days we use the same port for both the server and the registry. In the past we have used X as the registry-port and X+1 for the server-port to make the security-group rules easy. You connect to the registry-port in jconsole or whatever JMX client you are using.
We are having problem connecting to our Java applications running in Amazon's EC2 cluster.
It turns out that the problem was a combination of two missing settings. The first forces the JRE to prefer ipv4 and not v6. This was necessary (I guess) since we are trying to connect to it via a v4 address:
-Djava.net.preferIPv4Stack=true
The real blocker was the fact that JMX works by first contacting the RMI port which responds with the hostname and port for the JMX client to connect. With no additional settings it will use the local IP of the box which is a 10.X.X.X virtual address which a remote client cannot route to. We needed to add the following setting which is the external hostname or IP of the server -- in this case it is the elastic hostname of the server.
-Djava.rmi.server.hostname=ec2-107-X-X-X.compute-1.amazonaws.com
The trick, if you are trying to automate your EC2 instances (and why the hell would you not), is how to find this address at runtime. To do that you need to put something like the following in our application boot script:
# get our _external_ hostname
RMI_HOST=`wget -q -O - http://169.254.169.254/latest/meta-data/public-hostname`
...
java -server \
-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=$RMI_HOST \
-jar foo.jar other parameters here > java.log 2>&1
The mysterious 169.254.169.254 IP in the wget command above provides information that the EC2 instance can request about itself. I'm disappointed that this does not include tags which are only available in an authenticated call.
I initially was using the extern ipv4 address but it looks like the JDK tries to make a connection to the server-port when it starts up. If it uses the external IP then this was slowing our application boot time until that timed out. The public-hostname resolves locally to the 10-net address and to the public-ipv4 externally. So the application now is starting fast and JMX clients still work. Woo hoo!
Hope this helps someone else. Cost me 3 hours today.
To force your JMX server to start the server and the RMI registry on designated ports so you can block them in the EC2 Security Groups, see this answer:
How to close rmiregistry running on particular port?
Edit:
We just had this problem re-occur. It seems that the Java JMX code is doing some hostname lookups on the hostname of the box and using them to try to connect and verify the JMX connection.
The issue seems to be a requirement that the local hostname of the box should resolve to the local-ip of the box. For example, if your /etc/sysconfig/network has HOSTNAME=server1.foobar.com then if you do a DNS lookup on server1.foobar.com, you should get to the 10-NET virtual address. We were generating our own /etc/hosts file and the hostname of the local host was missing from the file. This caused our applications to either pause on startup or not startup at all.
Lastly
One way to simplify your JMX creation is to use my SimpleJMX package.
Per the second answer Why does JMX connection to Amazon EC2 fail?, the difficulty here is that by default the RMI port is selected at random, and clients need access to both the JMX and RMI ports. If you're running jdk7u4 or later, the RMI port can be specified via an app property. Starting my server with the following JMX settings worked for me:
Without authentication:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=<public EC2 hostname>
With authentication:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password.file=/path/to/jmxremote.password
-Djava.rmi.server.hostname=<public EC2 hostname>
I also opened ports 9998-9999 in the EC2 security group for my instance.
A bit different approach by using ssh tunnels
(On the Remote machine) Pass the following flags to the JVM
-Dcom.sun.management.jmxremote.port=1099
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=127.0.0.1
(On the Remote machine) Check which ports java started to use
$ netstat -tulpn | grep java
tcp 0 0 0.0.0.0:37484 0.0.0.0:* LISTEN 2904/java
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN 2904/java
tcp 0 0 0.0.0.0:45828 0.0.0.0:* LISTEN 2904/java
(On the local machine) Make ssh tunnels for all the ports
ssh -N -L 1099:127.0.0.1:1099 ubuntu#<ec2_ip>
ssh -N -L 37484:127.0.0.1:37484 ubuntu#<ec2_ip>
ssh -N -L 45828:127.0.0.1:45828 ubuntu#<ec2_ip>`
(On the local machine) Connect by Java Mission Control to localhost:1099
The answer given by Gray worked for me, however I find that I have to open TCP ports 0 to 65535 or I don't get in. I think that you can connect on the main JMX port, and then get another one assigned. I got that from this blog post that has always worked well for me.
We are using AWS Elastic Container Service for running our spring boot services.
The below config allowed us to connect to our docker containers.
Without Authentication:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9090 \
-Dcom.sun.management.jmxremote.rmi.port=9090 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=$(/usr/bin/curl -s --connect-timeout 2 \
http://169.254.169.254/latest/meta-data/public-ipv4)
I found it crisp and also doesn't require any other servicer side init script.