I'm trying to connect VisualVM to a remote JMX. IntelliJ can connect without hassle, but VisualVM fails with the following in log output:
Listening for transport dt_socket at address: 5005
Debugger failed to attach: recv failed during handshake: Resource temporarily unavailable
Debugger failed to attach: handshake failed - received >< - expected >JDWP-Handshake<
Here are the JVM properties I'm passing at runtime:
-XX:MaxPermSize=50g -XX:+HeapDumpOnOutOfMemoryError -Xmx50g -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=100.100.100.100 -Dcom.sun.management.jmxremote.port=5006 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
Using VisualVM 1.3.9. Not sure why it fails why IntelliJ succeeds. I've tried disabling SSL auth, etc.
The socket listening on 5005 is not a JMX listener, it's the java debug wire protocol listener. As such, VisualVM will not be able to connect to that port, and that's why you're seeing a handshake failure.
Related
My application is running in EC2 as a docker with java application.
I'm exposing 5005 port for debug, and locally it works perfectly. However on EC2 environment I get
java.net.ConnectException "Connection refused (Connection refused)"
when trying to connect using Intelij.
Security group is set to open ports 80, 5005, 22
Docker is exposing port 80 and 5005
Application is running with java args
-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=5005,suspend=n
Am I missing something ?
For those who are still interested, here is a way how to create a Remote JVM Debug on EC2 with docker
On the yaml file add the 'port' attribute.
ports:
- "5005:5005"
To the dockerfile run the Jar with the following command
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
On the inbound role on the EC2:
Cutsom TCP => 5005 => Your IP
In the Intellij create Remote JVM debug
In the host enter the host ip
Port: 5005
Choose JDK 9 or later as the address should be with *:5005
Click the Debug Button and it should work
asadmin start-domain --debug=true
JDK: jdk1.8.0_172
Payara: 4.1.2.181
produces the following output.
Waiting for domain1 to start ..Error starting domain domain1.
The server exited prematurely with exit code 2.
Before it died, it produced the following output:
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=192m; support was removed in 8.0
ERROR: transport error 202: connect failed: Connection refused
ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [debugInit.c:750]
Command start-domain failed.
The glassfish log file only contains this:
There are no additional clues as to why it cannot start in debug mode.
asadmin start-domain works just perfectly.
[2018-05-15T14:43:27.424+0200] [] [INFO] [NCLS-GFLAUNCHER-00005] [javax.enterprise.launcher] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1526388207424] [levelValue: 800] [[
JVM invocation command line:
C:\Program Files\Java\jdk1.8.0_172\bin\java.exe
-cp
C:/Java/payara41/glassfish/modules/glassfish.jar
-agentlib:jdwp=transport=dt_socket,address=9009,server=n,suspend=y
-XX:+UnlockDiagnosticVMOptions
-XX:NewRatio=2
-XX:MaxPermSize=192m
-Xmx512m
-javaagent:C:/Java/payara41/glassfish/lib/monitor/flashlight-agent.jar
-client
-Djavax.xml.accessExternalSchema=all
-Djavax.net.ssl.trustStore=C:\Java\payara41\glassfish\domains\domain1/config/cacerts.jks
-Djdk.tls.rejectClientInitiatedRenegotiation=true
-Djdk.corba.allowOutputStreamSubclass=true
-Dfelix.fileinstall.dir=C:\Java\payara41\glassfish/modules/autostart/
-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall
-Dcom.sun.aas.installRoot=C:\Java\payara41\glassfish
-Dfelix.fileinstall.poll=5000
-Djava.endorsed.dirs=C:\Java\payara41\glassfish/modules/endorsed;C:\Java\payara41\glassfish/lib/endorsed
-Djava.security.policy=C:\Java\payara41\glassfish\domains\domain1/config/server.policy
-Dosgi.shell.telnet.maxconn=1
-Dfelix.fileinstall.bundles.startTransient=true
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dfelix.fileinstall.log.level=2
-Djavax.net.ssl.keyStore=C:\Java\payara41\glassfish\domains\domain1/config/keystore.jks
-Djava.security.auth.login.config=C:\Java\payara41\glassfish\domains\domain1/config/login.conf
-Dfelix.fileinstall.disableConfigSave=false
-Dorg.glassfish.grizzly.DEFAULT_MEMORY_MANAGER=org.glassfish.grizzly.memory.HeapMemoryManager
-Dfelix.fileinstall.bundles.new.start=true
-Dcom.sun.aas.instanceRoot=C:\Java\payara41\glassfish\domains\domain1
-Dosgi.shell.telnet.port=6666
-Dgosh.args=--nointeractive
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Dorg.jboss.weld.serialization.beanIdentifierIndexOptimization=false
-Dosgi.shell.telnet.ip=127.0.0.1
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Djava.awt.headless=true
-Dcom.ctc.wstx.returnNullForDefaultNamespace=true
-Djava.ext.dirs=C:\Program Files\Java\jdk1.8.0_172/lib/ext;C:\Program Files\Java\jdk1.8.0_172/jre/lib/ext;C:\Java\payara41\glassfish\domains\domain1/lib/ext
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Djava.library.path=C:/Java/payara41/glassfish/lib;C:/Program Files (x86)/Common Files/Oracle/Java/javapath;C:/Windows/Sun/Java/bin;C:/Windows/System32;C:/Windows;C:/Utilities;C:/Program Files/Common Files/microsoft shared/Microsoft Online Services;C:/Program Files (x86)/Common Files/microsoft shared/Microsoft Online Services;C:/ProgramData/Oracle/Java/javapath;C:/Windows/System32/wbem;C:/Windows/System32/WindowsPowerShell/v1.0;C:/Program Files/Intel/WiFi/bin;C:/Program Files/Common Files/Intel/WirelessCommon;C:/ProgramData/chocolatey/bin;C:/Program Files/Beyond Compare 4;C:/Users/QXV0615/AppData/Local/Microsoft/WindowsApps;C:/Program Files/PortableGit/cmd;C:/Users/QXV0615/AppData/Local/hyper/app-2.0.0/resources/bin;C:/Program Files/Microsoft VS Code;C:/Java/payara41/bin;C:/Program Files/Java/jdk1.8.0_172/bin;C:/Java/apache-maven-3.5.3/bin;C:/Users/QXV0615/AppData/Local/Programs/Git/cmd;C:/Bin;C:/Program Files/PostgreSQL/10/bin;C:/Projects/mcp/mcp
com.sun.enterprise.glassfish.bootstrap.ASMain
-upgrade
false
-domaindir
C:/Java/payara41/glassfish/domains/domain1
-read-stdin
true
-asadmin-args
--host,,,localhost,,,--port,,,4848,,,--secure=false,,,--terse=false,,,--echo=false,,,--interactive=true,,,start-domain,,,--verbose=false,,,--watchdog=false,,,--debug=true,,,--domaindir,,,C:\Java\payara41\glassfish\domains,,,domain1
-domainname
domain1
-instancename
server
-type
DAS
-verbose
false
-asadmin-classpath
C:/Java/payara41/glassfish/lib/client/appserver-cli.jar
-debug
true
-asadmin-classname
com.sun.enterprise.admin.cli.AdminMain
-watchdog
false]]
any tips on how I should resolve this?
edit
I have discovered that the domain.xml configuration files for debug startup were different between my version 172 and 181 are different.
174:
<java-config classpath-suffix="" debug-options="-agentlib:jdwp=transport=dt_socket,address=9009,server=n,suspend=y" java-home="C:\Program Files\Java\jdk1.8.0_172" system-classpath="">
181:
<java-config classpath-suffix="" debug-options="-agentlib:jdwp=transport=dt_socket,address=9009,server=n,suspend=y" java-home="C:\Program Files\Java\jdk1.8.0_172" system-classpath="">
The server, suspend and java-home are different. The JDK path in the configuration file is correct.
They now both start, but why?
That looks like the ports used by Payara are already in use. The most likely reason for that is you already have a running instance of Payara which you haven't stopped.
You check what ports are in use by running netstat -ano.
Also, ensure that the startup options in domain.xml are correct. Some IDE's like Intelij Idea might change the options for local debugging - which break remote debugging. The options need to be restored.
I am trying to run GlassFish4 on Windows 7. At glassfish4\bin I run asadmin start-domain -d and I get the following error message:
Waiting for domain 1 to start. Error starting domain domain1. The
server exited prematurely with exit code 1.
Before it died, it produced the following output:
Listening for transport dt_socket at address: 9009 Error: could not
find or load main class files.
Command start-domain failed
I checked the PATH and CLASSPATH and things appear to be ok but obviously something is wrong here.
Can you try starting the domain without the -d flag? I think that is trying to it in debug mode. Port 9009 is not a standard GlassFish port but is for JPDA debugging, which also uses dt_socket as a transport.
I started a jstatd on the remote server (Ubuntu Server 14.04):
jstatd -J-Djava.security.policy=.jstatd.all.policy -J-Djava.rmi.server.logCalltrue -p 9099
and try to connect to it with jvisualvm on windows. I checked netstat, the connection is established, and on the remote it logs the call:
Sep 11, 2015 12:48:51 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(4)-10.82.199.0: [10.82.199.0: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Sep 11, 2015 12:48:55 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(4)-10.82.199.0: [10.82.199.0: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Sep 11, 2015 12:48:59 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(4)-10.82.199.0: [10.82.199.0: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
All signs are saying that it's working. but however no applications is showing in jvisualvm:
Apparently VisualVM expects a consistent DNS name for the server you're trying to connect to remotely (the Ubuntu Server 14.04 in your case). Hence, if you're specifying an IP address instead of a DNS name to VisualVM you should add the following to your jstatd startup line:
-J-Djava.rmi.server.hostname=<the IP address to your Ubuntu server here>
Additionally, I found out that specifying the port option (-p 9099 in your case) is not supported in some VisualVM releases:
Known limitation: In this VisualVM release the jstatd's default port and rminame must be used when starting the jstatd utility, i.e. the use of the -p and -n options is not supported.
VisualVM Troubleshooting Guide
All in all, you should try running the following jstatd line on your Ubuntu Server:
jstatd -J-Djava.security.policy=.jstatd.all.policy -J-Djava.rmi.server.hostname=10.82.83.117 -J-Djava.rmi.server.logCalltrue
Sources:
http://www.catify.com/2012/09/26/remote-monitoring-with-visualvm/
It worked for me :)
jstatd -p 1099 -J-Djava.rmi.sver.hostname=10.250.105.112 -J-Djava.security.policy=<(echo 'grant codebase "file:${java.home}/../lib/tools.jar" {permission java.security.AllPermission;};')
Works for Me Perfectly
In case this helps someone else...
I was running into problems where neither jstatd nor adding a plain JMX connection in VisualVM worked. The former would not give any error messages, it just wouldn't list any apps. The latter would give me an error saying "Cannot connect to some-server:30648 using service:jmx:rmi:///jndi/rmi://some-server:30648/jmxrmi.
Trying to use the excellent sjk-plus tool to manually connect to the JMX service gave the following error:
$ java --add-opens java.base/jdk.internal.perf=ALL-UNNAMED \
--add-opens jdk.attach/sun.tools.attach=ALL-UNNAMED \
-Dsjk.breakCage=false \
-jar scripts/sjk-plus-0.14.jar
mx --get --allMatched -b com.acme.some.package:name=* -f Count \
-s some-server:30648
JMX Connection failed: java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
Do you see it? 127.0.1.1, what is that weird IP address doing there?
This was caused by a particular entry in the /etc/hosts file on the server:
user#some-server:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 some-server
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Changing the some-server entry in the hosts file and restarting the process made it work with sjk-plus, and made it discoverable with jstatd as well.
in java 11+
jstatd -J-Djava.rmi.server.logCalls=true \
-J-Djava.security.policy=.jstatd.all.policy \
-J-Djava.net.preferIPv4Stack=true \
-J-Djava.security.policy=<(echo 'grant codebase "jrt:/jdk.jstatd" {permission java.security.AllPermission;}; grant codebase "jrt:/jdk.internal.jvmstat" {permission java.security.AllPermission;};')
I'm trying query a remote JVM with jps using jstatd, in order to eventually monitor it using VisualVM.
I got jstatd running with the following security policy:
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
jstatd is running on a 64-bit Linux box with a 1.6.0_10 version HotSpot vm. The jstatd command is:
jstatd -J-Djava.security.policy=jstatd.tools.policy -J-Djava.rmi.server.logCalls=true
I'm trying to run jps from a Windows 7 machine. Due to firewall restrictions, I'm tunneling the RMI data through an SSH tunnel to my Windows machine such that the jps command line is:
.\jps.exe -m -l rmi://localhost
When I run jps, I see the connection attempt in the jstatd log, which looks like this:
Feb 1, 2011 11:50:34 AM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-127.0.0.1: [127.0.0.1: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(ja va.lang.String)]
but on the jps side I get the following error:
Error communicating with remote host: Connection refused to host: 192.168.1.137; nested exception is:
java.net.ConnectException: Connection refused: connect
Based on the connection attempt listed in the jstatd log, I think jps is actually reaching the host, but for some reason is getting blocked. Is there some security policy I have set or some other setting somewhere I can change so that I can get jps to pull stats from the remote jstatd?
My guess is that you're only forwarding the RMI registry port (1099), but you need to also open another port.
Check which ports on the remote side
# netstat -nap | grep jstatd
tcp 0 0 :::1099 :::* LISTEN 453/jstatd
tcp 0 0 :::58204 :::* LISTEN 453/jstatd
In this case you will need to forward port 58204 as well as 1099
Here is how you could easily do this.
Launch ejstatd in your remote host this way (executing from the ejstatd folder): mvn exec:java -Dexec.args="-pr 2000 -ph 2001 -pv 2002"
Open those 3 ports on your remote host and make them available to your local machine: 2000, 2001 and 2002
On your local machine, you will be able to use jps replacing <remotehost> with your remote host name: jps -m -l rmi://<remotehost>:2000
Disclaimer: I'm the author of the open source ejstatd tool