I'm trying query a remote JVM with jps using jstatd, in order to eventually monitor it using VisualVM.
I got jstatd running with the following security policy:
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
jstatd is running on a 64-bit Linux box with a 1.6.0_10 version HotSpot vm. The jstatd command is:
jstatd -J-Djava.security.policy=jstatd.tools.policy -J-Djava.rmi.server.logCalls=true
I'm trying to run jps from a Windows 7 machine. Due to firewall restrictions, I'm tunneling the RMI data through an SSH tunnel to my Windows machine such that the jps command line is:
.\jps.exe -m -l rmi://localhost
When I run jps, I see the connection attempt in the jstatd log, which looks like this:
Feb 1, 2011 11:50:34 AM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-127.0.0.1: [127.0.0.1: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(ja va.lang.String)]
but on the jps side I get the following error:
Error communicating with remote host: Connection refused to host: 192.168.1.137; nested exception is:
java.net.ConnectException: Connection refused: connect
Based on the connection attempt listed in the jstatd log, I think jps is actually reaching the host, but for some reason is getting blocked. Is there some security policy I have set or some other setting somewhere I can change so that I can get jps to pull stats from the remote jstatd?
My guess is that you're only forwarding the RMI registry port (1099), but you need to also open another port.
Check which ports on the remote side
# netstat -nap | grep jstatd
tcp 0 0 :::1099 :::* LISTEN 453/jstatd
tcp 0 0 :::58204 :::* LISTEN 453/jstatd
In this case you will need to forward port 58204 as well as 1099
Here is how you could easily do this.
Launch ejstatd in your remote host this way (executing from the ejstatd folder): mvn exec:java -Dexec.args="-pr 2000 -ph 2001 -pv 2002"
Open those 3 ports on your remote host and make them available to your local machine: 2000, 2001 and 2002
On your local machine, you will be able to use jps replacing <remotehost> with your remote host name: jps -m -l rmi://<remotehost>:2000
Disclaimer: I'm the author of the open source ejstatd tool
Related
My application is running in EC2 as a docker with java application.
I'm exposing 5005 port for debug, and locally it works perfectly. However on EC2 environment I get
java.net.ConnectException "Connection refused (Connection refused)"
when trying to connect using Intelij.
Security group is set to open ports 80, 5005, 22
Docker is exposing port 80 and 5005
Application is running with java args
-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=5005,suspend=n
Am I missing something ?
For those who are still interested, here is a way how to create a Remote JVM Debug on EC2 with docker
On the yaml file add the 'port' attribute.
ports:
- "5005:5005"
To the dockerfile run the Jar with the following command
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
On the inbound role on the EC2:
Cutsom TCP => 5005 => Your IP
In the Intellij create Remote JVM debug
In the host enter the host ip
Port: 5005
Choose JDK 9 or later as the address should be with *:5005
Click the Debug Button and it should work
I am trying to run GlassFish4 on Windows 7. At glassfish4\bin I run asadmin start-domain -d and I get the following error message:
Waiting for domain 1 to start. Error starting domain domain1. The
server exited prematurely with exit code 1.
Before it died, it produced the following output:
Listening for transport dt_socket at address: 9009 Error: could not
find or load main class files.
Command start-domain failed
I checked the PATH and CLASSPATH and things appear to be ok but obviously something is wrong here.
Can you try starting the domain without the -d flag? I think that is trying to it in debug mode. Port 9009 is not a standard GlassFish port but is for JPDA debugging, which also uses dt_socket as a transport.
I started a jstatd on the remote server (Ubuntu Server 14.04):
jstatd -J-Djava.security.policy=.jstatd.all.policy -J-Djava.rmi.server.logCalltrue -p 9099
and try to connect to it with jvisualvm on windows. I checked netstat, the connection is established, and on the remote it logs the call:
Sep 11, 2015 12:48:51 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(4)-10.82.199.0: [10.82.199.0: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Sep 11, 2015 12:48:55 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(4)-10.82.199.0: [10.82.199.0: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Sep 11, 2015 12:48:59 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(4)-10.82.199.0: [10.82.199.0: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
All signs are saying that it's working. but however no applications is showing in jvisualvm:
Apparently VisualVM expects a consistent DNS name for the server you're trying to connect to remotely (the Ubuntu Server 14.04 in your case). Hence, if you're specifying an IP address instead of a DNS name to VisualVM you should add the following to your jstatd startup line:
-J-Djava.rmi.server.hostname=<the IP address to your Ubuntu server here>
Additionally, I found out that specifying the port option (-p 9099 in your case) is not supported in some VisualVM releases:
Known limitation: In this VisualVM release the jstatd's default port and rminame must be used when starting the jstatd utility, i.e. the use of the -p and -n options is not supported.
VisualVM Troubleshooting Guide
All in all, you should try running the following jstatd line on your Ubuntu Server:
jstatd -J-Djava.security.policy=.jstatd.all.policy -J-Djava.rmi.server.hostname=10.82.83.117 -J-Djava.rmi.server.logCalltrue
Sources:
http://www.catify.com/2012/09/26/remote-monitoring-with-visualvm/
It worked for me :)
jstatd -p 1099 -J-Djava.rmi.sver.hostname=10.250.105.112 -J-Djava.security.policy=<(echo 'grant codebase "file:${java.home}/../lib/tools.jar" {permission java.security.AllPermission;};')
Works for Me Perfectly
In case this helps someone else...
I was running into problems where neither jstatd nor adding a plain JMX connection in VisualVM worked. The former would not give any error messages, it just wouldn't list any apps. The latter would give me an error saying "Cannot connect to some-server:30648 using service:jmx:rmi:///jndi/rmi://some-server:30648/jmxrmi.
Trying to use the excellent sjk-plus tool to manually connect to the JMX service gave the following error:
$ java --add-opens java.base/jdk.internal.perf=ALL-UNNAMED \
--add-opens jdk.attach/sun.tools.attach=ALL-UNNAMED \
-Dsjk.breakCage=false \
-jar scripts/sjk-plus-0.14.jar
mx --get --allMatched -b com.acme.some.package:name=* -f Count \
-s some-server:30648
JMX Connection failed: java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
Do you see it? 127.0.1.1, what is that weird IP address doing there?
This was caused by a particular entry in the /etc/hosts file on the server:
user#some-server:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 some-server
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Changing the some-server entry in the hosts file and restarting the process made it work with sjk-plus, and made it discoverable with jstatd as well.
in java 11+
jstatd -J-Djava.rmi.server.logCalls=true \
-J-Djava.security.policy=.jstatd.all.policy \
-J-Djava.net.preferIPv4Stack=true \
-J-Djava.security.policy=<(echo 'grant codebase "jrt:/jdk.jstatd" {permission java.security.AllPermission;}; grant codebase "jrt:/jdk.internal.jvmstat" {permission java.security.AllPermission;};')
I trying to run an UPnP service on my docker container using the Cling UPNP library (http://4thline.org/projects/cling/). There is a simple program that creates a device (in software) that hosts some service. This is written in Java and when I try to run the program I get the following exception (Note: This runs perfectly fine directly on my Ubuntu machine):
Jun 5, 2015 1:47:24 AM org.teleal.cling.UpnpServiceImpl <init>
INFO: >>> Starting UPnP service...
Jun 5, 2015 1:47:24 AM org.teleal.cling.UpnpServiceImpl <init>
INFO: Using configuration: org.teleal.cling.DefaultUpnpServiceConfiguration
Jun 5, 2015 1:47:24 AM org.teleal.cling.transport.RouterImpl <init>
INFO: Creating Router: org.teleal.cling.transport.RouterImpl
Exception occured: org.teleal.cling.transport.spi.InitializationException: Could not discover any bindable network interfaces and/or addresses
org.teleal.cling.transport.spi.InitializationException: **Could not discover any bindable network interfaces and/or addresses
at org.teleal.cling.transport.impl.NetworkAddressFactoryImpl.<init>(NetworkAddressFactoryImpl.java:99)**
For anyone that finds this and needs the answer.
Your container is obscuring your external network. In other words, by default containers are isolated and cannot see the outer network which is of course required in order to open the ports in your IGD.
You can run your container as a "host" to make it non isolated, simply add --network host to your container creation command.
Example (taken from https://docs.docker.com/network/network-tutorial-host/#procedure):
docker run --rm -d --network host --name my_nginx nginx
I have tested this using docker-compose.yml which looks a bit different:
version: "3.4"
services:
upnp:
container_name: upnp
restart: on-failure:10
network_mode: host # works while the container runs
build:
context: .
network: host # works during the build if needed
version 3.4 is very important and the network: host will not work otherwise!
My upnp container Dockerfile looks like so:
FROM alpine:latest
RUN apk update
RUN apk add bash
RUN apk add miniupnpc
RUN mkdir /scripts
WORKDIR /scripts
COPY update_upnp .
RUN chmod a+x ./update_upnp
# cron to update each UPnP entries every 10 minutes
RUN echo -e "*/10\t*\t*\t*\t*\tbash /scripts/update_upnp 8080 TCP" >> /etc/crontabs/root
CMD ["crond", "-f"]
# on start, open needed ports
ENTRYPOINT bash update_upnp 80 TCP && bash update_upnp 8080 TCP
update_upnp script is simply using upnpc (installed as miniupnpc in the Dockerfile above) to open the ports.
Hopefully this will help somebody.
Edit: Here is how update_upnp script may look like:
#!/bin/bash
port=$1
protocol=$2
echo "Updating UPnP entry with port [$port] and protocol [$protocol]"
gateway=$(ip route | head -1 | awk '{print $3}')
echo "Detected gateway is [$gateway]"
# port - e.g. 80
# protocol - TCP or UDP
upnpc -e 'your-app-name' -r $port $protocol
echo "Done updating UPnP entry with port [$port] and protocol [$protocol]"
GlassFish 3.0.1
RHEL 6.3
I changed the http-listener-2 port via the GlassFish Admin web page from 8181 to 443 and now GlassFish will not start because of a port conflict on port 443.
0500|SEVERE|glassfish3.0.1|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=11;_ThreadName=FelixStartLevel;|Shutting down v3 due to startup exception : Address already in use: 443=com.sun.enterprise.v3.services.impl.monitor.MonitorableSSLSelectorHandler#29ed3da1|#]
The process using port while GlassFish is starting (and only when GlassFish is starting) is GlassFish (or the JVM running GlassFish)
root> ls -l /proc/3242/exe
lrwxrwxrwx 1 root root 0 Sep 17 17:19 /proc/3242/exe -> /usr/java/jdk1.6.0_34/bin/java
Quite a mess. Now how do I get http-listener-2 back on to port 8181 and off port 443?
There must be a file (probably an XML file) which stores this information.
I can't use any asadmin commands because GlassFish is not up and running. Isn't there an 'unset' command I could use?
The cause was that there existed 2 http-listeners on the same port (443). Removed the extra/duplicate http-listener from the GlassFish config file domain.xml and rebooted Glassfish successfully.
Asadmin is used to start the server, but asadmin is not dependent on the server started.
Check the documentation on how to use asadmin commands.
On changing the port no, check this link out.