Develop Reference

unable to connect to jmx port remotely

I'm trying to connect to a jmx port remotely but I can't seem to connect to it even though the port is open. Its a java process running in a container on a server thats a Nomad worker. Its running on 29406.
Here is what netstat shows:
netstat -tulpn | grep 29406
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 10.137.2.166:29406 0.0.0.0:* LISTEN -
udp 0 0 10.137.2.166:29406 0.0.0.0:* -
And this is whats in /etc/hosts
cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
I've downloaded jmxterm on the server to try and connect to it, and noticed an interesting behavior. When I try using localhost to connect to the port, I get this:
#RuntimeIOException: Runtime IO exception: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
When I use its own IP address, it then seems to work:
$>open 10.137.2.166:29406
#Connection to 10.137.2.166:29406 is opened
$>
Curious to understand why localhost doesn't work when I'm running this on the server itself...
The only way I've gotten jconsole (running on my laptop) to connect to it is by using an ssh tunnel like this:
ssh -Nf -D 7777 10.137.2.166
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 service:jmx:rmi:///jndi/rmi://10.137.2.166:29406/jmxrmi -J-DsocksNonProxyHosts=
I feel like I should be able to connect to it without creating a tunnel but unsure why I can't. If I run telnet locally from my laptop to the host, the connection does seem to open...
telnet 10.137.2.166 29406
Trying 10.137.2.166...
Connected to 10.137.2.166.
Escape character is '^]'.

To successful JMX handshake
the jmx server should be available by a host name outside (should also be declared on server jvm via java.rmi.server.hostname system property)
in addition to one open port (can be explicitly declared via com.sun.management.jmxremote.rmi.port jvm property) the jmx server chooses random another that's used for new jmx connection. It's quite problematic because you can't foresee particular port in order to exclude it from server's firewall restrictions, so the tunneling is necessary.

Server listened at only 10.137.2.166.
When you trying to create new socket with localhost domain, your application tying to establish 127.0.0.1 adress but your application not listening at this ip.
If you want to connect with localhost domain you have few options for solving.
Change your server configuration to listen on 127.0.0.1 and 10.137.2.166 at same time.
Change your server configuration to listen on 0.0.0.0 .
Listening at 0.0.0.0 its not recommended for security reasons .
Use iptables to forward port. Requires root privileges.
sysctl net.ipv4.ip_forward=1
iptables -t nat -A PREROUTING -p tcp -i lo --dport 29406 -j DNAT --to-destination 10.137.2.166:29406
iptables -A FORWARD -p tcp -d 10.137.2.166 --dport 29406 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
if you don't have root privileges you can use socat.
socat TCP-LISTEN:29406,fork,bind=127.0.0.1 TCP:10.137.2.166:29406

I only used jmx for visualvm connection and in this case they are two ports required to be available:
com.sun.management.jmxremote.port=9010
com.sun.management.jmxremote.rmi.port=9011
Also the java.rmi.server.hostname need to be set accordingly to the right network interface as the port will be bound only on that interface.
Once the ports are available from your client, you can use the jmx connection on the jmxremote.port port.

Unable to debug app remotely - port isn't accessible externally

I need to debug my app remotely, but I unable to do that, due to the following error:
Unable to open debugger port (X.X.X.X:8000): java.net.ConnectException "Connection timed out: connect"
I have:
Java app running in Tomcat on a remote server (Debian)
IntelliJ Idea running locally (Windows)
I came to the conclusion that cause of problem is that port 8000, used for remote debugging, is inaccessible via external ip, only via 'localhost'. Following are causes, why I think so:
A remote debug is working on an expected port:
root#victor-app-server:/opt/tomcat-home/bin# netstat -tulpn | grep 8000
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 3773/java
I able to telnet to it via 'localhost':
root#victor-app-server:/opt/tomcat-home/bin# telnet localhost 8000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
But unable to telnet via external ip:
root#victor-app-server:/opt/tomcat-home/bin# telnet X.X.X.X 8000
Trying X.X.X.X...
telnet: Unable to connect to remote host: Connection timed out
Here is output of iptables:
root#victor-app-server:/opt/tomcat-home/bin# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
How can I fix this issue with 8000 port? Or maybe there is other cause to my main problem?

This is the command that did the trick for me:
sudo ufw allow <debug_port>

I had a same problem via Java 11 and address property such as these JVM arguments:
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=6565
However my problem is solved by change value of address property like this sample:
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=*:6565
Now 6565 port is accessible externally.

Try "iptables -F" which unblocks the firewalls

How to profile remote ubuntu JVM using VisualVM?

I am trying to profile remote JVM using VisualVM. I have a remote production ubuntu machine on which my Java application is running and that's what I need to profile. I was following this tutorial to profile a remote server.
I started jstatd on my ubuntu production machine like this -
root#productionMachineA:/home/david# /usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/jstatd -J-Djava.security.policy=permissions.txt -J-Djava.rmi.server.hostname=100.41.76.19 -J-Djava.rmi.server.logCalls=true -J-Djava.net.preferIPv4Stack=true
Here 100.41.76.19 is the IP Address of my production ubuntu machine. After starting jstatd on the ubunut machine, I did -
netstat -nlp | grep jstatd
And I can see this -
root#productionMachineA:~$ netstat -nlp | grep jstatd
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN 32103/jstatd
tcp 0 0 0.0.0.0:60707 0.0.0.0:* LISTEN 32103/jstatd
which looks to me jstatd is running fine I guess. Now I opened VisualVM on my desktop, right click on Remote and select Add Remote Host, and finally type the IP address of the production machineA. And afterwards I don't see anything happening on VisualVM which makes me think something is wrong for sure.
Can anyone tell me what's wrong and what are the things I should try on? If anyone can provide steps by steps what I am supposed to do then it will be of great help.
Update:-
After adding port 1099 on my remote connection.
I got this error. Cannot connect to 100.41.76.19 using service:jmx:rmi.....
From my local desktop, I tried telnet on remote machine on port 1099 and this is what I got -
david#localDesktop ~
$ telnet 100.41.76.19 1099
david#localDesktop ~
$

Netty server remote connection

Netty server, Fedora. I just can't connect to the server from remote host and no listening socket is displayed via netstat util. However I can establish the connection running client and server on the same machine. That's simply like that:
port = System.getProperty(PORT_PROPERTY);
Preconditions.checkNotNull(port, "Network error, port property is not set");
hostAddress = new InetSocketAddress(Integer.valueOf(port));
...
serverChannel = bootstrap.bind(hostAddress);
I've tried initializing hostAddress with the port only, localhost IP, 0.0.0.0 IP, and IP of my network. Nothing helps. What could be the root of problem?

Here's some suggestions that should help disagnosing the problem:
For clarity (until you resolve this), stick to using
new InetSocketAddress("0.0.0.0", Integer.valueOf(port))
since this will ensure you bind to all interfaces.
Invoke the JVM with -Djava.net.preferIPv4Stack=true to force the JVM into IPV4. I have found it easier to muck with these issues when in IPV4 since is it less complicated than V6.
Get the PID of the JVM and then issue a netstat like this:
sudo netstat -ap --numeric-ports | grep <PID>
This should display all sockets for your JVM instance. (Please post this output if you're still not able to connect remotely. Also post the output of ifconfig)

Unable to connect to a remote JVM

I'm working on a Java 10 application that uses an embedded Jetty server to provide control from a local network, and I'm attempting to connect to the JVM and failing. It's running on Ubuntu 18.04 LTS desktop.
My startup script has the following lines:
java -Xdebug -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n -Djava.library_path=${LIB_PATH} -classpath ${CP} -jar ${APP_DIR}/app.jar
I have ufw on the system, and I've verified that the port is open. My output from ufw status includes:
8000 ALLOW Anywhere
8000 (v6) ALLOW Anywhere
In IntelliJ, my debug configuration is
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8000
When I try to connect, I get an error message that the connection is refused.
with the IP of the machine in the config's address box.
Looking at the output of netstat -l, I see the following:
tcp 0 0 localhost:8000 0.0.0.0:* LISTEN
Does this mean that the debugger is only listening for connection on the localhost? Do I need to do something to have it listen on a network?

So I found the answer fairly quick. I needed to modify the line I use for the server so that it reads:
java -Xdebug -agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n -Djava.library_path=${LIB_PATH} -classpath ${CP} -jar ${APP_DIR}/app.jar
So that is listens on all interfaces.