Develop Reference

unable to connect to jmx port remotely

I'm trying to connect to a jmx port remotely but I can't seem to connect to it even though the port is open. Its a java process running in a container on a server thats a Nomad worker. Its running on 29406.
Here is what netstat shows:
netstat -tulpn | grep 29406
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 10.137.2.166:29406 0.0.0.0:* LISTEN -
udp 0 0 10.137.2.166:29406 0.0.0.0:* -
And this is whats in /etc/hosts
cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
I've downloaded jmxterm on the server to try and connect to it, and noticed an interesting behavior. When I try using localhost to connect to the port, I get this:
#RuntimeIOException: Runtime IO exception: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
When I use its own IP address, it then seems to work:
$>open 10.137.2.166:29406
#Connection to 10.137.2.166:29406 is opened
$>
Curious to understand why localhost doesn't work when I'm running this on the server itself...
The only way I've gotten jconsole (running on my laptop) to connect to it is by using an ssh tunnel like this:
ssh -Nf -D 7777 10.137.2.166
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 service:jmx:rmi:///jndi/rmi://10.137.2.166:29406/jmxrmi -J-DsocksNonProxyHosts=
I feel like I should be able to connect to it without creating a tunnel but unsure why I can't. If I run telnet locally from my laptop to the host, the connection does seem to open...
telnet 10.137.2.166 29406
Trying 10.137.2.166...
Connected to 10.137.2.166.
Escape character is '^]'.

To successful JMX handshake
the jmx server should be available by a host name outside (should also be declared on server jvm via java.rmi.server.hostname system property)
in addition to one open port (can be explicitly declared via com.sun.management.jmxremote.rmi.port jvm property) the jmx server chooses random another that's used for new jmx connection. It's quite problematic because you can't foresee particular port in order to exclude it from server's firewall restrictions, so the tunneling is necessary.

Server listened at only 10.137.2.166.
When you trying to create new socket with localhost domain, your application tying to establish 127.0.0.1 adress but your application not listening at this ip.
If you want to connect with localhost domain you have few options for solving.
Change your server configuration to listen on 127.0.0.1 and 10.137.2.166 at same time.
Change your server configuration to listen on 0.0.0.0 .
Listening at 0.0.0.0 its not recommended for security reasons .
Use iptables to forward port. Requires root privileges.
sysctl net.ipv4.ip_forward=1
iptables -t nat -A PREROUTING -p tcp -i lo --dport 29406 -j DNAT --to-destination 10.137.2.166:29406
iptables -A FORWARD -p tcp -d 10.137.2.166 --dport 29406 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
if you don't have root privileges you can use socat.
socat TCP-LISTEN:29406,fork,bind=127.0.0.1 TCP:10.137.2.166:29406

I only used jmx for visualvm connection and in this case they are two ports required to be available:
com.sun.management.jmxremote.port=9010
com.sun.management.jmxremote.rmi.port=9011
Also the java.rmi.server.hostname need to be set accordingly to the right network interface as the port will be bound only on that interface.
Once the ports are available from your client, you can use the jmx connection on the jmxremote.port port.

How to profile remote ubuntu JVM using VisualVM?

I am trying to profile remote JVM using VisualVM. I have a remote production ubuntu machine on which my Java application is running and that's what I need to profile. I was following this tutorial to profile a remote server.
I started jstatd on my ubuntu production machine like this -
root#productionMachineA:/home/david# /usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/jstatd -J-Djava.security.policy=permissions.txt -J-Djava.rmi.server.hostname=100.41.76.19 -J-Djava.rmi.server.logCalls=true -J-Djava.net.preferIPv4Stack=true
Here 100.41.76.19 is the IP Address of my production ubuntu machine. After starting jstatd on the ubunut machine, I did -
netstat -nlp | grep jstatd
And I can see this -
root#productionMachineA:~$ netstat -nlp | grep jstatd
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN 32103/jstatd
tcp 0 0 0.0.0.0:60707 0.0.0.0:* LISTEN 32103/jstatd
which looks to me jstatd is running fine I guess. Now I opened VisualVM on my desktop, right click on Remote and select Add Remote Host, and finally type the IP address of the production machineA. And afterwards I don't see anything happening on VisualVM which makes me think something is wrong for sure.
Can anyone tell me what's wrong and what are the things I should try on? If anyone can provide steps by steps what I am supposed to do then it will be of great help.
Update:-
After adding port 1099 on my remote connection.
I got this error. Cannot connect to 100.41.76.19 using service:jmx:rmi.....
From my local desktop, I tried telnet on remote machine on port 1099 and this is what I got -
david#localDesktop ~
$ telnet 100.41.76.19 1099
david#localDesktop ~
$

tomcat7 redhat could not start up

I hava installed on RedHatEnterpriseLinux java1.7 and apache tomcat 7, starting tomcat on port 8080 and checking through
netstat -nap | grep 8080
and getting
tcp 0 0 :::8080 :::* LISTEN 29178/java
but when i try to open tomcat home page from browser, browser is loading and loading and nothing appears. and when I try to stop tomcat server, I am getting
java.net.ConnectionException: Connection Refused error.
How can it be solved?

The Tomcat server may have bound to port 8080, but was unable to complete startup for some reason, and so not is not servicing requests yet. Check the tomcat logs for errors, and make sure that the server has started successfully. Successful startup of tomcat is usually indicated by the log statement "Server startup in XXXX ms" in catalina.out.

The problem could be in iptables. Try to open port 8080:
# lokkit --port=8080:tcp

Glassfish error upon attempting to use in Netbeans

Good day
I am trying to study JAVA EE so I installed the Glassfish 3. But when I attempted to deploy my project on Netbeans 6.9. I get the following error:
SEVERE: Shutting down v3 due to startup exception : Address already in use: bind: 8080=com.sun.enterprise.v3.services.impl.monitor.MonitorableSelectorHandler#106433d
And the server won't start.
It seems like that the port 8080 is already in used.
I go to Control Panel -> Administrative Tools -> Services but I don't know which application to kill because the port is not indicated there. How can I know which application is currently running at port 8080 so I could kill it?
EDIT: As per your suggestions, I did the netstat -aon
The result is as follows:
TCP 0.0.0.0:3700 0.0.0.0:0 LISTENING 4724
TCP 0.0.0.0:4848 0.0.0.0:0 LISTENING 4724
TCP 0.0.0.0:8080 0.0.0.0:0 LISTENING 4724
TCP 0.0.0.0:8181 0.0.0.0:0 LISTENING 4724
TCP 0.0.0.0:8686 0.0.0.0:0 LISTENING 4724
Can I kill this? I don't know what this means -> "LISTENING".. Sorry I don't understand this result much..
Anyway I killed the application as suggested by #Jigar Joshi and it Worked!
Thank you all for your help!

Go to Command prompt
Type the following command
netstat -aon | findstr "8080"
ex : TCP 10.12.230.222:2049 10.12.240.69:8080 ESTABLISHED 3476
Get the process id from the last column and run the following command
tasklist | findstr "3476"
for example you might get like this
firefox.exe 3476 RDP-Tcp#5 0 168,668 K
go to task manager and kill the firefox or whatever running on 8080 and start the server.

error message tells that the port is already binded with some other process now as you are running on windows machine
&
go to Control Panel -> Administrative Tools -> Services but I don't know which application I want to kill because the port is not indicated there. How can I know which application is currently running at port 8080 so I could kill it?
goto command prompt
netstat -aon
it will show you something like
TCP 192.1.200.48:2053 24.43.246.60:443 ESTABLISHED 248
TCP 192.1.200.48:2055 24.43.246.60:443 ESTABLISHED 248
TCP 192.1.200.48:2126 213.146.189.201:12350 ESTABLISHED 1308
TCP 192.1.200.48:3918 192.1.200.2:8073 ESTABLISHED 1504
TCP 192.1.200.48:3975 192.1.200.11:49892 TIME_WAIT 0
TCP 192.1.200.48:3976 192.1.200.11:49892 TIME_WAIT 0
TCP 192.1.200.48:4039 209.85.153.100:80 ESTABLISHED 248
TCP 192.1.200.48:8080 209.85.153.100:80 ESTABLISHED 248
check which process has binded your port. here in above example its 248 now if you are sure that you need to kill that process fire
Linux:
kill -9 248
Windows:
taskkill /f /pid 248
it will kill that process

Java App servers typically run on port 8080. Look from Apache Tomcat or another java web server that you might have installed and started.

Unable to connect to a remote JVM

I'm working on a Java 10 application that uses an embedded Jetty server to provide control from a local network, and I'm attempting to connect to the JVM and failing. It's running on Ubuntu 18.04 LTS desktop.
My startup script has the following lines:
java -Xdebug -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n -Djava.library_path=${LIB_PATH} -classpath ${CP} -jar ${APP_DIR}/app.jar
I have ufw on the system, and I've verified that the port is open. My output from ufw status includes:
8000 ALLOW Anywhere
8000 (v6) ALLOW Anywhere
In IntelliJ, my debug configuration is
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8000
When I try to connect, I get an error message that the connection is refused.
with the IP of the machine in the config's address box.
Looking at the output of netstat -l, I see the following:
tcp 0 0 localhost:8000 0.0.0.0:* LISTEN
Does this mean that the debugger is only listening for connection on the localhost? Do I need to do something to have it listen on a network?

So I found the answer fairly quick. I needed to modify the line I use for the server so that it reads:
java -Xdebug -agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n -Djava.library_path=${LIB_PATH} -classpath ${CP} -jar ${APP_DIR}/app.jar
So that is listens on all interfaces.