Develop Reference

Tomcat Keep Listening to tcp 1099 port

I have application deployed on tomcat 8.5.63 version and Java 1.8 version. Whenever I am deploying the application tomcat is opening up tcp port 1099 for JMX rmi by default. And to which I am able to connect without any authentication from remote client using jvisualvm tool. I don't want that port to get opened up by default. I have tried changing the various jvm arguments based on other solution in SO but no luck.
Given below are the recent configuration which I have tried.
CATALINA_OPTS="-Xms512m -Xmx1024m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=16105 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.login.config=Tomcat -Djava.security.auth.login.config=$CATALINA_HOME/login.config -Dcom.sun.management.jmxremote.access.file=$CATALINA_BASE/conf/jmxremote.access -Dcom.sun.management.jmxremote.rmi.port=16106
I have specifically mentioned the -Dcom.sun.management.jmxremote.rmi.port=16106 to be used for JMX rmi but not able to understand why it still listening to port 1099. Please advise.
Note: I did saw some answers mentioning to pass XX:+DisableAttachMechanism as jvm argument to block jxm. But what if we want to have have jxm also enabled and default port(1099) should not be used.

Connect JVisualVM to a Remote Wildfly Instance?

I'm trying to connect JVisualVM, running on my local machine, to a remote machine which is running a WildFly server (version 8.1.0, to be specific.)
I didn't configure the WildFly server myself, and I don't know who did, but I do know that I can log in as an administrative user from my local machine by pointing my browser at:
https://[ip address of the remote machine]:9443/console
Note that it's https, not ordinary http, and that the port for that has been set to 9443 (I think the default is 8080 or 9990 or something... IDK, I saw a lot of port numbers online. I have been explicitly told that http was disabled for this WildFly server).
I can SSH into the remote machine. I can navigate to the bin directory for WildFly and run jboss-client.sh. I have to connect on port 9999 (I think the default is 9990 for that?)
I copied the jboss-client.jar (under bin/client) to my local machine and ran JVisualVM from the command line like this:
.\jvisualvm.exe -cp:a C:\[path to]\jboss-client.jar
It launches fine. File > Add Remote Host: Then I entered the IP. OK. I right clicked on it under Remote in the tree and picked Add JMX Connection. I entered
service:jmx:http-remoting-jmx://[ip]:9999
I checked off that I wanted to use the security credentials and entered the username and password. Checked off to save the security credentials. Left "Do not require SSL Connection" unchecked. Hit OK. It immediately spat out the message
Cannot connect to admin#service:jmx:http-remoting-jmx://[ip]:9999 using service:jmx:http-remoting-jmx://[ip]:9999
I also tried the port 9443, 9990, and 8080 instead. None of those worked. I tried https instead of http in the protocol name. That also didn't work.
What am I missing? How is it that I can access the console, and connect with jboss-client.sh, but I can't use JVisualVM? Is there some log I can use somewhere to see what's wrong? Maybe someone can point out a configuration I've missed somewhere?
Not sure if it's important or not, but my local machine is running Windows 10 with JDK8 installed. The WildFly server is using Java 6 on CentOS 6.3.

You need to add the jboss-client.jar (or jboss-cli-client.jar) to the class path for JVisualVM. The library can be found in the bin/client directory of the WildFly install.
I used the following command to add the library to the class path.
jvisualvm --cp:a ~/servers/wildfly-10.0.0.Final/bin/client/jboss-client.jar
Then I used service:jmx:remote+http://[ip]:[port] and was able to connect.

I don't know if someone else is also (still) having the same issue (Wildfly10 on a remote machine where management console is available at 9443 with HTTPS). The following worked for me.
For ssh connections:
Starting jvisualvm with jboss-client.jar
jvisualvm --cp:a #JBOSS_HOME/bin/client/jboss-client.jar
Using the following connection string:
service:jmx:remote+https://remote-server:9443
NOTE: I used here remote+https
Provide username and password
Hope this helps.

you missed run jstatd command in remote host ,
this little program is RMI server that possible connection from client to remote host though you using jmx connection it used jmxrmi protocol for that connection .
so first in remote host create file name as security.policy with this contain :
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
off course you must in file section for linux put explicit path and then of creation this file put it in bin directory of jdk.home
then you should run this command on remote host
$JAVA_HOME/bin/jstatd -J-Djava.security.policy=path of /security.policy -J-Djava.rmi.server.hostname=remote ip address -J-Djava.net.preferIPv4Stack=true
then you could connect to server off course with correct settings.

Include jboss-cli-client.jar and jboss-client.jar under \lib\visualvm\platform\lib and restart jvisualvm to pickup new jars.

Handshake failed - connection prematurally closed error when debugging Solr in Intellij

So i was going to debug my Solr filter plugins on Intellij Community Edition. After i ran the program from comand prompt with this command
java -jar start.jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8983
I started my Intellij debugger with this config:
Transport : socket
Debugger mode : attach
Host : localhost
Port : 8983
But when I ran the debugger I got this error:
Error running Debugger: Unable to open debugger port (localhost:8983):
java.io.IOException "handshake failed - connection prematurally closed"
Any idea how to fix this?

I got that error when trying to access to debug port on a Docker container.
If you are trying to access the debug port inside a Docker container make sure you are specifying the port as *:5005
E.g.
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
This has been changes since Java 9.
See: REGRESSION: Remote debugging does not work on JDK 9
It's not a bug. It's a security.
Before the JDK-8041435
If you have a server with EXT and INT interfaces and start Java process with address=5900 it binds to both interfaces and allow anybody from entire world to connect to your java process unless you block it on firewall.
After JDK-8041435 socket transport try to guess localhost and bind to localhost only. I.e. socket transport by default works only if both client and server are located on the same machine. It's not an easy task to guess proper localhost. so ever same-machine configuration might not work in some situation because of network setup.
You can restore old, insecure behavior using * (asteric)
i.e.
-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5900
should work exactly as it was before JDK-8041435
But it's recommended to explicitly specify ip address to bind when it possible.
And JDWP socket connector accept only local connections by default
The JDWP socket connector has been changed to bind to localhost only if no ip address or hostname is specified on the agent command line. A hostname of asterisk (*) may be used to achieve the old behavior which is to bind the JDWP socket connector to all available interfaces; this is not secure and not recommended.

It should be something like this,
java "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8983" -jar start.jar
it's working now

I had this error with OpenJDK 11 inside Docker container and setting environment variable JAVA_DEBUG_PORT to "*:5005" worked for me.

You forgot to specify -Xdebug on the java command line.
Edit: As in
java -jar start.jar -Xdebug -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8983

It has helped me, at least in Intellij IDEA:
java -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=0.0.0.0:5005
try to add ip 0.0.0.0.

How to connect to Java instances running on EC2 using JMX

We are having problem connecting to our Java applications running in Amazon's EC2 cluster. We definitely have allowed both the "JMX port" (which is usually the RMI registry port) and the server port (which does most of the work) to the security-group for the instances in question. Jconsole connects but seems to hang and never show any information.
We are running our java with something like the following:
java -server -jar foo.jar other parameters here > java.log 2>&1
We have tried:
Telnets to the ports connect but no information is displayed.
We can run jconsole on the instance itself using remote-X11 over ssh and it connects and shows information. So the JRE is exporting it locally.
Opening all ports in the security group. Weeee.
Using tcpdump to make sure the traffic is not going to other ports.
Simulating it locally. We can always connect to our local JREs or those running elsewhere on our network using the same application parameters.
java -version outputs:
OpenJDK Runtime Environment (IcedTea6 1.11.5) (amazon-53.1.11.5.47.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
As an aside, we are using my Simple JMX package which allows us to set both the RMI registry and server ports which are typically semi-randomly chosen by the RMI registry. You can also force this with something like the following JMX URI:
service:jmx:rmi://localhost:" + serverPort + "/jndi/rmi://:" + registryPort + "/jmxrmi"
These days we use the same port for both the server and the registry. In the past we have used X as the registry-port and X+1 for the server-port to make the security-group rules easy. You connect to the registry-port in jconsole or whatever JMX client you are using.

We are having problem connecting to our Java applications running in Amazon's EC2 cluster.
It turns out that the problem was a combination of two missing settings. The first forces the JRE to prefer ipv4 and not v6. This was necessary (I guess) since we are trying to connect to it via a v4 address:
-Djava.net.preferIPv4Stack=true
The real blocker was the fact that JMX works by first contacting the RMI port which responds with the hostname and port for the JMX client to connect. With no additional settings it will use the local IP of the box which is a 10.X.X.X virtual address which a remote client cannot route to. We needed to add the following setting which is the external hostname or IP of the server -- in this case it is the elastic hostname of the server.
-Djava.rmi.server.hostname=ec2-107-X-X-X.compute-1.amazonaws.com
The trick, if you are trying to automate your EC2 instances (and why the hell would you not), is how to find this address at runtime. To do that you need to put something like the following in our application boot script:
# get our _external_ hostname
RMI_HOST=`wget -q -O - http://169.254.169.254/latest/meta-data/public-hostname`
...
java -server \
-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=$RMI_HOST \
-jar foo.jar other parameters here > java.log 2>&1
The mysterious 169.254.169.254 IP in the wget command above provides information that the EC2 instance can request about itself. I'm disappointed that this does not include tags which are only available in an authenticated call.
I initially was using the extern ipv4 address but it looks like the JDK tries to make a connection to the server-port when it starts up. If it uses the external IP then this was slowing our application boot time until that timed out. The public-hostname resolves locally to the 10-net address and to the public-ipv4 externally. So the application now is starting fast and JMX clients still work. Woo hoo!
Hope this helps someone else. Cost me 3 hours today.
To force your JMX server to start the server and the RMI registry on designated ports so you can block them in the EC2 Security Groups, see this answer:
How to close rmiregistry running on particular port?
Edit:
We just had this problem re-occur. It seems that the Java JMX code is doing some hostname lookups on the hostname of the box and using them to try to connect and verify the JMX connection.
The issue seems to be a requirement that the local hostname of the box should resolve to the local-ip of the box. For example, if your /etc/sysconfig/network has HOSTNAME=server1.foobar.com then if you do a DNS lookup on server1.foobar.com, you should get to the 10-NET virtual address. We were generating our own /etc/hosts file and the hostname of the local host was missing from the file. This caused our applications to either pause on startup or not startup at all.
Lastly
One way to simplify your JMX creation is to use my SimpleJMX package.

Per the second answer Why does JMX connection to Amazon EC2 fail?, the difficulty here is that by default the RMI port is selected at random, and clients need access to both the JMX and RMI ports. If you're running jdk7u4 or later, the RMI port can be specified via an app property. Starting my server with the following JMX settings worked for me:
Without authentication:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=<public EC2 hostname>
With authentication:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password.file=/path/to/jmxremote.password
-Djava.rmi.server.hostname=<public EC2 hostname>
I also opened ports 9998-9999 in the EC2 security group for my instance.

A bit different approach by using ssh tunnels
(On the Remote machine) Pass the following flags to the JVM
-Dcom.sun.management.jmxremote.port=1099
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=127.0.0.1
(On the Remote machine) Check which ports java started to use
$ netstat -tulpn | grep java
tcp 0 0 0.0.0.0:37484 0.0.0.0:* LISTEN 2904/java
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN 2904/java
tcp 0 0 0.0.0.0:45828 0.0.0.0:* LISTEN 2904/java
(On the local machine) Make ssh tunnels for all the ports
ssh -N -L 1099:127.0.0.1:1099 ubuntu#<ec2_ip>
ssh -N -L 37484:127.0.0.1:37484 ubuntu#<ec2_ip>
ssh -N -L 45828:127.0.0.1:45828 ubuntu#<ec2_ip>`
(On the local machine) Connect by Java Mission Control to localhost:1099

The answer given by Gray worked for me, however I find that I have to open TCP ports 0 to 65535 or I don't get in. I think that you can connect on the main JMX port, and then get another one assigned. I got that from this blog post that has always worked well for me.

We are using AWS Elastic Container Service for running our spring boot services.
The below config allowed us to connect to our docker containers.
Without Authentication:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9090 \
-Dcom.sun.management.jmxremote.rmi.port=9090 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=$(/usr/bin/curl -s --connect-timeout 2 \
http://169.254.169.254/latest/meta-data/public-ipv4)
I found it crisp and also doesn't require any other servicer side init script.

Address already in use: JVM_Bind java

Some times whenever I restart the application, which is built on
Java
Struts
Mysql and Jboss 4.05 Version
I get the error as Address already in use: JVM_Bind
Only fix that i know is to restart the machine and try again, it will work.
Else Some times I do Ctrl-Alt-Del and Stop all the process related to Java, some times this also works.
But what is the exact reason and how can we prevent this problem ?

Address already in use: JVM_Bind
means that some other application is already listening on the port your current application is trying to bind.
what you need to do is, either change the port for your current application or better; just find out the already running application and kill it.
on Linux you can find the application pid by using,
netstat -tulpn

In windows this scenario happens when Eclipse crashes without a clean shutdown it will have the local Jetty or Tomcat server keep running.
When you reopen Eclipse and try to start server again this will lead to the "Address already in use: JVM_Bind"
You can solve this by opening Task Manager and find the javaw.exe process and ending it.
Then you can restart the server on Eclipse.

On windows, in an elevated cmd/ps
net stop winnat
then
net start winnat
Resets the windows' NAT service and that removes faulty listeners.
This method works for most 'cannot listen on this port' errors, like as in Docker, JVM etc.

I usually come across this when the port which the server (I use JBoss) is already in use
Usual suspects
Apache Http Server => turn down the service if working in windows.
IIS => stop the ISS using
Skype =>yea I got skype attaching itself to port 80
To change the port to which JBoss 4.2.x binds itself go to:
"C:\jboss4.2.2\server\default\deploy\jboss-web.deployer\server.xml"
here default is the instance of the server
change the port here :
<Connector port="8080"
address="${jboss.bind.address}" >
In the above example the port is bound to 8080

Open command line and type: netstat -a -o -n or tasklist to see currently running processes.
Find port that related to Java and type: taskkill /F /PID <your PID number>.
Click Enter.

Is it possible that MySql listening on the same port as JBoss?
Is there a port number given in the error message - something like Address already in use: JVM_Bind:8080
You can change the port in JBoss server.xml to test this.

For the sake of completeness as many may fall in this SO for other reasons than OP's question, here is another info that saved my day :
TL;DR;
Check that the port you want to open is not reserved (even though no application has opened it)
On windows :
netsh interface ipv4 show excludedportrange protocol=tcp
Explanation :
At least on windows, another possible cause for the java.net.BindException: Address already in use: JVM_Bind is that the operating system has "reserved" the port.
If you use the commands from other answer in this topic, they will tell you that no application is listening to the port, yet you cannot open it.
Some windows update like this one reserves range port that can then no longer be claimed by processes.
Though not explicit on the port that can no longer be opened, the update note mentions a command that can help troubleshooting the issue :
netsh interface ipv4 show excludedportrange protocol=tcp
this will yield a list of port range that are blocked :
Protocol tcp Port Exclusion Ranges
Start Port End Port
---------- --------
5357 5357
49709 49808
49809 49908
49909 50008
50009 50108
50109 50208
50280 50379
* - Administered port exclusions.

The quick answer on how to prevent it is that you most likely need to stop JBoss before starting it again.
You should be able to call the "Terminate" button in the Console view to shutdown the server.

That error means that the you are trying to create a new ServerSocket on a port already in use by another ServerSocket. So try to make your application closing all sockets and connections you know about and be sure your application is completely terminated. Also check if there is another proces you launched by your program.

It can be also caused by double definition of port 8080 in ..\tomcat\conf\server.xml :
<Connector port="8080"
enableLookups="false" redirectPort="8443" debug="0"/>
<Connector port="8080"
enableLookups="false" address="127.0.0.1" maxParameterCount="30000"/>

please try following options for JVM binding exception:
start and stop the server. and check the server process ids and kill and stop the server.
go to control panel->administrative tool-> service-> check all server and stop all the servers and then start your own server.
change the Browser which your using. for example if
your using IE ,change it to Mozilla firefox.

I was having this problem too. For me, I couldn't start/stop openfire (it said it was stopped, but everything was still running)
sudo /etc/init.d/openfire stop
sudo /etc/init.d/openfire start
Also, restarting apache did not help either
sudo /etc/init.d/apache2 restart
The errors were inside:
/opt/openfire/logs/stderror.log
Error creating server listener on port 5269: Address already in use
Error creating server listener on port 5222: Address already in use
The way I fixed this, I had to actually turn off the server inside the admin area for my host.

I had the same on Windows. My solution was to get which port the debug wants to connect to. (In IntelliJ a red rectangle already giving the info: "Error running Tomcat: Unable to open debugger port (127.0.0.1:XXXXX): ... Already in use...")
Let's say XXXXX is the port number.
Then i searched for the problem and the PID in a cmd window:
netstat -ano | find "CLOSE_WAIT" | find ":XXXXX"
I got the PID number as the last number in the result line. (Let's say YYYY)
Finally:
TASKKILL /PID YYYY
An extra info: Winscp logged out meanwhile, probably it was causing my problem. :)

This recently happen to me when enabling JMX on two running tomcat service within Eclipse. I mistakenly put the same port for each server.
Simply give each jmx remote a different port
Server 1
-Dcom.sun.management.jmxremote.port=9000
Server 2
-Dcom.sun.management.jmxremote.port=9001

You can try deleting the Team Server credentials, most likely those will include some kind of port in the server column.
Like https://wathever.visualstudio.com:443
Go to Windows/Preferences
expand Team then Team Foundation Server
go to Credentials and remove whichever is there.

On Mac, even when I'd search with netstat, no processes were binding to that port, but I'd still get the "Address already in use" Bind exception.
In this case, going into
Safari -> Preferences -> Privacy -> Manage Website Data... -> finding "localhost" -> Remove
fixes the problem, if there really is no process running with that port. It seems Safari's web cache maintains a reference to the port somehow, and removing that cache frees up the port.

Sometimes, I use lsof -i:8080 and I get nothing back, no pid. Well something has to be taking place. If this is your case, it could be docker. By using docker ps you can check if you have any old dockers lying around and kill them. This could be the culprit for a port already in use error.