We are having problem connecting to our Java applications running in Amazon's EC2 cluster. We definitely have allowed both the "JMX port" (which is usually the RMI registry port) and the server port (which does most of the work) to the security-group for the instances in question. Jconsole connects but seems to hang and never show any information.
We are running our java with something like the following:
java -server -jar foo.jar other parameters here > java.log 2>&1
We have tried:
Telnets to the ports connect but no information is displayed.
We can run jconsole on the instance itself using remote-X11 over ssh and it connects and shows information. So the JRE is exporting it locally.
Opening all ports in the security group. Weeee.
Using tcpdump to make sure the traffic is not going to other ports.
Simulating it locally. We can always connect to our local JREs or those running elsewhere on our network using the same application parameters.
java -version outputs:
OpenJDK Runtime Environment (IcedTea6 1.11.5) (amazon-53.1.11.5.47.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
As an aside, we are using my Simple JMX package which allows us to set both the RMI registry and server ports which are typically semi-randomly chosen by the RMI registry. You can also force this with something like the following JMX URI:
service:jmx:rmi://localhost:" + serverPort + "/jndi/rmi://:" + registryPort + "/jmxrmi"
These days we use the same port for both the server and the registry. In the past we have used X as the registry-port and X+1 for the server-port to make the security-group rules easy. You connect to the registry-port in jconsole or whatever JMX client you are using.
We are having problem connecting to our Java applications running in Amazon's EC2 cluster.
It turns out that the problem was a combination of two missing settings. The first forces the JRE to prefer ipv4 and not v6. This was necessary (I guess) since we are trying to connect to it via a v4 address:
-Djava.net.preferIPv4Stack=true
The real blocker was the fact that JMX works by first contacting the RMI port which responds with the hostname and port for the JMX client to connect. With no additional settings it will use the local IP of the box which is a 10.X.X.X virtual address which a remote client cannot route to. We needed to add the following setting which is the external hostname or IP of the server -- in this case it is the elastic hostname of the server.
-Djava.rmi.server.hostname=ec2-107-X-X-X.compute-1.amazonaws.com
The trick, if you are trying to automate your EC2 instances (and why the hell would you not), is how to find this address at runtime. To do that you need to put something like the following in our application boot script:
# get our _external_ hostname
RMI_HOST=`wget -q -O - http://169.254.169.254/latest/meta-data/public-hostname`
...
java -server \
-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=$RMI_HOST \
-jar foo.jar other parameters here > java.log 2>&1
The mysterious 169.254.169.254 IP in the wget command above provides information that the EC2 instance can request about itself. I'm disappointed that this does not include tags which are only available in an authenticated call.
I initially was using the extern ipv4 address but it looks like the JDK tries to make a connection to the server-port when it starts up. If it uses the external IP then this was slowing our application boot time until that timed out. The public-hostname resolves locally to the 10-net address and to the public-ipv4 externally. So the application now is starting fast and JMX clients still work. Woo hoo!
Hope this helps someone else. Cost me 3 hours today.
To force your JMX server to start the server and the RMI registry on designated ports so you can block them in the EC2 Security Groups, see this answer:
How to close rmiregistry running on particular port?
Edit:
We just had this problem re-occur. It seems that the Java JMX code is doing some hostname lookups on the hostname of the box and using them to try to connect and verify the JMX connection.
The issue seems to be a requirement that the local hostname of the box should resolve to the local-ip of the box. For example, if your /etc/sysconfig/network has HOSTNAME=server1.foobar.com then if you do a DNS lookup on server1.foobar.com, you should get to the 10-NET virtual address. We were generating our own /etc/hosts file and the hostname of the local host was missing from the file. This caused our applications to either pause on startup or not startup at all.
Lastly
One way to simplify your JMX creation is to use my SimpleJMX package.
Per the second answer Why does JMX connection to Amazon EC2 fail?, the difficulty here is that by default the RMI port is selected at random, and clients need access to both the JMX and RMI ports. If you're running jdk7u4 or later, the RMI port can be specified via an app property. Starting my server with the following JMX settings worked for me:
Without authentication:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=<public EC2 hostname>
With authentication:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password.file=/path/to/jmxremote.password
-Djava.rmi.server.hostname=<public EC2 hostname>
I also opened ports 9998-9999 in the EC2 security group for my instance.
A bit different approach by using ssh tunnels
(On the Remote machine) Pass the following flags to the JVM
-Dcom.sun.management.jmxremote.port=1099
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=127.0.0.1
(On the Remote machine) Check which ports java started to use
$ netstat -tulpn | grep java
tcp 0 0 0.0.0.0:37484 0.0.0.0:* LISTEN 2904/java
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN 2904/java
tcp 0 0 0.0.0.0:45828 0.0.0.0:* LISTEN 2904/java
(On the local machine) Make ssh tunnels for all the ports
ssh -N -L 1099:127.0.0.1:1099 ubuntu#<ec2_ip>
ssh -N -L 37484:127.0.0.1:37484 ubuntu#<ec2_ip>
ssh -N -L 45828:127.0.0.1:45828 ubuntu#<ec2_ip>`
(On the local machine) Connect by Java Mission Control to localhost:1099
The answer given by Gray worked for me, however I find that I have to open TCP ports 0 to 65535 or I don't get in. I think that you can connect on the main JMX port, and then get another one assigned. I got that from this blog post that has always worked well for me.
We are using AWS Elastic Container Service for running our spring boot services.
The below config allowed us to connect to our docker containers.
Without Authentication:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9090 \
-Dcom.sun.management.jmxremote.rmi.port=9090 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=$(/usr/bin/curl -s --connect-timeout 2 \
http://169.254.169.254/latest/meta-data/public-ipv4)
I found it crisp and also doesn't require any other servicer side init script.
Related
I have application deployed on tomcat 8.5.63 version and Java 1.8 version. Whenever I am deploying the application tomcat is opening up tcp port 1099 for JMX rmi by default. And to which I am able to connect without any authentication from remote client using jvisualvm tool. I don't want that port to get opened up by default. I have tried changing the various jvm arguments based on other solution in SO but no luck.
Given below are the recent configuration which I have tried.
CATALINA_OPTS="-Xms512m -Xmx1024m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=16105 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.login.config=Tomcat -Djava.security.auth.login.config=$CATALINA_HOME/login.config -Dcom.sun.management.jmxremote.access.file=$CATALINA_BASE/conf/jmxremote.access -Dcom.sun.management.jmxremote.rmi.port=16106
I have specifically mentioned the -Dcom.sun.management.jmxremote.rmi.port=16106 to be used for JMX rmi but not able to understand why it still listening to port 1099. Please advise.
Note: I did saw some answers mentioning to pass XX:+DisableAttachMechanism as jvm argument to block jxm. But what if we want to have have jxm also enabled and default port(1099) should not be used.
I've installed the minikube instance on my local computer (--driver=docker). The minikube ip is 192.168.49.2. When I start minikube (minikube start --memory 7168) I get no errors on console. But trying to ping the minikube ip fails. What I do wrong?
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube Ready control-plane,master 9d v1.20.2 192.168.49.2 <none> Ubuntu 20.04.1 LTS 5.4.72-microsoft-standard-WSL2 docker://20.10.3
Recall that minikube is local Kubernetes - it runs a single-node Kubernetes cluster on your personal computer so that you can try out Kubernetes. Now, it doesn't run the Kubernetes cluster in your local box, it runs it inside a VM.
That is why you can't simply access Node IP from your local. Another way to see it is that in Kubernetes you can create NodePort Service to access your workload outside of your cluster but this doesn't work when you are running Kubernetes using minikube - and the reason is the same as mentioned above.
Now, how you work around that is by using minikube service <<YOUR_SERVICE_NAME>> command. This will create a tunnel to access your application - which is exposed using the Service - from outside of the K8S cluster.
You can try minikube tunnel as mentioned by #Hackerman but I have never tried it.
Just to add a bit on top of the previous answer. There is docker bridge limitation that makes it impossible to route the traffic to Linux containers. That is why the minikube tunnel and service were implemented as workaround for that.
minikube tunnel runs as a process, creating a network route on the
host to the service CIDR of the cluster using the cluster’s IP address
as a gateway. The tunnel command exposes the external IP directly to
any program running on the host operating system.
Alternative way to that you may find interesting would be using an ingress which was enabled in #9761 pull request:
.\minikube-windows-amd64.exe addons enable ingress I1121 00:59:39.443965 3000 translate.go:89] Failed to load translation file for en: Asset translations/en.json not found
* After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
* Verifying ingress addon...
* The 'ingress' addon is enabled
On your windows system, after the creation of the container that is created by the initial "minikube start", you can see the "minikube instance" by typing "docker ps". This is the minikube 'master' node running in this container.
It would look something like this:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6293ca0ba5b0 gcr.io/k8s-minikube/kicbase:v0.0.25 "/usr/local/bin/entr…" 2 hours ago Up About an hour 127.0.0.1:59539->22/tcp, 127.0.0.1:59540->2376/tcp, 127.0.0.1:59537->5000/tcp, 127.0.0.1:59538->8443/tcp, 127.0.0.1:59536->32443/tcp minikube
In the PORTS column you will see ports that are forwarded by virtue of the way that minikube start the docker container. You can see these kinds of forwards are handled by docker the same as for any contain that you might do a "docker run -p port:port"
Notice that the first port forward in this list is the ssh port: "127.0.0.1:59539->22/tcp".
When you do a "minikube tunnel", minikube will open ssh.exe instances that you can see in your windows task manager if you enable the Command Line display in the column settings.
Those 'tunnels' would look like this:
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -N docker#127.0.0.1 -p 59539 -i C:\Users\steve.sims\.minikube\machines\minikube\id_rsa "-L 8080:10.102.174.166:8080"
If I take that command apart and only run this from the command prompt:
ssh docker#127.0.0.1 -p 59539 -i C:\users\steve.sims\.minikube\machines\minikube\id_rsa
Then I get an interactive remote window into the minikube VM (or node). Typing 'ifconfig eth0' I get:
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.49.2 netmask 255.255.255.0 broadcast 192.168.49.255
So, indeed my minikube ip is 192.168.49.2, but it is an internal address as the folks above mentioned.
Other than that, the -N parameter on the ssh means "no command" and the -L on the end is the port forward flag in port-to-forward:destination-socket format. Of course, so all of the tunnels are coming across that initial docker -p port:port forward that minikube established when the container was started.
If it is useful, you can create your own ssh instances from that line's format by script and they will work just as well.
I'm trying to remotely connect to a server with JConsole to monitor & make use of the MBeans registered there.
I can VNC onto the server and run JConsole locally with "localhost:8050", but any attempt to connect remotely, via ip address or hostname, fails in a NullPointerException.
java.lang.NullPointerException
at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:281)
at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:227)
at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:334)
at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:296)
at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:280)
I've read through the other excellent q&a's here and on many other websites. I've tried opening ports with iptables, editing the hosts file. ssl and authentication are disabled, local.only is disabled. I've disabled the proxy and tried the JMXServiceURL too, to no avail.
How come i am able to run JConsole & connect locally but not remotely?
I even have a second server, running win2008 & tomcat, that plays along perfectly!
Any ideas?
Thanks!
Martin
JAVA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=<port no> \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.local.only=false \
-Djava.rmi.server.hostname=<server ip>"
Try to use this setting on your application server. It worked for me on Tomcat.
I'm on a vista machine. I've started tomcat 5.5.27 with these options:
CATALINA_OPTS="-Dcom.sun.management.jmxremote.port=9003 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"
When I connect via jconsole and added the following service url
service:jmx:rmi:///jndi/rmi://localhost:9003/jmxrmi
it would not connect. Any ideas ?
Ok, I orignally supposed the URL given by op was wrong but it turns out no. So I can't answer.
Still, here are the basics:
For a simple connection through jconsole.
If you know that the JMX Server you want to connect to has the RMI registry port at 9003 for example, connect using
localhost:9003
instead.
Otherwise, here's what I found out from the ground up:
Suppose you have the JMX Server (alias 'JMX Agent' alias 'the JVM you want to connect to') running on 'TARGET MACHINE' with the RMI registry port at 'RMI REGISTRY PORT' and the JMX RMI server port at 'JMX RMI SERVER PORT'.
Note:
The RMI registry tells JMX clients where to find the JMX RMI server port; information can be obtained under key jmxrmi.
The RMI registry port is generally known as it is set through system properties at JVM startup.
The JMX RMI server port is generally not known as the JVM chooses it at random (if no other precautions are taken).
The following URI will lead to success (tested)
service:jmx:rmi://<TARGET_MACHINE>:<JMX_RMI_SERVER_PORT>/jndi/rmi://<TARGET_MACHINE>:<RMI_REGISTRY_PORT>/jmxrmi
This looks nasty. Let's cut it apart.
This URI is an RFC2609 "Service Location Protocol URL" (well, it's really an URI, right?)
It is composed of:
service - a constant
jmx:rmi - the service type composed of: abstract type jmx and URL scheme rmi
the rest - the sap (service access protocol specification)
sap is decomposed into:
//<TARGET_MACHINE>:<JMX_RMI_SERVER_PORT> - ipsite
/jndi/rmi://<TARGET_MACHINE>:<RMI_REGISTRY_PORT>/jmxrmi - URL part
A well-informed JMX client connects to the "ipsite" to do JMX-over-RMI exchanges; but what of the JMX client that doesn't KNOW that port? Patience...
URL part is decomposed into:
/jndi/ - This seems to tell the JMX client that it can get lookup information at the location that follows
rmi://<TARGET_MACHINE>:<RMI_REGISTRY_PORT>/jmxrmi - Yep, we get information about the JMX RMI Server at the RMI registry, under the lookup key jmxrmi
This is somewhat cart-before-horse, as one has to contact the RMI registry given by the latter part of the SLP URL first.
After scratching head, intuitively, let's try:
service:jmx:rmi://<TARGET_MACHINE>/jndi/rmi://<TARGET_MACHINE>:<RMI_REGISTRY_PORT>/jmxrmi
Yes, that works! The JMX RMI server port is nicely obtained from the registry. On second thoughts, the target machine should also be obtained from the registry, thus:
service:jmx:rmi:///jndi/rmi://<TARGET_MACHINE>:<RMI_REGISTRY_PORT>/jmxrmi
Even better, that works, too!
References:
1 download.oracle.com/javase/6/docs/api/javax/management/remote/rmi/package-summary.html
2 download.oracle.com/javase/6/docs/api/javax/management/remote/JMXServiceURL.html
3 mx4j.sourceforge.net/docs/ch03s04.html
4 download.oracle.com/javase/6/docs/technotes/guides/management/agent.html#gdevg
5 http://www.rfc-editor.org/rfc/rfc2609.txt
On Ubuntu 10.04, using OpenJDK 6 and Tomcat 6.0.29, I was unable to activate JMX for a local jconsole session, no matter how many com.sun.management.jmxremote.* options I passed to java with CATALINA_OPTS. The problem was the -Djava.io.tmpdir setting, which defaults to $CATALINA_BASE/temp. I simply had to set:
CATALINA_TMPDIR="/tmp"
at the beginning of bin/catalina.sh and I was able to connect locally with jconsole, jmap, jps etc. There was no need for any com.sun.management.jmxremote.* settings at all.
Are the processes run under the same user?
You can also check by running jps and jconsole (both in the JDK_HOME/bin directory)
This is also needed for OS X 10.7 aka Lion.
I answered a similar question here:java.rmi.ConnectException: Connection refused to host: 127.0.1.1;
I found many of the Q&A on this topic, not nothing was helping me - that's because my issue was more basic ( what can I say I am not a networking guru :) ). My ip address in /etc/hosts was incorrect. What I had tried included the following for CATALINA_OPTS:
CATALINA_OPTS="$CATALINA_OPTS -Djava.awt.headless=true -Xmx128M -server
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=7091
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=A.B.C.D" #howeverI put the wrong ip here!
export CATALINA_OPTS
My problem was that I had changed my ip address many months ago, but never updated my /etc/hosts file. it seems that by default the jconsole uses the hostname -i ip address in some fashion even though I was viewing local processes. The best solution was to simply change the /etc/hosts file.
The other solution which can work is to get your correct ip address from /sbin/ifconfig and use that ip address when specifying the ip address in, for example, a catalina.sh script:
-Djava.rmi.server.hostname=A.B.C.D
I'm working on a Java 10 application that uses an embedded Jetty server to provide control from a local network, and I'm attempting to connect to the JVM and failing. It's running on Ubuntu 18.04 LTS desktop.
My startup script has the following lines:
java -Xdebug -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n -Djava.library_path=${LIB_PATH} -classpath ${CP} -jar ${APP_DIR}/app.jar
I have ufw on the system, and I've verified that the port is open. My output from ufw status includes:
8000 ALLOW Anywhere
8000 (v6) ALLOW Anywhere
In IntelliJ, my debug configuration is
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8000
When I try to connect, I get an error message that the connection is refused.
with the IP of the machine in the config's address box.
Looking at the output of netstat -l, I see the following:
tcp 0 0 localhost:8000 0.0.0.0:* LISTEN
Does this mean that the debugger is only listening for connection on the localhost? Do I need to do something to have it listen on a network?
So I found the answer fairly quick. I needed to modify the line I use for the server so that it reads:
java -Xdebug -agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n -Djava.library_path=${LIB_PATH} -classpath ${CP} -jar ${APP_DIR}/app.jar
So that is listens on all interfaces.