Prometheus JMX Exporter java agent for Kafka won't run - java

I am attempting to setup confluent kafka v5.4 and running the prometheus JMX exporter. I have found this blog for how to get this setup https://alex.dzyoba.com/blog/jmx-exporter/ . Kafka is setup and runs just fine but the endpoint on port 8080 returns nothing. I've tried just about everything for how I call the javaagent in the systemd script but nothing seems to work.
Description=Confluent Kafka Broker
After=network.target network-online.target remote-fs.target zookeeper.service
[Service]
Type=forking
User=confluent
Group=confluent
Environment="KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent=/opt/prometheus/jmx_prometheus_javaagent.jar=8080:/opt/prometheus/config.yaml"
Environment=LOG_DIR=/var/log/confluent
ExecStart=/opt/confluent/confluent-5.4.0/bin/kafka-server-start -daemon /opt/confluent/confluent-5.4.0/etc/kafka/server.properties
ExecStop=/opt/confluent/confluent-5.4.0/bin/kafka-server-stop
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
Any ideas on how to call that java agent in the systemd script to get it to work correctly? I have tried multiple options for calling the OPTS with none of them working. I've tried putting the -javaagent command in KAFKA_OPTS. Feel like I've tried just about every option. Kafka logs don't give any clues and I'm not sure of where else to look at logs for this type of issue.
OS Centos 7 JMX exporter 0.12.0 Java openJDK 11
Logs that I have found are not telling me anything as to why it's not running. Maybe I'm looking at the wrong logs.
Edit:
conflue+ 11578 47.4 13.8 8679808 536764 ? Sl 11:59 0:35 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xlog:gc*:file=/var/log/confluent/kafkaServer-gc.log:time,tags:filecount=10,filesize=102400 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent=/opt/prometheus/jmx_prometheus_javaagent.jar=8080:/opt/prometheus/config.yaml -Dkafka.logs.dir=/var/log/confluent -Dlog4j.configuration=file:/etc/kafka/log4j.properties -cp /opt/confluent/confluent-5.4.0/bin/../ce-broker-plugins/build/libs/*:/opt/confluent/confluent-5.4.0/bin/../ce-broker-plugins/build/dependant-libs/*:/opt/confluent/confluent-5.4.0/bin/../ce-auth-providers/build/libs/*:/opt/confluent/confluent-5.4.0/bin/../ce-auth-providers/build/dependant-libs/*:/opt/confluent/confluent-5.4.0/bin/../ce-rest-server/build/libs/*:/opt/confluent/confluent-5.4.0/bin/../ce-rest-server/build/dependant-libs/*:/opt/confluent/confluent-5.4.0/bin/../ce-audit/build/libs/*:/opt/confluent/confluent-5.4.0/bin/../ce-audit/build/dependant-libs/*:/opt/confluent/confluent-5.4.0/bin/../share/java/kafka/*:/opt/confluent/confluent-5.4.0/bin/../share/java/confluent-metadata-service/*:/opt/confluent/confluent-5.4.0/bin/../share/java/rest-utils/*:/opt/confluent/confluent-5.4.0/bin/../share/java/confluent-common/*:/opt/confluent/confluent-5.4.0/bin/../share/java/confluent-security/schema-validator/*:/opt/confluent/confluent-5.4.0/bin/../support-metrics-client/build/dependant-libs-2.12.10/*:/opt/confluent/confluent-5.4.0/bin/../support-metrics-client/build/libs/*:/usr/share/java/support-metrics-client/*:/opt/confluent/confluent-5.4.0/bin/../support-metrics-fullcollector/build/dependant-libs-2.12.10/*:/opt/confluent/confluent-5.4.0/bin/../support-metrics-fullcollector/build/libs/*:/usr/share/java/support-metrics-fullcollector/* io.confluent.support.metrics.SupportedKafka /opt/confluent/confluent-5.4.0/etc/kafka/server.properties

Make sure you run systemctl daemon-reload each time you edit a service file.
Also, I would suggest not using LOG_DIR, removing the RollingFileAppender from the log4j.properties, then letting journalctl handle all your logging from SystemD

Figured this out
ExecStart=/opt/confluent/confluent-5.4.0/bin/kafka-server-start -daemon /opt/confluent/confluent-5.4.0/etc/kafka/server.properties
was not correct
ExecStart=/opt/confluent/confluent-5.4.0/bin/kafka-server-start -daemon /etc/kafka/server.properties
was correct, even though the 2 are symlinked.

Related

Why the JDK's jps command does not list the process of JBoss instance?

I am currently using JBoss at work to run some Java web application in Linux. The running instance of JBoss is listed with the process list command ps aux |grep java, and the relevant output is something like jboss 19622 0.3 35.8 3410688 1391068 ? Sl Dec13 3:27 /opt/wsp/jdk1.8.0/bin/java -D[Standalone] -server -XX:+UseCompressedOops -Xms1024m -Xmx1024m -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true...
According to my understanding, there should be corresponding JVMs listed by using JDK's command jps. But when I typed jps in terminal, only one record is listed, something like 12073 jps. I am pretty confused about why is it like that, anyone can explain? Thanks in advance!
The reason is that jBoss is run by a different user. ps aux shows every process of the system, while jps is restricted by permissions of the user running the command.

Tomcat says OutOfMemoryError every time when I visit JasperServer application deployed to wepapps

I installed the jasperserver(version: 5.6) in Windows 7 through the official document, use an exsiting Tomcat(version: 7) and an existing PostgreSQL(version: 9.2.8), I checked the installation.log file in JasperServer's installation path, and everything is OK.
But when I starting the Tomcat by double click 'Tomcat.exe' and it always says the following error:
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
I searched for these problem and some says to configure the JVM options in %CATALINA_HOME%\bin\setenv.bat or %CATALINA_HOME\bin\%catalina.bat, but I can't find any one of them. I think this is because I'm using an exsiting Tomcat and the installation of JasperServer modified some files in %CATALINA_HOME%. So I searched 'setenv.bat' from %JasperServer% and find it in %JasperServer%\scripts. Here is my configuration:
set JAVA_OPTS=%JAVA_OPTS% -Xms2048m -Xmx3072m -XX:PermSize=1024m -XX:MaxPermSize=2048m -Xss2m -XX:+UseCompressedOoops -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
Also, I did followed the official document here, and nothing helps, Error exists.
Is there anyone who met the same problem? What should I do? I really need help. Thanks.
It seems like the memory parameters you have specified have not been picked up? Check it out, for example by looking at the process table similar to following example:
ivos-mbp:demo ivomagi$ jps
1562 start.jar
1572 Jps
my-mbp:demo me$ ps axu 1562
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
me 1562 0.2 2.8 2965576 230784 s001 S 12:17PM 0:22.62 /usr/bin/java -Dcom.sun.management.jmxremote -Xmx168m
In case you do not see the specified -Xmx3072m in the printout, you need to doublecheck where from the configuration is loaded.
Please, set CATALINA_OPTS instead of JAVA_OPS. Tomcat refers to CATALINA_OPTS on the start:
set "CATALINA_OPTS= -Xms2048m -Xmx3072m -XX:PermSize=1024m -XX:MaxPermSize=2048m -Xss2m -XX:+UseCompressedOoops -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled"

Can't stop tomcat normally when I configure jmxremote

I added a jmxremote configuraiton in the catalina.bat:
set JAVA_OPTS=-Dcom.sun.management.jmxremote.port=9004 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
so that I could start jconsole to monitor the tomcat's performance.
But I got a problem that I couldn't stop tomcat normally through $CATALINA_HOME\catalina.bat stop, neither did $CATALINA_HOME\shutdown.bat
Any suggestions?
In order to monitor the java process, you need to add the following system properties to the command line:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
In the visualvm you just use the connection string host:9999.
However, sometimes the RMI listener listens to the wrong IP address, one which is inaccessible to the visualvm. Thanks to Pavel’s tip, I found a way to overcome this is by adding the following parameters:
-Djava.rmi.server.hostname=$(hostname)
-Djava.rmi.server.useLocalHostname=true
Now it works like a charm!
For completeness, I’d mention you can secure the connection to the JVM, either by requiring user/password or by using SSL. If you are interested, please see this guide.
Make sure that you put the definitions in a place only the start command sees, but not the shutdown. The reason or this is that the jmx remote create a listening socket, making the shutdown to listen to the same port if not configured properly.
Are you sure Tomcat started successfully?
Usually you need to add this property before the other jmx properties:
-Dcom.sun.management.jmxremote
http://tomcat.apache.org/tomcat-6.0-doc/monitoring.html#Enabling_JMX_Remote
Better to add these parameters to the CATALINA_OPTS in the file setenv.sh or setenv.bat
# JMX OPTIONS
CATALINA_OPT="$CATALINA_OPT "-Dcom.sun.management.jmxremote
CATALINA_OPT="$CATALINA_OPT "-Dcom.sun.management.jmxremote.port=9012
CATALINA_OPT="$CATALINA_OPT "-Dcom.sun.management.jmxremote.local.only=false
CATALINA_OPT="$CATALINA_OPT "-Dcom.sun.management.jmxremote.authenticate=false
CATALINA_OPT="$CATALINA_OPT "-Dcom.sun.management.jmxremote.ssl=false

How to activate JMX on my JVM for access with jconsole?

How to activate JMX on a JVM for access with jconsole?
The relevant documentation can be found here:
http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
Start your program with following parameters:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.rmi.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
For instance like this:
java -Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9010 \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-jar Notepad.jar
-Dcom.sun.management.jmxremote.local.only=false is not necessarily required
but without it, it doesn't work on Ubuntu. The error would be something like
this:
01 Oct 2008 2:16:22 PM sun.rmi.transport. customer .TCPTransport$AcceptLoop executeAcceptLoop
WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=37278] throws
java.io.IOException: The server sockets created using the LocalRMIServerSocketFactory only accept connections from clients running on the host where the RMI remote objects have been exported.
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:89)
at sun.rmi.transport. customer .TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport. customer .TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:636)
see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6754672
Also be careful with -Dcom.sun.management.jmxremote.authenticate=false which
makes access available for anyone, but if you only use it to track the JVM on
your local machine it doesn't matter.
Update:
In some cases I was not able to reach the server. This was then fixed if I set this parameter as well: -Djava.rmi.server.hostname=127.0.0.1
Running in a Docker container introduced a whole slew of additional problems for connecting so hopefully this helps someone. I ended up needed to add the following options which I'll explain below:
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=${DOCKER_HOST_IP}
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
DOCKER_HOST_IP
Unlike using jconsole locally, you have to advertise a different IP than you'll probably see from within the container. You'll need to replace ${DOCKER_HOST_IP} with the externally resolvable IP (DNS Name) of your Docker host.
JMX Remote & RMI Ports
It looks like JMX also requires access to a remote management interface (jstat) that uses a different port to transfer some data when arbitrating the connection. I didn't see anywhere immediately obvious in jconsole to set this value. In the linked article the process was:
Try and connect from jconsole with logging enabled
Fail
Figure out which port jconsole attempted to use
Use iptables/firewall rules as necessary to allow that port to connect
While that works, it's certainly not an automatable solution. I opted for an upgrade from jconsole to VisualVM since it let's you to explicitly specify the port on which jstatd is running. In VisualVM, add a New Remote Host and update it with values that correlate to the ones specified above:
Then right-click the new Remote Host Connection and Add JMX Connection...
Don't forget to check the checkbox for Do not require SSL connection. Hopefully, that should allow you to connect.
Note, Java 6 in the latest incarnation allows for jconsole to attach itself to a running process even after it has been started without JMX incantations.
If that is available to you, also consider jvisualvm as it provides a wealth of information on running processes, including a profiler.
I'm using WAS ND 7.0
My JVM need all the following arguments to be monitored in JConsole
-Djavax.management.builder.initial=
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8855
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
On Linux, I used the following params:
-Djavax.management.builder.initial=
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
and also I edited /etc/hosts so that the hostname resolves to the host address (192.168.0.x) rather than the loopback address (127.0.0.1)
along with below command line parameters ,
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Sometimes in the linux servers , imx connection doesn't get succeeded. that is because , in cloud linux host, in /etc/hosts so that the hostname resolves to the host address.
the best way to fix it is, ping the particular linux server from other machine in network and use that host IP address in the
-Djava.rmi.server.hostname=IP address that obtained when you ping that linux server.
But never rely on the ipaddress that you get from linux server using ifconfig.me. the ip that you get there is masked one which is present in the host file.
The below options works for me:
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.rmi.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname={host name}
and remember to open 9010 port in the server
sudo ufw allow 9010/udp
sudo ufw allow 9010/tcp
sudo ufw reload
Run your java application with the following command line parameters:
-Dcom.sun.management.jmxremote.port=8855
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
It is important to use the -Dcom.sun.management.jmxremote.ssl=false parameter if you don't want to setup digital certificates on the jmx host.
If you started your application on a machine having IP address 192.168.0.1, open jconsole, put 192.168.0.1:8855 in the Remote Process field, and click Connect.
Step 1: Run the application using following parameters.
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Above arguments bind the application to the port 9999.
Step 2: Launch jconsole by executing the command jconsole in command prompt or terminal.
Select ‘Remote Process:’ and enter the url as {IP_Address}:9999 and click on Connect button to connect to the remote application.
You can refer this link for complete application.
RUN LOCAL PROCESS JCONSOLE using Remote Process option
To run locally, this worked for me -
I added this in my vm args -
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=6001
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=localhost
-Dcom.sun.management.jmxremote.rmi.port=6001
I opened JConsole via Intellij Terminal
It was showing me all PID's in grey in local
So I selected remote process and logged in using host - localhost:6001
Keep empty username & password
Then click connect
Make sure no other process is running on port 6001. You can also use other ports.
First you need to check if your java process is already running with JMX parameters. Do this:
ps -ef | grep java
Check your java process you need to monitor. If you can see jmx rmi parameter Djmx.rmi.registry.port=xxxx then use the port mentioned here in your java visualvm to connect it remotely under jmx connection.
If it's not running through jmx rmi port then you need to run your java process with below mentioned parameters :
-Djmx.rmi.registry.port=1234 -Djmx.rmi.port=1235 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
Note: port numbers are based on your choice.
Now you can use this port for jmx coneection. Here it is port 1234.
I had this exact issue, and created a GitHub project for testing and figuring out the correct settings.
It contains a working Dockerfile with supporting scripts, and a simple docker-compose.yml for quick testing.

Unable to use JConsole with Tomcat running as windows service

I am running tomcat 6.0.18 as a windows service. In the service applet the jvm is configured default, i.e. it is using jvm.dll of the JRE.
I am trying to monitor this application with JConsole but cannot connect to it locally. I added the parameter -Dcom.sun.management.jmxremote (which works when starting tomcat with the start.bat script). But the jvm does not seem to pick up the parameter.
There's a nice GUI to edit the options, no need to muck around in the registry.
Open up the C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\tomcat6.exe (or just double-click on the monitor icon in the task bar). Go to the Java pane, add the following to the list of arguments, and restart Tomcat.
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8086
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
Then you can connect with JConsole or the newer VisualVM.
Here's the prescribed way for changing jvmoptions & interacting with the service:
http://tomcat.apache.org/tomcat-5.5-doc/windows-service-howto.html
I would try going into your registry at HKLM/Software/Apache Software Foundation/Procrun 2.0//Parameters/Java and editing the "Options" multi-string value directly.
If Tomcat is running as a Windows service, and you want to attach to the JVM locally, you need to run VisualVM or JConsole as the System account. You can use Sysinternals PsExec.exe to accomplish this.
psexec.exe -i -s c:\visualvm\bin\visualvm.exe
I'm posting it mainly to record this information to myself, I haven't validated it - but this is supposed to work as well:
http://mysqlandsqlserver.blogspot.com/2010/02/jconsolejmap-and-tomcat-as-windows.html
There is still a rather simple way to connect JConsole to Java process started as Windows Service using the local mode which I discovered here.
Basically it says that in order to connect to Java process launched as a Windows Service you need to launch JConsole as a Windows Service (you can do it using windows native api or using any wrapper like yajsw.)
By the way, this will free you from restarting the Java Process which was critical for me.
Add the following near the top of your catalina.bat
set JAVA_OPTS=%JAVA_OPTS% -Dcom.sun.management.jmxremote ^
-Dcom.sun.management.jmxremote.port=8086 ^
-Dcom.sun.management.jmxremote.ssl=false ^
-Dcom.sun.management.jmxremote.authenticate=false
Stop and restart tomcat (obviously)
Run jconsole.exe. If your tomcat is running as service, then run jconsole.exe as administrator.
Select Remote Process and enter localhost:8086

Categories

Resources