Develop Reference

JBoss 7.1.1 JVM monitoring using SNMP and PRTG

Recently I had given task in my company to monitor JVM of JBoss application server 7.1.1 using PRTG on Windows plateform. After searching I came to know to monitor using PRTG snmp enabling is required. JVM starts and listen on specified port and PRTG communicates using snmp to get the jvm information like heap , non heap , thread etc information.

To achieve this, I followed these steps:
Configured Jboss to start with snmp agent enabled.
Added following lines in standalone.conf.bat
set "JAVA_OPTS=%JAVA_OPTS% -Dcom.sun.management.snmp.port=1161 -Dcom.sun.management.snmp.interface=0.0.0.0 -Dcom.sun.management.snmp.acl=false"
That was giving me error of LogManager on start up.To resolve this I have added following lines in the same file.
set "JAVA_OPTS=%JAVA_OPTS%
-Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:%JBOSS_HOME%\jboss-logmanager-1.2.2.GA.jar;%JBOSS_HOME%\jboss-logmanager-log4j-1.0.0.GA.jar;%JBOSS_HOME%\log4j-1.2.16.jar"
That solved my problem and the server started successfully.
For testing connection I have used following command.
snmpwalk -v2c -c public 127.0.0.1:1161
SNMPv2-SMI::enterprises.42.2.145.3.163.1.1.4
PRTG configuration was done by system administrator. :-)

Eclipse Java EE + Bitnami Tomcat Stack

This is my situation:
Eclipse ide that i use to develop java web apps.
Tomcat from apache.
Tomcat stack from bitnami.
OS Windows 8
If i deploy and debug to the apache tomcat all work without problem.
If i try the same thing with the bitnami stack, i see the exact same output from the console, like it is starting well, but actually it doesn't and it gets to the timeout saying it was unable to start withing 45 seconds.
I tried to increase the timeout but that's not the problem.
In both cases the Server Location is set to Use Tomcat installation, and i added my project to the source, everything else in the server config is default.
I'm not an expert of tomcat and java webapp deploying, and i need to get it working with the bitnami stack.
Any hint will be appreciated.

Ok i solved it, seems more a problem from eclipse.
In the server configuration i noticed the HTTP port was not listed and it was commented in server.xml
Could this be because the bitnami stack uses port 80 instead of 8080?
Anyway setting the port 80 in server.xml solved the problem.

Monitor Websphere Liberty Profile with JConsole

I'm trying to monitor my WLP v8.5.5 with JConsole.
My Liberty profile is up and running and has SSL properlly configured.
I also have configured as features the monitor-1.0 and restConnector-1.0.
After that I sftp'ed from my WLP the restConnector.jar to my machine and created a keystore on my machine as described here
After that I'm trying to start JConsole with this command:
jconsole -J-Djava.class.path="C:\Program Files\Java\jdk1.7.0_67\lib\jconsole.jar;
C:\Program Files\Java\jdk1.7.0_67\lib\tools.jar;
<whereIDownloaded>\restConnector.jar"
-J-Djavax.net.ssl.keyStore="<locationToCreated>\keystore.jks"
-J-Djavax.net.ssl.keyStorePassword=<password>
-J-Djavax.net.ssl.keyStoreType=jks
-J-Dcom.ibm.ws.jmx.connector.client.disableURLHostnameVerification=true
However, when I try to connect to my WLP via remote with the string:
service:jmx:rest://<ip>:<httpsPort>/IBMJMXConnectorREST
As a credential I'm passing a user configured on <administrator-role>.
I get a JConsole error saying:
Secure connection failed. Retry insecurely?
I don't get why I'm seeing this error, and I don't see any logs to see why it failed.
My next step if getting the keystore from WLP and try that on my machine but I don't think that makes much sense.
Does anyone have any suggestion on where should I look for logs or on that I'm doing wrong?
Thanks in advance!
Reference:
http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.wlp.nd.doc/ae/twlp_mon.html?cp=SSAW57_8.5.5%2F1-3-11-0-6&lang=en

Ensure you don't have localConnector-1.0 feature. If you have that, please remove it. You can find more details here Remote monitoring of Liberty with Health Center. Although it discusses Health Center connection, I was testing using JConsole also.
UPDATE
You might be invoking console with incorrect params - it should be trustStore not keystore see below. And for start use key.jks copied from Liberty (as you need Liberty certificate as trusted)
jconsole -J-Djava.class.path=%JAVA_HOME%/lib/jconsole.jar;
%JAVA_HOME%/lib/tools.jar;
%WLP_HOME%/clients/restConnector.jar
-J-Djavax.net.ssl.trustStore=key.jks
-J-Djavax.net.ssl.trustStorePassword=Liberty
-J-Djavax.net.ssl.trustStoreType=jks
For details check Configuring secure JMX connection to the Liberty

As an additional note - you must not have a webAppSecurity entry with loginFormURL set in your server.xml as this will cause the JMX login to fail (e.g., <webAppSecurity loginFormURL="login.html"/> is problematic). Each individual WAR needs to set their own login-config and not rely on any global method of configuring logins.
`

How can I monitor/log Tomcat's thread pool?

I have a Tomcat installation where I suspect the thread pool may be decreasing over time due to threads not being properly released. I get an error in catalina.out when maxthreads is reached, but I would like to log the number of threads in use to a file every five minutes so I can verify this hypothesis. Would anyone please be able to advise how this can be be done?
Also in this installation there is no Tomcat manager, it appears whoever did the original installation deleted the manager webapp for some reason. I'm not sure if manager would be able to do the above or if I can reinstall it without damaging the existing installation? All I really want to do is keep track of the thread pool.
Also, I noticed that maxthreads for Tomcat is 200, but the max number of concurrent connections for Apache is lower (Apache is using mod_proxy and mod_proxy_ajp (AJP 1.3) to feed Tomcat). That seems wrong too, what is the correct relationship between these numbers?
Any help much appreciated :D
Update: Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.
Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.

Direct JMX access
Try adding this to catalina.sh/bat:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=5005
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
UPDATE: Alex P suggest that the following settings might also be required in some situations:
-Dcom.sun.management.jmxremote.host=localhost
This enables remote anonymous JMX connections on port 5005. You may also consider JVisualVM which is much more please and allows to browse JMX via plugin.
What you are looking for is Catalina -> ThreadPool -> http-bio-8080 -> various interesting metrics.
JMX proxy servlet
Easier method might be to use Tomcat's JMX proxy servlet under: http://localhost:8080/manager/jmxproxy. For instance try this query:
$ curl --user tomcat:tomcat http://localhost:8080/manager/jmxproxy?qry=Catalina:name=%22http-bio-8080%22,type=ThreadPool
A little bit of grepping and scripting and you can easily and remotely monitor your application. Note that tomcat:tomcat is the username/password of user having manager-jmx role in conf/tomcat-users.xml.

You can deploy jolokia.war and then retrieve mbeans values in JSON (without the manager):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool?ignoreErrors=true
If you want only some values (currentThreadsBusy, maxThreads, currentThreadCount, connectionCount):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool/currentThreadsBusy,maxThreads,currentThreadCount,connectionCount?ignoreErrors=true
{
request: {
mbean: "Catalina:name="http-nio-8080",type=ThreadPool",
attribute: [
"currentThreadsBusy",
"maxThreads",
"currentThreadCount",
"connectionCount"
],
type: "read"
},
value: {
currentThreadsBusy: 1,
connectionCount: 4,
currentThreadCount: 10,
maxThreads: 200
},
timestamp: 1490396960,
status: 200
}
Note: This example works on Tomcat7 +.

For a more enterprise solution. I have been using New Relic in our production environment.
This provides a graph of the changes to the threadpool over time.

There are cheaper tools out meanwhile: I am using this jar here: https://docs.cyclopsgroup.org/jmxterm
You can automate it via shell/batch scripts. I regexed the output and let prometheus poll it for displaying it in grafana.

How can I make "jconsole" work with Websphere 6.1?

I've deployed some Managed Beans on WebSphere 6.1 and I've managed to invoke them through a standalone client, but when I try to use the application "jconsole" distributed with the standard JDK can can't make it works.
Has anyone achieved to connect the jconsole with WAS 6.1?
IBM WebSphere 6.1 it's supossed to support JSR 160 JavaTM Management Extensions (JMX) Remote API. Furthermore, it uses the MX4J implementation (http://mx4j.sourceforge.net). But I can't make it works with neither "jconsole" nor "MC4J".
I have the Classpath and the JAVA_HOME correctly setted, so the issue it's not there.

WebSphere's support for JMX is crap. Particularly, if you need to connect to any secured JMX beans. Here's an interesting tidbit, their own implementation of jConsole will not connect to their own JVM. I have had a PMR open with IBM for over a year to fix this issue, and have gotten nothing but the runaround. They clearly don't want to fix this issue.
The only way I have been able to invoke remote secured JMX beans hosted on WebSphere has been to implement a client using the "WebSphere application client". This is basically a stripped down app server used for stuff like this.
Open a PMR with IBM. Perhaps if more people report this issue, they will actually fix it.
Update: You can run your application as a WebSphere Application Client in RAD. Open the run menu, then choose "Run...". In the dialog that opens, towards the bottom on the left hand side, you will see "WebSphere v6.1 Application Client". I'm not sure how to start and Application Client outside of RAD.

IT WORKS !
http://issues.apache.org/jira/browse/GERONIMO-4534;jsessionid=FB20DD5973F01DD2D470FB9A1B45D209?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
1) Change the config.xml and start the server.
-see here how to change config.xml: http://publib.boulder.ibm.com/wasce/V2.1.0/en/working-with-jconsole.html
2) start the jconsole with : jconsole -J-Djavax.net.ssl.keyStore=%GERONIMO_HOME%\var\security\keystores\geronimo-default -J-Djavax.net.ssl.keyStorePassword=secret -J-Djavax.net.ssl.trustStore=%GERONIMO_HOME%\var\security\keystores\geronimo-default -J-Djavax.net.ssl.trustStorePassword=secret -J-Djava.class.path=%JAVA_HOME%\lib\jconsole.jar;%JAVA_HOME%\lib\tools.jar;%GERONIMO_HOME%\repository\org\apache\geronimo\framework\geronimo-kernel\2.1.4\geronimo-kernel-2.1.4.jar
[or your version of geronimo-kernel jar]
3) in the jconsole interface->advanced, input:
JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/JMXSecureConnector
user name: system
password: manager
4) click the connect button.

If you want the WebSphere MBeans this one works for me:
The key is to configure the classpath and the security properly.
in one line:
jconsole -J-Dwas.install.root=C:/was61 -J-Djava.ext.dirs=C:/was61/plugins;C:/was61/plugins/com.ibm.ws.security.crypto_6.1.0;C:/was61/lib;C:/was61/java/jre/lib/ext -J-Dcom.ibm.SSL.ConfigURL="file:../../properties/ssl.client.props" -J-Dcom.ibm.CORBA.ConfigURL="file:../../properties/sas.client.props" service:jmx:iiop://host:port/jndi/JMXConnector
where port = bootstrap port ex: (2809)
Be careful when setting the sas and the ssl props.
Robert

I have successfully connected to ActiveMQ and ServiceMix using the JConsole. Does WAS 6.1 use Java Management Extension (JMX) technology? JMX is required for JConsole.
If your path is set correctly it should work fine. On windows you go to System Properties -> Advanced Tab -> Environment Variables. Have your JAVA_HOME System variable set to the path of your JDK or JRE and your Path variable with %JAVA_HOME%/bin added somewhere in there. Then all you need to do is go to Start->Run->JConsole. Select the correct Process Name and your done.
Where are you having problems at? I hope this helps.
Edit:
Here is the Java Doc's on JConsole.

Hmm... I know that WebSphere is kind of hard to configure. Thats part of the reason we used ServiceMix for our ESB. Maybe its not enabled by default in WebSphere and you would have to turn it on in the config somewhere.

Websphere 6.1 does not support the JConsole for some reason even though it fully implements the JMS specs. Seems to be a week area at the moment. Your best bet is to look at the Admin client to implement you own console.

You all seem to be incorrect. I am running Websphere 6.1.041 , using JDK 1.5 , and I just started up Jconsole and used the "simple connect" tab to connect to localhost with port=0 and without a username and password and it works fine.