I have a Tomcat installation where I suspect the thread pool may be decreasing over time due to threads not being properly released. I get an error in catalina.out when maxthreads is reached, but I would like to log the number of threads in use to a file every five minutes so I can verify this hypothesis. Would anyone please be able to advise how this can be be done?
Also in this installation there is no Tomcat manager, it appears whoever did the original installation deleted the manager webapp for some reason. I'm not sure if manager would be able to do the above or if I can reinstall it without damaging the existing installation? All I really want to do is keep track of the thread pool.
Also, I noticed that maxthreads for Tomcat is 200, but the max number of concurrent connections for Apache is lower (Apache is using mod_proxy and mod_proxy_ajp (AJP 1.3) to feed Tomcat). That seems wrong too, what is the correct relationship between these numbers?
Any help much appreciated :D
Update: Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.
Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.
Direct JMX access
Try adding this to catalina.sh/bat:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=5005
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
UPDATE: Alex P suggest that the following settings might also be required in some situations:
-Dcom.sun.management.jmxremote.host=localhost
This enables remote anonymous JMX connections on port 5005. You may also consider JVisualVM which is much more please and allows to browse JMX via plugin.
What you are looking for is Catalina -> ThreadPool -> http-bio-8080 -> various interesting metrics.
JMX proxy servlet
Easier method might be to use Tomcat's JMX proxy servlet under: http://localhost:8080/manager/jmxproxy. For instance try this query:
$ curl --user tomcat:tomcat http://localhost:8080/manager/jmxproxy?qry=Catalina:name=%22http-bio-8080%22,type=ThreadPool
A little bit of grepping and scripting and you can easily and remotely monitor your application. Note that tomcat:tomcat is the username/password of user having manager-jmx role in conf/tomcat-users.xml.
You can deploy jolokia.war and then retrieve mbeans values in JSON (without the manager):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool?ignoreErrors=true
If you want only some values (currentThreadsBusy, maxThreads, currentThreadCount, connectionCount):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool/currentThreadsBusy,maxThreads,currentThreadCount,connectionCount?ignoreErrors=true
{
request: {
mbean: "Catalina:name="http-nio-8080",type=ThreadPool",
attribute: [
"currentThreadsBusy",
"maxThreads",
"currentThreadCount",
"connectionCount"
],
type: "read"
},
value: {
currentThreadsBusy: 1,
connectionCount: 4,
currentThreadCount: 10,
maxThreads: 200
},
timestamp: 1490396960,
status: 200
}
Note: This example works on Tomcat7 +.
For a more enterprise solution. I have been using New Relic in our production environment.
This provides a graph of the changes to the threadpool over time.
There are cheaper tools out meanwhile: I am using this jar here: https://docs.cyclopsgroup.org/jmxterm
You can automate it via shell/batch scripts. I regexed the output and let prometheus poll it for displaying it in grafana.
Related
So apologies if what I ask is trivial but I am experimenting with Memcached and Jmeter. I have a Memcached server setup (as far as I can tell) and am able to make telnet requests to it via telnet IP PORT and additionally set and get using commands set and get appropriately.
Now point me to a different application if perhaps this is the wrong choice; but my understanding was that Jmeter should allow me to pound the server with equivalent Set and Get requests.
Unfortunately the experimental platform is a remote linux PC running Rockylinux which is similar to CentOS / RedHat to my understanding (I didn't set this part up); and as a result I do not have a GUI to launch while on the Linux PC. I have however opened Jmeter up on my local PC on windows and understand I should be able to send the test file over and run it.
I followed these instructions to try to setup a TCP sampler and set the "text to send" field as below; after doing the additional step in the link regarding the precompiler.
set tutorialspoint 0 900 9${CR}${LF}
memcached${CR}${LF}
quit${CR}${LF}
Running the above as a headless jmeter session doesn't generate any errors called [./jmeter -n -t "Sample.jmx" -l testresults.jtl"] but when I connect via telnet I'm also not seeing the value for the key "tutorialspoint" get updated. When manually doing the get and set I am seeing updates. Any ideas what I could be doing wrong? Checking the log indicates ResponseCode 200 OK as expected. Is there a good method to debug something in a Headless setup?
Thanks for your time.
I believe the easiest way is using Memcached Java Client library
Download spymemcached-2.12.3.jar and drop it to "lib" folder of your JMeter installation (or any other location in JMeter Classpath)
Restart JMeter to pick the .jar up
Add JSR223 Sampler to your test plan and use the following code snippets:
def client = new net.spy.memcached.MemcachedClient(new InetSocketAddress('your-memcached-host', your-memcached-port)) - for connecting to the server
client.set('tutorialspoint', 900, 'memcached').done to write memcached to the tutorialspoint key for 15 minutes
client.get('tutorialspoint') - to read the value of tutorialspoint key
client.shutdown() - to disconnect
More information on Groovy scripting in JMeter: Apache Groovy - Why and How You Should Use It
Demo:
Could you please assist me with the following issue.
I use Apiman version 1.2.1
FROM jboss/wildfly:9.0.2.Final
ENV APIMAN_VERSION 1.2.1.Final
I expose this version via kubernetes, as a persistent volume I use postgres in the same container. Once I create it at the first time, after this in the apiman I have added Organization/ API/.... and all necessary staff.
I press on button to publish api, and may check that it works perfect, so I use kubectl port-forward pod-name 8080:8080 and may check my gateway via browser http:localhost:8080/apiman-gateway/ORgId/bla/bla/bla/bla?givemedescriptionbyid=1.
After this one I go to console and kill apiman pod, as a result of pod restart, the same operation kubectl port-forward new-pod-name 8080:8080, and I can see that the all data of apiman like organization, apis and all other staff is already there.
But one big problem if you try to call gateway again, it tells you that:
{"responseCode":500,"message":"API not
found.","trace":"io.apiman.gateway.engine.beans.exceptions.InvalidApiException:
API not found.\n\tat
io.apiman.gateway.engine.impl.ApiRequestExecutorImpl$3.handle(ApiRequestExecutorImpl.java:278)\n\tat
io.apiman.gateway.engine.impl.ApiRequestExecutorImpl$3.handle(ApiRequestExecutorImpl.java:271)\n\tat
io.apiman.gateway.engine.impl.SecureRegistryWrapper$1.handle(SecureRegistryWrapper.java:122)\n\tat
io.apiman.gateway.engine.impl.SecureRegistryWrapper$1.handle(SecureRegistryWrapper.java:111)\n\tat
io.apiman.gateway.engine.es.CachingESRegistry.getApi(CachingESRegistry.java:116)\n\tat
io.apiman.gateway.engine.impl.SecureRegistryWrapper.getApi(SecureRegistryWrapper.java:111)\n\tat
io.apiman.gateway.engine.impl.ApiRequestExecutorImpl.execute(ApiRequestExecutorImpl.java:270)\n\tat
io.apiman.gateway.platforms.servlet.GatewayServlet.doAction(GatewayServlet.java:232)\n\tat
io.apiman.gateway.platforms.servlet.GatewayServlet.doGet(GatewayServlet.java:77)\n\tat
javax.servlet.http.HttpServlet.service(HttpServlet.java:687)\n\tat
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)\n\tat
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)\n\tat
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)\n\tat
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)\n\tat
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)\n\tat
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)\n\tat
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)\n\tat
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)\n\tat
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)\n\tat
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)\n\tat
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)\n\tat
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)\n\tat
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)\n\tat
io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)\n\tat
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)\n\tat
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)\n\tat
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)\n\tat
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)\n\tat
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)\n\tat
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)\n\tat
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)\n\tat
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)\n\tat
io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)\n\tat
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat
java.lang.Thread.run(Thread.java:745)\n"}
I can see, that by metrics of calls in api manager, all my calls can reach apiman-gateway, but I get 500 response code.
If anybody will met such error, so, I have solved it with the following steps, first of all we need to use only new version of APIMAN, list of fixes already implemented. And also apiman-gateway cannot live without elastic search to restore itself. Hence elasticsearch configuration should to be provided.
I am trying, for the first time, to run a java EE 7 web app without an IDE and struggling through the learning curve. To keep things simple I have started with Glassfish 4.1 and Derby 10.11.1.2 which is what I used for development.
My current problem is an error configuring the connection pool on the Glassfish web interface. The error is 'Connection authentication failure occurred. Reason: Userid or password invalid'. However I have ij runing in a dos prompt and the Glashfish web console in Chrome.
from ij I can type 'connect 'jdbc:derby:localhost:1527/databasename;user=username;password=password'; and I connect fine and can look at tables, etc. In the Glassfish web console I have name/value pairs with the same 'user' and 'password' values and it fails (exact same letters & case). I've tried changing the 'databaseName' property to make sure the error wasn't misleading and confirmed it has found the database correctly.
At face value, the error message seems wrong as I have proven from the DOS window. I am sure I am doing something wrong but am lost at how to diagnose it. Any suggestions?
More details of the steps taken: (Although I am sure some of these steps superceed others my lack of success has me in the mindset that redundancy is better than omission - all-in-all it seems much more complicated than it needs to be for a basic setup). My target is Wildfly but Glassfish documentation is better so I figured I would start there.
relevant Windows environment variables:
JAVA_HOME=C:\Program Files\Java\jdk1.8.0_40
DERBY_HOME=%JAVA_HOME%\db
DERBY_INSTALL=%DERBY_HOME%
JAVADB_HOME=%DERBY_HOME%
GLASSFISH=C:\Program Files\Java\GlassFish-4.1
Path=%JAVA_HOME%\bin;%DERBY_HOME%\bin;%GLASSFISH%\bin;%GLASSFISH%\glassfish
CLASSPATH=%DERBY_HOME%\lib\derbynet.jar;%DERBY_HOME%\lib\derbytools.jar;%DERBY_HOME%\derbyclient.jar;%DERBY_HOME%\lib\derbyrun.jar
WILDFLY_HOME=C:\program files\java\wildfly\8.2.0.Final
sysinfo shows all the right paths and versions
Copied many of the derby jars from %derby_home%\lib to %glassfish%\glassfish\domains\domain1\lib\ext per a post by BalusC. I think classpath duplicates this it but can't hurt
Started Domain 'asadmin start-domain' defaulting to domain 1
Started DB by changing directories and using java. Wanted to use 'asadmin start-database --dbhome DB path' but this always started the wrong version. Glassfish and JDK both package derby/javadb and I couldn't figure out how to upgrade Glassfish or start the correct version any other way
cd \users\john\.netbeans-derby (root of the DB created by Netbeans)
Java –jar “%derby_home%\lib\derbyrun.jar" server start
connect browser to 'localhost:4848' to create JDBC connection pool and resource
connection pool: pool name=connectionPool, resource type=javax.sql.DataSource, DB driver Vendor=Derby, introspect was not enabled (found very little documentation to explain what it is) - step 2 was left as defaults
editing connectionPool - enabled Ping on General tab to identify errors, Additional Properties tab: changed User to DB username, left other attributes as default, added... password=db password, databaseName=ClubScoring
errors were presented along the way which prompted setting each attribute. No error was presented for lack of URL but tested steps below with and without URL=jdbc:derby://localhost:1527/ClubScoring
jdbc resources. Added jdbc/ClubScoring, pool=connectionPool => selected and enabled
starting db from correct directory using 'asadmin start-database' does let the application find the correct DB but it crashes since the DB was created with version 10.11.1.2 and this command starts the Glassfish version of 10.10.2.0
Sorry for making this so long but I haven't found anywhere that lays out all of the steps in a simple fashion and figured the details would save time in the long run.
I was dealing with this issue today and found a solution:
https://db.apache.org/derby/docs/10.14/ref/rrefattribsecmech.html
Basically the only security options allowable (as far as 10.14 is concerned) is 3, 4 and 9.
4 is the default and means "Just UserID". You would think this means that as long as a UserID is given, you could connect. But having both set appears to be an issue.
In order to use a (clear text) password and userID, you have to specify 3. It works also if the attribute is deleted or 0 as well but I don't think you get any security that way.
We are tracking down some memory issues in our application and we have visibility into the size of our sessions where the problem seems to be. It is only affecting certain sessions where they seem to balloon out of control and we'd like to have the ability to invalidate those sessions more-or-less 'by hand' to reclaim that memory . Is there a way to do this through JMX? We are using JBoss 4.5.2 .
Thanks in advance.
The answer is yes, you can.
--this gets the web module MBeans loaded in JBoss--
/opt/lib/jboss-4.2.3.GA/bin/twiddle.sh -s <--ServerIP-->:1099 query 'jboss.web:*' | grep "type=Manager"
--Once you have that you can get the active sessions from that MBean--
/opt/lib/jboss-4.2.3.GA/bin/twiddle.sh -s 172.16.0.216:1099 invoke "<--MBean-->" listSessionIds
--And finally, once you have that list, you can pick out the session of interest and expire it--
/opt/lib/jboss-4.2.3.GA/bin/twiddle.sh -s <--ServerIP-->:1099 invoke "<--MBean-->" expireSession <--SessionID-->
You would replace <--ServerIP--> with the ip the server is bound on...it need not be local. This can be run remotely as long as you have access to port 1099.
<--MBean--> would be replaced with one of the results of the first query.
<--SessionID--> would be replaced with one of the session ids from the second command.
Also you would replace /opt/lib/jboss-4.2.3.GA/bin/twiddle.sh with the location of twiddle on your machine. It is included in JBoss.
Thanks Ichorus. Your solution helped me with at least some way to see all the active sessions on the server at any given time.
Btw...Here are couple of things needed before running twiddle.sh
Make sure JAVA_HOME and JBOSS_HOME are set in the environment
And Make Sure JBOSS_CLASSPATH is either not set or has all the libs given in twiddle.sh.i.e {JBOSS_HOME}\client\jbossall-client.jar;{JBOSS_HOME}\client\getopt.jar;{JBOSS_HOME}\client\log4j.jar; {JBOSS_HOME}\lib\jboss-jmx.jar
Other wise you would run into ClassNotFound exceptions
Sudheer
I've deployed some Managed Beans on WebSphere 6.1 and I've managed to invoke them through a standalone client, but when I try to use the application "jconsole" distributed with the standard JDK can can't make it works.
Has anyone achieved to connect the jconsole with WAS 6.1?
IBM WebSphere 6.1 it's supossed to support JSR 160 JavaTM Management Extensions (JMX) Remote API. Furthermore, it uses the MX4J implementation (http://mx4j.sourceforge.net). But I can't make it works with neither "jconsole" nor "MC4J".
I have the Classpath and the JAVA_HOME correctly setted, so the issue it's not there.
WebSphere's support for JMX is crap. Particularly, if you need to connect to any secured JMX beans. Here's an interesting tidbit, their own implementation of jConsole will not connect to their own JVM. I have had a PMR open with IBM for over a year to fix this issue, and have gotten nothing but the runaround. They clearly don't want to fix this issue.
The only way I have been able to invoke remote secured JMX beans hosted on WebSphere has been to implement a client using the "WebSphere application client". This is basically a stripped down app server used for stuff like this.
Open a PMR with IBM. Perhaps if more people report this issue, they will actually fix it.
Update: You can run your application as a WebSphere Application Client in RAD. Open the run menu, then choose "Run...". In the dialog that opens, towards the bottom on the left hand side, you will see "WebSphere v6.1 Application Client". I'm not sure how to start and Application Client outside of RAD.
IT WORKS !
http://issues.apache.org/jira/browse/GERONIMO-4534;jsessionid=FB20DD5973F01DD2D470FB9A1B45D209?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
1) Change the config.xml and start the server.
-see here how to change config.xml: http://publib.boulder.ibm.com/wasce/V2.1.0/en/working-with-jconsole.html
2) start the jconsole with : jconsole -J-Djavax.net.ssl.keyStore=%GERONIMO_HOME%\var\security\keystores\geronimo-default -J-Djavax.net.ssl.keyStorePassword=secret -J-Djavax.net.ssl.trustStore=%GERONIMO_HOME%\var\security\keystores\geronimo-default -J-Djavax.net.ssl.trustStorePassword=secret -J-Djava.class.path=%JAVA_HOME%\lib\jconsole.jar;%JAVA_HOME%\lib\tools.jar;%GERONIMO_HOME%\repository\org\apache\geronimo\framework\geronimo-kernel\2.1.4\geronimo-kernel-2.1.4.jar
[or your version of geronimo-kernel jar]
3) in the jconsole interface->advanced, input:
JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/JMXSecureConnector
user name: system
password: manager
4) click the connect button.
If you want the WebSphere MBeans this one works for me:
The key is to configure the classpath and the security properly.
in one line:
jconsole -J-Dwas.install.root=C:/was61 -J-Djava.ext.dirs=C:/was61/plugins;C:/was61/plugins/com.ibm.ws.security.crypto_6.1.0;C:/was61/lib;C:/was61/java/jre/lib/ext -J-Dcom.ibm.SSL.ConfigURL="file:../../properties/ssl.client.props" -J-Dcom.ibm.CORBA.ConfigURL="file:../../properties/sas.client.props" service:jmx:iiop://host:port/jndi/JMXConnector
where port = bootstrap port ex: (2809)
Be careful when setting the sas and the ssl props.
Robert
I have successfully connected to ActiveMQ and ServiceMix using the JConsole. Does WAS 6.1 use Java Management Extension (JMX) technology? JMX is required for JConsole.
If your path is set correctly it should work fine. On windows you go to System Properties -> Advanced Tab -> Environment Variables. Have your JAVA_HOME System variable set to the path of your JDK or JRE and your Path variable with %JAVA_HOME%/bin added somewhere in there. Then all you need to do is go to Start->Run->JConsole. Select the correct Process Name and your done.
Where are you having problems at? I hope this helps.
Edit:
Here is the Java Doc's on JConsole.
Hmm... I know that WebSphere is kind of hard to configure. Thats part of the reason we used ServiceMix for our ESB. Maybe its not enabled by default in WebSphere and you would have to turn it on in the config somewhere.
Websphere 6.1 does not support the JConsole for some reason even though it fully implements the JMS specs. Seems to be a week area at the moment. Your best bet is to look at the Admin client to implement you own console.
You all seem to be incorrect. I am running Websphere 6.1.041 , using JDK 1.5 , and I just started up Jconsole and used the "simple connect" tab to connect to localhost with port=0 and without a username and password and it works fine.