We face intermittent High Thread Counts followed by JVM's slowdown and during the same timeframe, we could find the below mentioned exceptions. Please suggest if these are SYMPTOMS or CAUSES and help understand the remediation steps.
Mar 23, 2022 2:44:13 PM org.apache.catalina.connector.CoyoteAdapter checkRecycled
INFO: Encountered a non-recycled response and recycled it forcedly.
org.apache.catalina.connector.CoyoteAdapter$RecycleRequiredException
Mar 23, 2022 2:57:56 PM org.apache.coyote.AbstractProtocol$ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.NullPointerException: Inflater has been closed
Related
I am using org.asynchttpclient to post async request.
While shutting down tomcat, I am getting below logs:
SEVERE: The web application [/test] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Jul 04, 2017 10:53:00 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/test] appears to have started a thread named [AsyncHttpClient-2-1] but has failed to stop it. This is very likely to create a memory leak.
Jul 04, 2017 10:53:00 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/test] appears to have started a thread named [AsyncHttpClient-2-2] but has failed to stop it. This is very likely to create a memory leak.
Jul 04, 2017 10:53:00 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/test] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#71513074]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap#4aecc2e8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
How to close these threads to prevent memory leak.
This is really more of a warning and in most times can be ignored. You will see this error when the context is reloaded or shut down. If you are shutting down the server then the JVM will be shutdown also and the os will clean up any memory.
If you want to try and call close on the AsyncHttpClient which implements Closable it may stop this issue. You will probably have to put a wait in there to let it fully close the connection factory. You can look at the tomcat Lifecycle to add a shutdown handler to handle closing the client.
I'm working through an EJB tutorial where my client program invokes a method via remote stateless EJB to add a book. Upon exit the client retrieves and prints all the books from the EJB (I know it's not a good idea to store data in a list within a stateless EJB). All of this works fine, except the initial RMI also returns the following exception (I've included the full output from the client test as well).
Client output:
Nov 29, 2016 11:34:29 PM org.jboss.ejb.client.EJBClient <clinit>
INFO: JBoss EJB Client version 2.1.4.Final
**********************
Welcome to Book Store
**********************
Options
1. Add Book
2. Exit
Enter Choice: 1
Enter book name: Some book
Nov 29, 2016 11:34:44 PM org.xnio.Xnio <clinit>
INFO: XNIO version 3.4.0.Final
Nov 29, 2016 11:34:44 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.4.0.Final
Nov 29, 2016 11:34:44 PM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 4.0.21.Final
Nov 29, 2016 11:34:45 PM org.jboss.ejb.client.remoting.VersionReceiver handleMessage
INFO: EJBCLIENT000017: Received server version 2 and marshalling strategies [river]
Nov 29, 2016 11:34:45 PM org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver associate
INFO: EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext#4f7d0008, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection#271053e1,channel=jboss.ejb,nodename=slave01:server01]} on channel Channel ID 87a6ebda (outbound) of Remoting connection 64bfbc86 to /127.0.0.1:8133 of endpoint "client-endpoint" <64bf3bbf>
Nov 29, 2016 11:34:45 PM org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager getEJBReceiver
INFO: Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='slave01:server01', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='0.0.0.0', destinationPort=8080}], resolvedDestination=[Destination address=0.0.0.0, destination port=8080]} in cluster ejb
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.xnio.nio.WorkerThread$ConnectHandle.handleReady(WorkerThread.java:321)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)
at ...asynchronous invocation...(Unknown Source)
at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:294)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:430)
at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:153)
at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:133)
at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:78)
at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:79)
at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:469)
at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:443)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
**********************
Welcome to Book Store
**********************
Options
1. Add Book
2. Exit
Enter Choice: 2
Book(s) entered so far: 2
1. test1
2. Some book
***Using second lookup to get library stateless object***
Book(s) entered so far: 2
1. test1
2. Some book
So everything with the client, other than the exception, appears to work correctly. I suspect this issue has something to do with the zero'd out node addresses, but I'm not certain. The client properties file is below (in case that configuration is incorrect).
jboss-ejb-clients.properties:
endpoint.name=client-endpoint
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
invocation.timeout=3000
reconnect.tasks.timeout=2000
# User Credentials
username=user
password=pass
# Remote Connections
remote.connections=h1,h2
remote.connection.h1.host=127.0.0.1
remote.connection.h1.port=8133
remote.connection.h1.username=user
remote.connection.h1.password=pass
remote.connection.h2.host=127.0.0.1
remote.connection.h2.port=8134
remote.connection.h2.username=user
remote.connection.h2.password=pass
# Cluster
remote.clusters=ejb
remote.cluster.ejb.connect.timeout=2500
remote.cluster.ejb.max-allowed-connected-nodes=2
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED=false
remote.cluster.ejb.username=user
remote.cluster.ejb.password=pass
After extensive research (and a good amount of trial and error with test code), I found a book on Safari (Java EE 7 Development with WildFly) that lead me in the right direction. I had to drop the jboss-ejb-clients.properties file and add the ejb-client configuration found in the answer here to my main client class.
I'm using Spring 4.0, tomcat 8, spring-data-jpa 1.7.0.RELEASE, hibernate 4.1, mysql 5.6.21 Homebrew. I'm new in java, just moved form C++ background. I assume there must be some resource leaking. I have to close or free something before I exist the app. But no idea what to do. Please advise me!
Every time I restart I got following error:-
Nov 20, 2014 2:01:01 PM org.apache.catalina.loader.WebappClassLoaderBase
clearReferencesThreads
WARNING: The web application [MyJap] appears to have started a thread named [MySQL
Statement Cancellation Timer] but has failed to stop it. This is very likely to create a
memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.util.TimerThread.mainLoop(Timer.java:526)
java.util.TimerThread.run(Timer.java:505)
Nov 20, 2014 2:01:01 PM org.apache.coyote.AbstractProtocol start
I have an active/passive nodes with Tomcat using heartbeat. When I shutdown the active node, Tomcat on passive node starts. This is a piece of the starting trace:
INFO: Initializing Coyote HTTP/1.1 on http-8080 May 22, 2014
7:37:43 PM org.apache.catalina.startup.Catalina load INFO:
Initialization processed in 366 ms May 22, 2014 7:37:43 PM
org.apache.catalina.realm.JAASRealm setContainer INFO: Set JAAS
app name VentusProxy May 22, 2014 7:37:59 PM
org.apache.catalina.mbeans.JmxRemoteLifecycleListener createServer
It takes 15 seconds to initialize JAAS, that means 15 seconds more without service.
I don't use JAAS at all in my application, so I'd like to disable it or, at least, try to reduce this 15 seconds.
Can anybody tell me if this is possible and how?
I have been searching a lot but have not found a perfect one which can help me to solve this problem.
I have web services and am generating the stub using JAX-WS.
To access the methods of web services I have wriiten a class in which all the methods are static like
public static String getLocation()
{
//call to the web service
}
I am specifying static because I want to confirm this is not the root cause of my problem.
Now when I am checking the logs in the Tomcat directory the catilina log shows some thins like this...This error is occured when I Startup or shutdown the tomcat server
Mar 18, 2010 11:13:07 PM org.apache.catalina.core.ApplicationContext log
INFO: HTMLManager: stop: Stopping web application at '/testWeb'
Mar 18, 2010 11:13:07 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [leakingThread] but has failed to stop it. This is very likely to create a memory leak.
Another error I am seeing is
SEVERE: Unable to determine string representation of value of type [com.sun.xml.stream.writers.XMLStreamWriterImpl]
java.lang.UnsupportedOperationException
at com.sun.xml.stream.writers.XMLStreamWriterImpl.entrySet (XMLStreamWriterImpl.java:2134)
at java.util.AbstractMap.toString(AbstractMap.java:478)
at org.apache.catalina.loader.WebappClassLoader.clearThreadLocalMap(WebappClassLoader.java:2433)
at org.apache.catalina.loader.WebappClassLoader.clearReferencesThreadLocals(WebappClassLoader.java:2349)
at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClassLoader.java:1921)
at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.java:1833)
at org.apache.catalina.loader.WebappLoader.stop(WebappLoader.java:740)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4920)
at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:936)
Please can any one help me to clear these errors.
Thanks in advance.
My diagnossis: you have a Map implementation in a thread local, and this map doesn't support the operation entrySet, which is triggered by Map#toString. To be precise, your exception is thrown from this line of code in com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.
Tomcat's code that clears the thread local is quite unfortunately written to unconditionally call toString on objects just to be able to log them if the debug level is on.
If you can't get rid of using a thread-local for this, you may have quite some trouble working around this problem.
Your thread leak, by the way, is very probably the result of failed cleanup due to the above error.