I'm running a Tomcat 8.5.3 on Windows Server 2008R2 and Java 1.8.0_92.
The process is consuming a lot of CPU (~ 50% from 4 CPUs).
JTop shows that the two most consuming threads are, by far, https-jsse-nio-443-ClientPoller-0 and https-jsse-nio-443-ClientPoller-1.
The threads are mainly looping on thes four stacktraces :
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked sun.nio.ch.Util$2#2a3e6629
- locked java.util.Collections$UnmodifiableSet#7cdb1cd3
- locked sun.nio.ch.WindowsSelectorImpl#13dc3a00
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:791)
java.lang.Thread.run(Thread.java:745)
.
sun.nio.ch.WindowsSelectorImpl$SubSelector.processFDSet(WindowsSelectorImpl.java:345)
sun.nio.ch.WindowsSelectorImpl$SubSelector.processSelectedKeys(WindowsSelectorImpl.java:315)
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$2900(WindowsSelectorImpl.java:278)
sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:495)
sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked sun.nio.ch.Util$2#6f4350d0
- locked java.util.Collections$UnmodifiableSet#36157c3f
- locked sun.nio.ch.WindowsSelectorImpl#120cc3aa
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:791)
java.lang.Thread.run(Thread.java:745)
.
sun.nio.ch.WindowsSelectorImpl.resetWakeupSocket0(Native Method)
sun.nio.ch.WindowsSelectorImpl.resetWakeupSocket(WindowsSelectorImpl.java:473)
- locked java.lang.Object#450e5040
sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:174)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked sun.nio.ch.Util$2#6f4350d0
- locked java.util.Collections$UnmodifiableSet#36157c3f
- locked sun.nio.ch.WindowsSelectorImpl#120cc3aa
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:791)
java.lang.Thread.run(Thread.java:745)
.
java.lang.Object.notifyAll(Native Method)
sun.nio.ch.WindowsSelectorImpl$StartLock.startThreads(WindowsSelectorImpl.java:189)
- locked sun.nio.ch.WindowsSelectorImpl$StartLock#1c512d03
sun.nio.ch.WindowsSelectorImpl$StartLock.access$300(WindowsSelectorImpl.java:181)
sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:153)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked sun.nio.ch.Util$2#2a3e6629
- locked java.util.Collections$UnmodifiableSet#7cdb1cd3
- locked sun.nio.ch.WindowsSelectorImpl#13dc3a00
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:791)
java.lang.Thread.run(Thread.java:745)
Any idea ?
I had a similar problem on Tomcat 8.5.4 / Linux where I saw very high cpu usage on the poller threads (eg, https-jsse-nio-443-ClientPoller-0 and 1). Upgrading to 8.5.5 seems to have resolved the issue.
May have been this bug: https://bz.apache.org/bugzilla/show_bug.cgi?id=60030
Related
Why does StandardPBEByteEncryptor lock an object?
"pool-2-thread-115" - Thread t#169
java.lang.Thread.State: BLOCKED
at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.decrypt(StandardPBEByteEncryptor.java:1035)
- waiting to lock <55a2b29b> (a javax.crypto.Cipher) owned by "pool-2-thread-114" t#168
at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.decrypt(StandardPBEStringEncryptor.java:725)
at org.jasypt.util.text.BasicTextEncryptor.decrypt(BasicTextEncryptor.java:112)
at com.ahmetk.tims.statcalculator.kafka.consumer.XMPPTxnLogKafkaClient.process(XMPPTxnLogKafkaClient.java:129)
at com.ahmetk.tims.statcalculator.kafka.consumer.XMPPTxnLogKafkaClient.lambda$processMessage$0(XMPPTxnLogKafkaClient.java:107)
at com.ahmetk.tims.statcalculator.kafka.consumer.XMPPTxnLogKafkaClient$$Lambda$227/859670158.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- locked <21b2529d> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"pool-2-thread-114" - Thread t#168
java.lang.Thread.State: RUNNABLE
at sun.security.provider.DigestBase.engineDigest(DigestBase.java:158)
at java.security.MessageDigest$Delegate.engineDigest(MessageDigest.java:592)
at java.security.MessageDigest.digest(MessageDigest.java:365)
at com.sun.crypto.provider.PBES1Core.deriveCipherKey(PBES1Core.java:272)
at com.sun.crypto.provider.PBES1Core.init(PBES1Core.java:244)
at com.sun.crypto.provider.PBEWithMD5AndDESCipher.engineInit(PBEWithMD5AndDESCipher.java:221)
at javax.crypto.Cipher.init(Cipher.java:1394)
at javax.crypto.Cipher.init(Cipher.java:1327)
at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.decrypt(StandardPBEByteEncryptor.java:1036)
- locked <55a2b29b> (a javax.crypto.Cipher)
at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.decrypt(StandardPBEStringEncryptor.java:725)
at org.jasypt.util.text.BasicTextEncryptor.decrypt(BasicTextEncryptor.java:112)
at com.ahmetk.tims.statcalculator.kafka.consumer.XMPPTxnLogKafkaClient.process(XMPPTxnLogKafkaClient.java:129)
at com.ahmetk.tims.statcalculator.kafka.consumer.XMPPTxnLogKafkaClient.lambda$processMessage$0(XMPPTxnLogKafkaClient.java:107)
at com.ahmetk.tims.statcalculator.kafka.consumer.XMPPTxnLogKafkaClient$$Lambda$227/859670158.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Using Multi-Threaded Decryption
When we're operating on the multi-core machine we want to handle processing of decryption in parallel. To achieve a good performance we can use a PooledPBEStringEncryptor and the setPoolSize() API to create a pool of digesters. Each of them can be used by the different thread in parallel:
PooledPBEStringEncryptor encryptor = new PooledPBEStringEncryptor();
encryptor.setPoolSize(4);
encryptor.setPassword("some-random-data");
encryptor.setAlgorithm("PBEWithMD5AndTripleDES");
It's good practice to set pool size to be equal to the number of cores of the machine. The code for encryption and decryption is the same as previous ones.
Software libraries’ Versions in use: Spring 2.2.3.Release, Spring kafka 2.x , Kafka-Client 2.0.0
Java Version: OpenJdk 1.8
Platform Kafka version: Apache kafka 2.2.0 (https://docs.confluent.io/5.2.1/release-notes.html#apache-kafka-2-2-0-cp2)
Configuration: Topic has 6 partitions.
Data rate: The incoming data rate is 3-4 msgs/second.
Scenario:
With the above configuration a test was run continuously for 3 days. After 2.5 days, I encountered problem in terms of not receiving messages from topic in our consumer group.
Upon detailed investigation, I found out that consumer threads were blocked precisely 18 threads were in blocked state. Also, the thread graph is suggesting that consumer threads were waiting on “Java.nio.
PFA logs and screenshots .
a.) Logs
b.) Screenshots
a)Logs
"kafka-coordinator-heartbeat-thread | release-registry-group" #128 daemon prio=5 os_prio=0 tid=0x00007f1954001800 nid=0x8a waiting for monitor entry [0x00007f197f7f6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1005)
- waiting to lock <0x00000000c3382070> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
"org.springframework.kafka.KafkaListenerEndpointContainer#1-2-C-1" #127 prio=5 os_prio=0 tid=0x00007f1b6dbc1800 nid=0x89 runnable [0x00007f197f8f6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000c33822d8> (a sun.nio.ch.Util$3)
- locked <0x00000000c33822c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c33822e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.kafka.common.network.Selector.select(Selector.java:689)
at org.apache.kafka.common.network.Selector.poll(Selector.java:409)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:271)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:218)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:230)
- locked <0x00000000c3382070> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:314)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1175)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:732)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:689)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
b.) Screenshots
I would like to understand root cause and possible solutions to fix it and thanks in advance.
We use spring-boot-starter-amqp.1.7.4.Release 。when we try to kill this process，can’t stop it。
this is thread dump。please help me？
========================================================
"Thread-57" #104 prio=5 os_prio=0 tid=0x00007f4420038800 nid=0x39f6 in Object.wait() [0x00007f443d8d2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:49)
- locked <0x000000008ad818e0> (a com.rabbitmq.utility.BlockingValueOrException)
at com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:91)
- locked <0x000000008ad818e0> (a com.rabbitmq.utility.BlockingValueOrException)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:32)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:366)
at com.rabbitmq.client.impl.ChannelN.basicCancel(ChannelN.java:1280)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:980)
- locked <0x0000000089d2a198> (a java.lang.Object)
at com.sun.proxy.$Proxy107.basicCancel(Unknown Source)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.basicCancel(BlockingQueueConsumer.java:392)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doShutdown(SimpleMessageListenerContainer.java:902)
- locked <0x0000000089d2ac98> (a java.lang.Object)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.shutdown(AbstractMessageListenerContainer.java:558)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.destroy(AbstractMessageListenerContainer.java:522)
at org.springframework.amqp.rabbit.listener.RabbitListenerEndpointRegistry.destroy(RabbitListenerEndpointRegistry.java:210)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:256)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:576)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:552)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:953)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:521)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.destroySingletons(FactoryBeanRegistrySupport.java:227)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:960)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1035)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1011)
at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:933)
- locked <0x0000000088706660> (a java.lang.Object)
"SIGTERM handler" #310 daemon prio=9 os_prio=0 tid=0x00007f4430014800 nid=0x39f1 in Object.wait() [0x00007f44588d9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x0000000089e1f030> (a org.springframework.context.support.AbstractApplicationContext$1)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000000877fa068> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:748)
========================================================
add version info
springBootVersion = "1.5.7.RELEASE"
springVersion = "5.0.1.RELEASE"
compile "org.springframework.boot:spring-boot-starter:$springBootVersion"
compile "org.springframework:spring-tx:$springVersion"
compile "org.springframework:spring-core:$springVersion"
compile "org.springframework:spring-beans:$springVersion"
compile "org.springframework:spring-context:$springVersion"
compile "org.springframework:spring-expression:$springVersion"
compile "org.springframework:spring-aop:$springVersion"
compile "org.springframework:spring-context-support:$springVersion"
compile "org.springframework.boot:spring-boot-starter-amqp:$springBootVersion"
=========================================================
1.7.4 is nearly 2 years old; the current 1.7.x version is 1.7.12.
Similarly the current boot 1.5.x version is 1.5.19.
Furthermore the proper version of Boot to use with Spring Framework is 2.0.x. The current 2.0.x version is 2.0.8 (Spring 5.0.12).
It's generally better to let boot manage all the dependency versions (there is a gradle plugin).
That way, you don't have to worry about which version is compatible with which.
I suggest, first of all, you get your versions aligned properly and more up-to-date.
All that being said, the thread is blocked in the rabbitmq java client (on basicCancel).
at com.rabbitmq.client.impl.ChannelN.basicCancel(ChannelN.java:1280)
I don't know what conditions can cause that. Is there anything of interest in the application logs?
You would be better asking the rabbitmq engineers what could cause this; they monitor the rabbitmq-users Google group much more often than Stack Overflow.
I am getting OutOfMemoryError in my web application.
When I analysed thread dump i found one blocker thread and stack trace is :
Finalizer
Stack Trace is:
java.lang.Thread.State: BLOCKED
at net.sourceforge.jtds.jdbc.JtdsConnection.releaseTds(JtdsConnection.java:2024)
- waiting to lock <662301fc> (a net.sourceforge.jtds.jdbc.JtdsConnection) owned by "default task-204" t#896
at net.sourceforge.jtds.jdbc.JtdsStatement.close(JtdsStatement.java:972)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.close(JtdsPreparedStatement.java:707)
at net.sourceforge.jtds.jdbc.JtdsStatement.finalize(JtdsStatement.java:219)
at java.lang.System$2.invokeFinalize(System.java:1270)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
Locked ownable synchronizers:
- None
Details stack trace :
default task-204
Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:850)
at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:731)
- locked <5bc15901> (a java.util.concurrent.ConcurrentHashMap)
at net.sourceforge.jtds.jdbc.ResponseStream.getPacket(ResponseStream.java:477)
at net.sourceforge.jtds.jdbc.ResponseStream.read(ResponseStream.java:114)
at net.sourceforge.jtds.jdbc.ResponseStream.peek(ResponseStream.java:99)
at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:4127)
at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1086)
- locked <51eb98a0> (a net.sourceforge.jtds.jdbc.TdsCore)
at net.sourceforge.jtds.jdbc.JtdsCallableStatement.executeMSBatch(JtdsCallableStatement.java:165)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeBatch(JtdsStatement.java:1051)
- locked <662301fc> (a net.sourceforge.jtds.jdbc.JtdsConnection)
at org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1077)
at com.XXX.MyDao.myMethod(MyDao.java:102)
On line number 102 of MyDao.java I am executing callableStatement.executeBatch()
I am using jtds-1.3.1 for connection to database.
I analysed thread dump using http://fastthread.io
Image for the same:
After restarting my application it works fine.
You can read about finalizer thread from here.
I am facing new problem in glassfish server now :(
After server starts successfully after sometime it gets hang and stops processing requests
following are the exceptions in server log
[#|2011-05-08T17:44:34.027+0300|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=17;_ThreadName=Timer-18
;_RequestID=a47329c8-5efa-4e29-b90d-a5bda9c8f725;|
java.util.logging.ErrorManager: 5: Error in formatting Logrecord|#]
[#|2011-05-08T17:44:34.036+0300|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=17;_ThreadName=Timer-18
;_RequestID=a47329c8-5efa-4e29-b90d-a5bda9c8f725;|
java.security.ProviderException: implNextBytes() failed
at sun.security.pkcs11.P11SecureRandom.implNextBytes(P11SecureRandom.java:170)
at sun.security.pkcs11.P11SecureRandom.engineNextBytes(P11SecureRandom.java:117)
at java.security.SecureRandom.nextBytes(SecureRandom.java:413)
at java.util.UUID.randomUUID(UUID.java:161)
at com.sun.enterprise.admin.monitor.callflow.AgentImpl$ThreadLocalState.<init>(AgentImpl.java:162)
at com.sun.enterprise.admin.monitor.callflow.AgentImpl$1.initialValue(AgentImpl.java:256)
at com.sun.enterprise.admin.monitor.callflow.AgentImpl$1.initialValue(AgentImpl.java:255)
at java.lang.ThreadLocal$ThreadLocalMap.getAfterMiss(ThreadLocal.java:374)
at java.lang.ThreadLocal$ThreadLocalMap.get(ThreadLocal.java:347)
at java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:225)
at java.lang.ThreadLocal.get(ThreadLocal.java:127)
at com.sun.enterprise.admin.monitor.callflow.AgentImpl.getThreadLocalData(AgentImpl.java:1070)
at com.sun.enterprise.server.logging.UniformLogFormatter.uniformLogFormat(UniformLogFormatter.java:320)
at com.sun.enterprise.server.logging.UniformLogFormatter.format(UniformLogFormatter.java:151)
at java.util.logging.StreamHandler.publish(StreamHandler.java:179)
at com.sun.enterprise.server.logging.FileandSyslogHandler.publish(FileandSyslogHandler.java:512)
at java.util.logging.Logger.log(Logger.java:440)
at java.util.logging.Logger.doLog(Logger.java:462)
at java.util.logging.Logger.log(Logger.java:526)
at com.sun.enterprise.deployment.autodeploy.AutoDeployControllerImpl$AutoDeployTask.run(AutoDeployControllerImpl.java
:391)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR
at sun.security.pkcs11.wrapper.PKCS11.C_GenerateRandom(Native Method)
at sun.security.pkcs11.P11SecureRandom.implNextBytes(P11SecureRandom.java:167)
... 21 more
|#]
[#|2011-05-08T17:44:36.034+0300|SEVERE|sun-appserver2.1|javax.enterprise.system.tools.deployment|_ThreadID=17;_ThreadName=Tim
er-18;_RequestID=a47329c8-5efa-4e29-b90d-a5bda9c8f725;|"DPL8011: autodeployment failure while deploying the application : nul
l"|#]
[#|2011-05-08T17:44:38.034+0300|SEVERE|sun-appserver2.1|javax.enterprise.system.tools.deployment|_ThreadID=17;_ThreadName=Tim
er-18;_RequestID=a47329c8-5efa-4e29-b90d-a5bda9c8f725;|"DPL8011: autodeployment failure while deploying the application : nul
l"|#]
********
when it is in hang state i took thread dump and found 1 dead lock as below !!!
Found one Java-level deadlock:
=============================
"SelectorThread-4242":
waiting to lock monitor 0x0040ef60 (object 0x56cd6dd0, a org.apache.jasper.util.SystemLogHandler),
which is held by "Timer-2"
"Timer-2":
waiting to lock monitor 0x0040f5d8 (object 0x568bc440, a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream),
which is held by "SelectorThread-4242"
Java stack information for the threads listed above:
===================================================
"SelectorThread-4242":
at java.lang.Throwable.printStackTrace(Throwable.java:461)
- waiting to lock <0x56cd6dd0> (a org.apache.jasper.util.SystemLogHandler)
at java.lang.Throwable.printStackTrace(Throwable.java:452)
at java.util.logging.ErrorManager.error(ErrorManager.java:78)
- locked <0x5a4fe388> (a java.util.logging.ErrorManager)
at com.sun.enterprise.server.logging.UniformLogFormatter.uniformLogFormat(UniformLogFormatter.java:364)
at com.sun.enterprise.server.logging.UniformLogFormatter.format(UniformLogFormatter.java:151)
at java.util.logging.StreamHandler.publish(StreamHandler.java:179)
- locked <0x568825e0> (a com.sun.enterprise.server.logging.FileandSyslogHandler)
at com.sun.enterprise.server.logging.FileandSyslogHandler.publish(FileandSyslogHandler.java:512)
- locked <0x568825e0> (a com.sun.enterprise.server.logging.FileandSyslogHandler)
at java.util.logging.Logger.log(Logger.java:440)
at java.util.logging.Logger.doLog(Logger.java:462)
at java.util.logging.Logger.log(Logger.java:485)
at com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingByteArrayOutputStream.flush(SystemOutandErrHandler.java:368)
- locked <0x568b84f0> (a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingByteArrayOutputStream)
at java.io.PrintStream.write(PrintStream.java:414)
- locked <0x568bc440> (a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream)
at com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream.write(SystemOutandErrHandler.java:293)
at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404)
at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:115)
- locked <0x568bc470> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:169)
at java.io.PrintStream.write(PrintStream.java:459)
- locked <0x568bc440> (a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream)
at java.io.PrintStream.print(PrintStream.java:602)
at com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream.print(SystemOutandErrHandler.java:205)
at java.io.PrintStream.println(PrintStream.java:739)
- locked <0x568bc440> (a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream)
at com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream.println(SystemOutandErrHandler.java:187)
at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:254)
at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:254)
at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:254)
at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:254)
at java.util.logging.ErrorManager.error(ErrorManager.java:76)
- locked <0x5a4fe3a8> (a java.util.logging.ErrorManager)
at com.sun.enterprise.server.logging.UniformLogFormatter.uniformLogFormat(UniformLogFormatter.java:364)
at com.sun.enterprise.server.logging.UniformLogFormatter.format(UniformLogFormatter.java:151)
at com.sun.enterprise.server.logging.AMXLoggingHook.publish(AMXLoggingHook.java:198)
at com.sun.enterprise.server.logging.FileandSyslogHandler.publish(FileandSyslogHandler.java:531)
- locked <0x568825e0> (a com.sun.enterprise.server.logging.FileandSyslogHandler)
at java.util.logging.Logger.log(Logger.java:440)
at java.util.logging.Logger.doLog(Logger.java:462)
at java.util.logging.Logger.log(Logger.java:551)
at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1445)
at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1316)
- locked <0x56cdf878> (a [Ljava.lang.Object;)
at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1279)
at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1255)
"Timer-2":
at java.io.PrintStream.println(PrintStream.java:738)
- waiting to lock <0x568bc440> (a com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream)
at com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingPrintStream.println(SystemOutandErrHandler.java:178)
at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:258)
at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:258)
at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:258)
at org.apache.jasper.util.SystemLogHandler.println(SystemLogHandler.java:258)
at java.lang.Throwable.printStackTrace(Throwable.java:462)
- locked <0x56cd6dd0> (a org.apache.jasper.util.SystemLogHandler)
at java.lang.Throwable.printStackTrace(Throwable.java:452)
at com.sun.jbi.management.system.AdminService.heartBeat(AdminService.java:975)
at com.sun.jbi.management.system.AdminService.handleNotification(AdminService.java:198)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1652)
at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:221)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:184)
at javax.management.timer.Timer.sendNotification(Timer.java:1295)
- locked <0x57d2e590> (a javax.management.timer.TimerNotification)
at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1264)
at javax.management.timer.TimerAlarmClock.run(Timer.java:1347)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
**Found 1 deadlock.**
Please suggest
Thanks
Ali
got same issue
i resolve it cause my .properties file contained a simple quote ' , thus when Jersey logger tries to log it, it cannot parse the Json send back to the server.
So look which kind of String Jersey is trying to log and be carefull on special characters