Develop Reference

Thread is in WAITING state: java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

My application is under heavy load and I am getting below logs for
sudo -u tomcat jstack <java_process_id>
The below thread is consuming the messages from Kafka, and it got stuck. Since this thread is in WAITING state, no more kafka messages are being consumed.
"StreamThread-3" #91 daemon prio=5 os_prio=0 tid=0x00007f9b5c606000 nid=0x1e4d waiting on condition [0x00007f9b506c5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000073aad9718> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353)
at ch.qos.logback.core.AsyncAppenderBase.put(AsyncAppenderBase.java:160)
at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:148)
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.error(Logger.java:538)
at com.abc.system.solr.repo.AbstractSolrRepository.doSave(AbstractSolrRepository.java:316)
at com.abc.system.solr.repo.AbstractSolrRepository.save(AbstractSolrRepository.java:295)
I also found this post
WAITING at sun.misc.Unsafe.park(Native Method)
but it didn't help me in my case.
What else I could investigate to get more details in such case?

I also ran into same problem. But, luckily I got my issue resolved by playing around with the size of pool and number of producer and consumer.
Try to check if there is any way to configure following.
Size of your thread pool
Number of consumers/producers (if we can configure in kafka)
Make sure that thread pool should have enough threads to serve consumer and producer both.

Out of memory error and 100% CPU usage in Java web application with jTDS

I am getting OutOfMemoryError in my web application.
When I analysed thread dump i found one blocker thread and stack trace is :
Finalizer
Stack Trace is:
java.lang.Thread.State: BLOCKED
at net.sourceforge.jtds.jdbc.JtdsConnection.releaseTds(JtdsConnection.java:2024)
- waiting to lock <662301fc> (a net.sourceforge.jtds.jdbc.JtdsConnection) owned by "default task-204" t#896
at net.sourceforge.jtds.jdbc.JtdsStatement.close(JtdsStatement.java:972)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.close(JtdsPreparedStatement.java:707)
at net.sourceforge.jtds.jdbc.JtdsStatement.finalize(JtdsStatement.java:219)
at java.lang.System$2.invokeFinalize(System.java:1270)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
Locked ownable synchronizers:
- None
Details stack trace :
default task-204
Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:850)
at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:731)
- locked <5bc15901> (a java.util.concurrent.ConcurrentHashMap)
at net.sourceforge.jtds.jdbc.ResponseStream.getPacket(ResponseStream.java:477)
at net.sourceforge.jtds.jdbc.ResponseStream.read(ResponseStream.java:114)
at net.sourceforge.jtds.jdbc.ResponseStream.peek(ResponseStream.java:99)
at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:4127)
at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1086)
- locked <51eb98a0> (a net.sourceforge.jtds.jdbc.TdsCore)
at net.sourceforge.jtds.jdbc.JtdsCallableStatement.executeMSBatch(JtdsCallableStatement.java:165)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeBatch(JtdsStatement.java:1051)
- locked <662301fc> (a net.sourceforge.jtds.jdbc.JtdsConnection)
at org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1077)
at com.XXX.MyDao.myMethod(MyDao.java:102)
On line number 102 of MyDao.java I am executing callableStatement.executeBatch()
I am using jtds-1.3.1 for connection to database.
I analysed thread dump using http://fastthread.io
Image for the same:
After restarting my application it works fine.
You can read about finalizer thread from here.

Locked object found on oracle.jdbc.driver.T4CConnection

I am using JMC to perform application profiling and I did not see any locked/thread contention as shown in the screenshot below.
I ran the SQL below (every few secs) also did not return any result.
select
(select username from v$session where sid=a.sid) blocker,
a.sid,
' is blocking ',
(select username from v$session where sid=b.sid) blockee,
b.sid
from
v$lock a,
v$lock b
where
a.block = 1
and
b.request > 0
and
a.id1 = b.id1
and
a.id2 = b.id2;
What could be the caused of a lock database connection? Could it be database record/table locks?
Below is the thread dump which I have extracted during the execution of my program when it seems to be running forever.
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at oracle.net.ns.Packet.receive(Packet.java:283)
at oracle.net.ns.DataPacket.receive(DataPacket.java:103)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:230)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1122)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:863)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3620)
- locked <0x00000007af3423c0> (a oracle.jdbc.driver.T4CConnection)

You're confusing database locks with Java locks here. JMC only shows you the locks inside your Java program (synchronized blocks, waits etc), it knows nothing about what's going on inside your DB. Your SQL-query only shows the locks on the DB level (table locks, row locks etc) and knows nothing about the locks inside your Java program. Those are absolutely different areas and absolutely different locks.
What you have here is a dump of a thread that holds a lock on the object of type T4CConnection with the address 0x7af3423c0. It only means that this thread is in the process of executing a code inside some synchronized(connection) block. That's all. The thread is not blocked by other threads (otherwise its state wouldn't be RUNNABLE, it would be WAITING or BLOCKED). It's running and reading something from a network socket (probably, the response from the DB).
Such behaviour is absolutely normal. The DB driver does synchronization on the connection instance while it's in the process of executing an SQL-query to not allow other threads to use it in parallel.
There's nothing you should worry about on this screenshot and in this thread dump.

c3p0 hangs in awaitAvailable with hibernate

I have console application that hangs during execution. Here is my configuration:
cfg.setProperty("hibernate.connection.driver_class", "com.mysql.jdbc.Driver");
cfg.setProperty("hibernate.connection.url", "jdbc:mysql://localhost:3306/db?user=db&password=db");
cfg.setProperty("hibernate.connection.username", "db");
cfg.setProperty("hibernate.connection.password", "db");
cfg.setProperty("hibernate.connection.pool_size", "5");
cfg.setProperty("hibernate.connection.autocommit", "false");
cfg.setProperty("hibernate.c3p0.min_size", "5");
cfg.setProperty("hibernate.c3p0.max_size", "20");
cfg.setProperty("hibernate.c3p0.timeout", "300");
cfg.setProperty("hibernate.c3p0.max_statements", "50");
cfg.setProperty("hibernate.c3p0.idle_test_period", "3000");
Here is my stacktrace:
"main" prio=10 tid=0x000000000168f800 nid=0x1c37 in Object.wait() [0x00007fa60d0ad000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007400f4c68> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
- locked <0x00000007400f4c68> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:84)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:281)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1392)
at org.kriyak.parser.IndexArchiveRapid.indexFile(IndexArchiveRapid.java:70)
at org.kriyak.parser.IndexArchiveRapid.main(IndexArchiveRapid.java:53)
I open only one conencton and it doesn't seem that I leak them. And also I use one thread. I haven't adjusted any mysql settings except memory usage. Mysql works fine from console. Why can this happen? Is this c3p0 error?

does this happen immediately, or after a while? that is, do checkouts initially succeed, but then hang like this? if so, it looks like a Connection leak. please try setting c3p0 params unreturnedConnectionTimeout and debugUnreturnedConnectionStackTraces to see if there is a leak. See
http://www.mchange.com/projects/c3p0/#configuring_to_debug_and_workaround_broken_clients , http://www.mchange.com/projects/c3p0/#unreturnedConnectionTimeout , http://www.mchange.com/projects/c3p0/#debugUnreturnedConnectionStackTraces .
if this happens immediately, if no Connections are successfully checked out, the question is whether the pool ever succeeds at acquiring Connections. by default, if it never does succeed, after about 30 seconds your thread should break with a failure. (it doesn't look like you've done this, but if for example you'd set acquireRetryAttempts to zero, c3p0 might hang indefinitely waiting for Connections.)
to debug c3p0 issues, it's helpful to capture the version and config information that c3p0 dumps to logs at INFO level on pool initialization.
good luck!

Also, you haven't seem to initialize the checkoutTime parameter for c3p0, which specifies the amount of time a client should wait for acquiring a connection from the connection pool.
see http://www.mchange.com/projects/c3p0/#checkoutTimeout

You may need to increase c3p0.numHelperThreads. The helper threads are responsible for creating new database connections and adding them to the pool. If there aren't enough threads to keep up with demand then application threads will be stuck waiting in awaitAvailable(). To confirm this is the case look at all the HelperThreads and see if they are all in use and in the process of connecting to the database.

HttpURLConnection getting locked

I have a thread running under tomcat which creates a HttpUrlConnection and reads it through BufferedInputStream.
After fetching data for some urls, it stalls. I got the jstack of the process which says HttpUrlConnection is locked and BufferedInputStream is also locked.
"http-8080-1" daemon prio=10 tid=0x08683400 nid=0x79c9 runnable [0x8f618000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x956ef8c0> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1072)
- locked <0x956ef910> (a sun.net.www.protocol.http.HttpURLConnection)
Could somebody help here.
Thanks

You probably have a problem on the other end. read() on an InputStream is a blocking operation - from the javadoc (http://java.sun.com/javase/6/docs/api/): "This method blocks until input data is available, the end of the stream is detected, or an exception is thrown."
Is the server on the other end responding? Do you know if it's sent anything?
edit: To make it clearer, the thread is in RUNNABLE state, so you're not deadlocked - it sounds like that's what you're thinking that it is, but there's no evidence here of any deadlock.