Develop Reference

Tomcat 100% CPU Usage

I tried every thing to find the cause but not sure whats happing, tried to jProfile, but its not getting any results as cpu is already at 100%.
I am running a Spring application (WAR file) on amazon linux on
Tomcat8. no database operations but yes it access a rest api for different operations running in separate environment.
There are 3 servers have same issue running behind a load balancer in a beanstalk. with average of 518K requests per hour and average latency of 72.2 milliseconds
I feel like there is not much issue with the code itself
I just tried a command kill -3 pid (java process), and got some results in catalina.out file. I can see following error again and again
"http-nio-8080-exec-13" #42 daemon prio=5 os_prio=0 tid=0x00007f0898005800 nid=0xfb2 waiting on condition [0x00007f0882dec000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f1694f58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
"http-nio-8080-exec-18" #41 daemon prio=5 os_prio=0 tid=0x00007f088c024000 nid=0xfb1 waiting on condition [0x00007f0882eed000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f1694f58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
and at the end is following information
"VM Thread" os_prio=0 tid=0x00007f08d8081000 nid=0xf8b runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f08d801e800 nid=0xf89 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f08d8020800 nid=0xf8a runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f08d80f0000 nid=0xf92 waiting on condition
JNI global references: 478
Heap
PSYoungGen total 85504K, used 28368K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
eden space 83968K, 33% used [0x00000000fab00000,0x00000000fc61cd08,0x00000000ffd00000)
from space 1536K, 39% used [0x00000000ffd00000,0x00000000ffd97340,0x00000000ffe80000)
to space 1536K, 0% used [0x00000000ffe80000,0x00000000ffe80000,0x0000000100000000)
ParOldGen total 175104K, used 107142K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
object space 175104K, 61% used [0x00000000f0000000,0x00000000f68a1a48,0x00000000fab00000)
Metaspace used 56563K, capacity 60632K, committed 60800K, reserved 1103872K
class space used 5218K, capacity 5736K, committed 5760K, reserved 1048576K
Could someone please explain what is happening here?
Ok here are some of the logs I pulled from the log which are in running state
(couple of times in log)
"ajp-nio-8009-Acceptor-0" #23 daemon prio=5 os_prio=0 tid=0x00007f08d8535800 nid=0xfa0 runnable [0x00007f0883ffe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000000f0881618> (a java.lang.Object)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:682)
at java.lang.Thread.run(Thread.java:745)
"ajp-nio-8009-ClientPoller-1" #22 daemon prio=5 os_prio=0 tid=0x00007f08d8534000 nid=0xf9f runnable [0x00007f08a83b2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000f16a8100> (a sun.nio.ch.Util$2)
- locked <0x00000000f16a80f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f16a7fc8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1034)
at java.lang.Thread.run(Thread.java:745)
"ajp-nio-8009-ClientPoller-0" #21 daemon prio=5 os_prio=0 tid=0x00007f08d82f8800 nid=0xf9e runnable [0x00007f08a84b3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000f16a8900> (a sun.nio.ch.Util$2)
- locked <0x00000000f16a88f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f16a87c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1034)
at java.lang.Thread.run(Thread.java:745)
"http-nio-8080-Acceptor-0" #20 daemon prio=5 os_prio=0 tid=0x00007f08d82f7000 nid=0xf9d runnable [0x00007f08a85b4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000000f0882f68> (a java.lang.Object)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:682)
at java.lang.Thread.run(Thread.java:745)
"http-nio-8080-ClientPoller-1" #19 daemon prio=5 os_prio=0 tid=0x00007f08d82f5800 nid=0xf9c runnable [0x00007f08a86b5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000f1677900> (a sun.nio.ch.Util$2)
- locked <0x00000000f16778f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f16777a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1034)
at java.lang.Thread.run(Thread.java:745)
"http-nio-8080-ClientPoller-0" #18 daemon prio=5 os_prio=0 tid=0x00007f08d82f4000 nid=0xf9b runnable [0x00007f08a87b6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000f1666290> (a sun.nio.ch.Util$2)
- locked <0x00000000f1666280> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f1666138> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1034)
at java.lang.Thread.run(Thread.java:745)
There is another one Not sure if its causing a problem (but only once in log)
"http-nio-8080-exec-131" #160 daemon prio=5 os_prio=0 tid=0x00007f088c100800 nid=0x153e runnable [0x00007f08727e5000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000f688f4d0> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
- locked <0x00000000f6892258> (a sun.net.www.protocol.http.HttpURLConnection)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
- locked <0x00000000f6892258> (a sun.net.www.protocol.http.HttpURLConnection)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderFields(HttpURLConnection.java:2966)
at com.code.http.WebUtility.getUrlContents(WebUtility.java:163)
and the code on this lines is con.getHeaderFields(), where as con is HttpURLConnection
try {
con = (HttpURLConnection) url.opencon();
con.setDoInput(true);
con.setDoOutput(true);
con.connect();
writeJson (con);
this.header = con.getHeaderFields();//this is the line in running state
this.code = con.getResponseCode();
return readSuccessStream(con);
} catch (IOException e) {
if (con != null) {
return readFailureStream(con);
}
return "a nasty error occured";
} finally {
if (con != null) {
con.disconnect();
}
}
and few other treads in running state
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007f08d80db000 nid=0xf91 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f08d80c8800 nid=0xf90 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f08d80bb000 nid=0xf8f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f08d80b9000 nid=0xf8e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
and the last one
"main" #1 prio=5 os_prio=0 tid=0x00007f08d8009800 nid=0xf88 runnable [0x00007f08de871000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:446)
at org.apache.catalina.startup.Catalina.await(Catalina.java:717)
at org.apache.catalina.startup.Catalina.start(Catalina.java:663)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:351)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:485)

Thread states
The states of a thread in a Java virtual machine are:
NEW
A thread that has not yet started is in this state.
RUNNABLE
A thread executing in the Java virtual machine is in this state.
BLOCKED
A thread that is blocked waiting for a monitor lock is in this state.
WAITING
A thread that is waiting indefinitely for another thread to perform a particular action is in this state.
TIMED_WAITING
A thread that is waiting for another thread to perform an action for up to a specified waiting time is in this state.
TERMINATED
A thread that has exited is in this state.
In Java, threads are represented as Thread objects (java.lang.Thread). When you create a new thread, literally with new Thread(), you create a thread, but it isn't doing anything yet. Once you call Thread.start(), it moves from the NEW state into the RUNNABLE state. Runnable means it has pending work for the CPU to perform. While running, a thread can move between RUNNABLE, BLOCKED, WAITING, and TIMED_WAITING. When a thread exits, it is left in the TERMINATED state.
When trying to diagnose high CPU usage by a Java process, we can rule out threads which are NEW or TERMINATED since they are dead. A thread which is BLOCKED, WAITING, or TIMED_WAITING are not doing anything. They are, literally, waiting for some event in the future.
What does that mean? Only threads in the RUNNABLE state can be putting load on the CPU.
What are all those threads?
Tomcat uses a pool of threads to be able to handle multiple requests at the same time.
This first thread is the acceptor thread. Its purpose is to listen on the network socket for incoming requests. But in order to handle multiple requests simultaneously, it needs to delegate the work. Otherwise, each request would lock up the server until it finishes, meaning only one user can connect to the web server at a time. So this thread simple accepts the connections and then hands them off to another thread so it can accept more connections.
"http-nio-8080-Acceptor-0" #20 daemon prio=5 os_prio=0 tid=0x00007f08d82f7000 nid=0xf9d runnable [0x00007f08a85b4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000000f0882f68> (a java.lang.Object)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:682)
at java.lang.Thread.run(Thread.java:745)
These threads are the worker threads to which Tomcat assigns the requests:
"http-nio-8080-exec-13" #42 daemon prio=5 os_prio=0 tid=0x00007f0898005800 nid=0xfb2 waiting on condition [0x00007f0882dec000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f1694f58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
That means the thread is sleeping. It is waiting for the acceptor thread to give it work. It is contributing nothing to your 100% CPU usage.
Conclusion
The runnable threads I saw in your thread dumps show that reading from a network socket is where the CPU load is spent. I cannot say definitively why. There are two possible reasons: a problem with the network or remote system being extremely latent or reading is a problem due to memory usage.
Most of the time when I have seen this, the CPU work was coming from the garbage collector, which runs in another thread. When there is pressure to allocate memory and little is available, the garbage collector has to do a lot of expensive work to find available memory while the other threads wait. This thread might not be visible in the thread dumps you showed since it's not a thread created in Java code, but is an internal part of the Java virtual machine. I can't say for sure if that is the issue in your case. It could be a network problem or problem with the remote system you are reading from. I suggest looking carefully at every thread to try to find a pattern.

Tomcat not processing request

I have a problem with my production tomcat in which my web application is deployed. I runs fine for some long duration but after that whenever any user try to access the application through web browser then connection reset message will be shown on the browser literally application down.
I tried to increase the tomcat memory but the problem still continue. I have taken the thread dump also in which all the thread are showing in BLOCKED state.
Can anyone help me out !!!
Thread dump sample
"http-80-173" daemon prio=6 tid=0x55392800 nid=0xf84 waiting for monitor entry [0x64c8e000..0x64c8f9e8]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
at java.lang.StringBuffer.append(StringBuffer.java:320)
- locked <0x4472eeb8> (a java.lang.StringBuffer)
at java.text.MessageFormat.applyPattern(MessageFormat.java:436)
at java.text.MessageFormat.<init>(MessageFormat.java:350)
at java.text.MessageFormat.format(MessageFormat.java:811)
at org.apache.naming.StringManager.getString(StringManager.java:121)
at org.apache.naming.StringManager.getString(StringManager.java:144)
at org.apache.naming.resources.FileDirContext.getAttributes(FileDirContext.java:432)
at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:747)
at org.apache.naming.resources.ProxyDirContext.cacheLoad(ProxyDirContext.java:1531)
at org.apache.naming.resources.ProxyDirContext.cacheLookup(ProxyDirContext.java:1454)
at org.apache.naming.resources.ProxyDirContext.lookup(ProxyDirContext.java:288)
at org.apache.naming.resources.DirContextURLConnection.getInputStream(DirContextURLConnection.java:368)
at org.apache.jasper.compiler.Compiler.isOutDated(Compiler.java:435)
at org.apache.jasper.compiler.Compiler.isOutDated(Compiler.java:392)
All thread will be in same state.
"http-80-160" daemon prio=6 tid=0x5538dc00 nid=0x1e80 waiting for monitor entry [0x6453e000..0x6453fce8]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
at java.lang.StringBuffer.<init>(StringBuffer.java:79)
at com.microsoft.sqlserver.jdbc.SQLCollation.readCollation(Unknown Source)
at com.microsoft.sqlserver.jdbc.TypeInfo.init(Unknown Source)
at com.microsoft.sqlserver.jdbc.StreamColumns.processBytes(Unknown Source)
at com.microsoft.sqlserver.jdbc.IOBuffer.processPackets(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement$StatementExecutionRequest.executeStatement(Unknown Source)
at com.microsoft.sqlserver.jdbc.CancelableRequest.execute(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeRequest(Unknown Source)
**- locked <0x444b72c0> (a com.microsoft.sqlserver.jdbc.TDSWriter)**
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeQuery(Unknown Source)
at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
Also this is a common line i am getting in most of the cases:
locked <0x444b72c0> (a com.microsoft.sqlserver.jdbc.TDSWriter)
I am using JDBC driver 2 and my Operating system is Windows 2007
Also i am using connection pooling and during getting the connection it is showing locked
"http-80-172" daemon prio=6 tid=0x55392400 nid=0x2548 waiting for monitor entry [0x64bfe000..0x64bffa68]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.pool.impl.GenericObjectPool.getNumIdle(GenericObjectPool.java:911)
- waiting to lock <0x3ef89f88> (a org.apache.commons.dbcp.AbandonedObjectPool)
at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:78)
at org.apache.commons.dbcp.PoolingDriver.connect(PoolingDriver.java:176)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:207)
at com.cong.vts.dbutil.JDBCConnection.getConnection(JDBCConnection.java:42)
at com.cong.vts.dbutil.SQLFunctions.createCSSFile(SQLFunctions.java:10069)

Simple while loop waiting and never starts in Java 1.6 on Xubuntu 12.04

I have a really simple Java code:
public class Main {
public static void main(String[] args) {
int c = 0;
while (c < 10);
{
System.out.println(c);
c++;
}
System.out.println("exit");
}
}
The problem is, the while loop doesn't starts, the program just keeps waiting on the 7th line (at the beginning of the while loop).
I use Oracle Java JDK 1.6u33 on Xubuntu 12.04. On Windows this code works perfectly. Can somebody tell me, what can I do to run this code correctly on Ubuntu?
What I did (and the Thread dump):
gopher#orion:~/Projects/test/src$ javac Main.java
gopher#orion:~/Projects/test/src$ jar cvfm test.jar manifest.txt Main.class
added manifest
adding: Main.class(in = 489) (out= 338)(deflated 30%)
gopher#orion:~/Projects/test/src$ java -jar test.jar
^\2012-08-04 10:26:23
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03 mixed mode):
"Low Memory Detector" daemon prio=10 tid=0x00007fbf080b4800 nid=0xa26 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007fbf080b2800 nid=0xa25 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007fbf080af800 nid=0xa24 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007fbf080ad800 nid=0xa23 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007fbf08091000 nid=0xa22 in Object.wait() [0x00007fbf0d1f3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f5b61300> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00000000f5b61300> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00007fbf0808f000 nid=0xa21 in Object.wait() [0x00007fbf0d2f4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f5b611d8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00000000f5b611d8> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007fbf08007800 nid=0xa1d runnable [0x00007fbf0eb4b000]
java.lang.Thread.State: RUNNABLE
at Main.main(Main.java:8)
"VM Thread" prio=10 tid=0x00007fbf08088800 nid=0xa20 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fbf0801a800 nid=0xa1e runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fbf0801c000 nid=0xa1f runnable
"VM Periodic Task Thread" prio=10 tid=0x00007fbf080bf000 nid=0xa27 waiting on condition
JNI global references: 887
Heap
PSYoungGen total 9216K, used 317K [0x00000000f5b60000, 0x00000000f65a0000, 0x0000000100000000)
eden space 7936K, 4% used [0x00000000f5b60000,0x00000000f5baf628,0x00000000f6320000)
from space 1280K, 0% used [0x00000000f6460000,0x00000000f6460000,0x00000000f65a0000)
to space 1280K, 0% used [0x00000000f6320000,0x00000000f6320000,0x00000000f6460000)
PSOldGen total 21056K, used 0K [0x00000000e1200000, 0x00000000e2690000, 0x00000000f5b60000)
object space 21056K, 0% used [0x00000000e1200000,0x00000000e1200000,0x00000000e2690000)
PSPermGen total 21248K, used 2774K [0x00000000dc000000, 0x00000000dd4c0000, 0x00000000e1200000)
object space 21248K, 13% used [0x00000000dc000000,0x00000000dc2b5bb8,0x00000000dd4c0000)

You have a ; at the end of the while, remove it and it will work
while (c < 10);
the above will exectue a single line while() and because c is never increased it is an endless loop. change it into
while (c < 10) // no ; at the end

Remove ; from while loop. make it as follows
while (c < 10)

Java thread blocking

i have a problem with my java environement. I'm running Solr 1.3 (search engine) since more then a year now and suddenly i got alot of trouble with it. All my thread pool (250) got randomly blocked once or twice a day. I did not make any change on my solr application or my tomcat server.
I'm running tomcat 5.5.25 and Solr 1.3. I got a thread dump when the system is totally overloaded :
igot like 240 thread like this one :
"http-8080-Processor1" daemon prio=10 tid=0x0000000000b2e000 nid=0x193 waiting for monitor entry [0x000000004066c000..0x000000004066cb20]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.logging.StreamHandler.publish(StreamHandler.java:174)
- waiting to lock <0x00007fe37e72b340> (a java.util.logging.ConsoleHandler)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
at java.util.logging.Logger.log(Logger.java:472)
at java.util.logging.Logger.doLog(Logger.java:494)
at java.util.logging.Logger.log(Logger.java:517)
at java.util.logging.Logger.info(Logger.java:1036)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:619)
We can see that this thread is blocked and waiting on : <0x00007fe37e72b340>
The thread who actually own the <0x00007fe37e72b340> is this one :
"http-8080-Processor156" daemon prio=10 tid=0x0000000000df2000 nid=0x1e52 runnable [0x0000000044521000..0x0000000044521c20]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
- locked <0x00007fe37e3abcd8> (a java.io.BufferedOutputStream)
at java.io.PrintStream.write(PrintStream.java:430)
- locked <0x00007fe37e3abca0> (a java.io.PrintStream)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
- locked <0x00007fe37e72cd90> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
at java.util.logging.StreamHandler.flush(StreamHandler.java:225)
- locked <0x00007fe37e72b340> (a java.util.logging.ConsoleHandler)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:89)
at java.util.logging.Logger.log(Logger.java:472)
at java.util.logging.Logger.doLog(Logger.java:494)
at java.util.logging.Logger.log(Logger.java:517)
at java.util.logging.Logger.info(Logger.java:1036)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:619)
and this is the last part of my thread dump :
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007f6510349800 nid=0xbff waiting on condition [0x0000000041d8d000..0x0000000041d8dd20]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1548)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-1" prio=10 tid=0x0000000000c26400 nid=0xbfe waiting on condition [0x000000004200e000..0x000000004200eca0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f651b275510> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
at java.lang.Thread.run(Thread.java:619)
"Low Memory Detector" daemon prio=10 tid=0x00007f6510004400 nid=0xbfa runnable [0x0000000000000000..0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x00007f6510001000 nid=0xbf9 waiting on condition [0x0000000000000000..0x0000000040d5e340]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x00000000006bc400 nid=0xbf8 waiting on condition [0x0000000000000000..0x0000000040c5d2d0]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00000000006bb000 nid=0xbf7 runnable [0x0000000000000000..0x0000000040b5da30]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x0000000000690c00 nid=0xbf6 in Object.wait() [0x000000004065e000..0x000000004065ed20]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f651aa10258> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x00007f651aa10258> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x000000000068f400 nid=0xbf5 in Object.wait() [0x000000004055d000..0x000000004055dca0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f651aa10338> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00007f651aa10338> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0000000000622400 nid=0xbeb runnable [0x0000000000000000..0x00007fff69fcbba0]
java.lang.Thread.State: RUNNABLE
"VM Thread" prio=10 tid=0x000000000068a000 nid=0xbf4 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000000062cc00 nid=0xbec runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000000062e000 nid=0xbed runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x000000000062f400 nid=0xbee runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000000630400 nid=0xbef runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000000631800 nid=0xbf0 runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000000632c00 nid=0xbf1 runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000000634000 nid=0xbf2 runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000000635400 nid=0xbf3 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f6510006800 nid=0xbfb waiting on condition
JNI global references: 1201
I know this is not a thread deadlock prob since one thread is actually runnning with all the ressource every other thread want.
Anyone got an idea of what can cause this prob ?

All your Threads are logging things. They all need to write on the disk from time to time.
Everytime one of your 240 threads hits a logging line there will be disk access issues.
It baffles me that the Thread having the lock is in the RUNNABLE state.
I think it might be waiting for some external ressource to be released (like disk access for example)
Are you running low on disk space? Have you recently changed something in your storage system?

If you are running under Windows and the java application starts a console, be careful not to click in the DOS box. Window's crappy mark and copy "feature" blocks output to the ConsoleHandler. So any logger trying to write to the screen will block. Writing to the Console is done in a native call and so the java thread will appear to be in a RUNNING state when in fact it is blocked, it's just there's no way to feed that blocked status back to the application (because you are in native space).
If the application is blocked (you have clicked in the DOS box) press escape to continue.

I never used java.util.logging, so I don't know whether my suggestion is useful, but netherless:
try to use different instance of java.util.logging.Logger, so not all 240 threads will be blocked on the same monitor
(it will help if different instances of Logger use different instances of java.util.logging.ConsoleHandler) .

It seems that the thread that own "0x00007fe37e72b340" is blocked at the IO level. Maybe a disk (raid?) issue?
can you do a thread dump 5 minutes later the see if the same thread is still blocked?

Flushing after each log record is going to expensive if you have very verbose logs.
A quality fix would be to clean up the logging, probably based around auditing.
As a quick fix, override StreamHandler.flush or OutputStream.flush to not do so immediately. Only flush once every so often. Note, however that you could potentially lose logging data immediately before a crash if you do this.

According to your log, the issue concerns the use of java.util.logging.ConsoleHandler.
First try to disable the console handler by removing it from the 'handlers' and '.handlers' list in '${TOMCAT_HOME}/conf/logging.properties'. See whether the problem still occurs.
If that helps, than it is definitely an issue with the output of the ConsoleHandler. Try checking whether there is are issues concerning the 'catalina.out' file. This is the file to where tomcat redirects its console output.

Java thread dump where main thread has no call stack? (jsvc)

We have a java process running as a daemon (under jsvc). Every several days it just stops doing any work; output to the logfile stops (it is pretty verbose, on 5-minute intervals) and it consumes no CPU or IO.
There are no exceptions logged in the logfile nor in syserr or sysout. The last log statement is just prior to a db commit being done, but there is no open connection on the db server (MySQL) and reviewing the code, there should always be additional log output after that, even if it had encountered an exception that was going to bubble up.
The most curious thing I find is that in the thread dump (included below), there's no thread in our code at all, and the main thread seems to have no context whatsoever:
"main" prio=10 tid=0x0000000000614000 nid=0x445d runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
As noted earlier, this is a daemon process running using jsvc, but I don't know if that has anything to do with it (I can restructure the code to also allow running it directly, to test).
Any suggestions on what might be happening here?
Thanks... dwh
Full thread dump:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode):
"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00002aaaf81b8800 nid=0x447b in Object.wait() [0x00002aaaf6a22000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab5556d50> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <0x00002aaab5556d50> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Low Memory Detector" daemon prio=10 tid=0x00000000006a4000 nid=0x4479 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x00000000006a1000 nid=0x4477 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x000000000069d000 nid=0x4476 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x000000000069b000 nid=0x4465 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x0000000000678800 nid=0x4464 in Object.wait() [0x00002aaaf61d6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab54a1cb8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00002aaab54a1cb8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x0000000000676800 nid=0x4463 in Object.wait() [0x00002aaaf60d5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab54a1cf0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaab54a1cf0> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0000000000614000 nid=0x445d runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" prio=10 tid=0x0000000000670000 nid=0x4462 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000000061e000 nid=0x445e runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000000620000 nid=0x445f runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000000622000 nid=0x4460 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000000623800 nid=0x4461 runnable
"VM Periodic Task Thread" prio=10 tid=0x00000000006a6800 nid=0x447a waiting on condition
JNI global references: 797
Heap
PSYoungGen total 162944K, used 48388K [0x00002aaadff40000, 0x00002aaaf2ab0000, 0x00002aaaf5490000)
eden space 102784K, 47% used [0x00002aaadff40000,0x00002aaae2e81170,0x00002aaae63a0000)
from space 60160K, 0% used [0x00002aaaeb850000,0x00002aaaeb850000,0x00002aaaef310000)
to space 86720K, 0% used [0x00002aaae63a0000,0x00002aaae63a0000,0x00002aaaeb850000)
PSOldGen total 699072K, used 699072K [0x00002aaab5490000, 0x00002aaadff40000, 0x00002aaadff40000)
object space 699072K, 100% used [0x00002aaab5490000,0x00002aaadff40000,0x00002aaadff40000)
PSPermGen total 21248K, used 9252K [0x00002aaab0090000, 0x00002aaab1550000, 0x00002aaab5490000)
object space 21248K, 43% used [0x00002aaab0090000,0x00002aaab09993e8,0x00002aaab1550000)

Not all Throwables are Exceptions. Does your error logging code catch Errors (OutOfMemoryError, StackOverflowError, etc)?

Another couple of possibilities:
The exception might be being thrown on a worker thread that does not log exceptions. You can address this by using Thread.setDefaultUncaughtExceptionHandler(...).
The exception that is being thrown might override the Throwable.fillInStackTrace() method. (This is a long shot ... but apparently some people do this in a misguided attempt to prevent reverse engineering.)