Develop Reference

Java blocked thread on String.toUpperCase()

One of the threads has a lock for more than 3 seconds when querying Oracle Database. This causes many blocked threads when accesing Oracle database, and hence sudden increases in number of threads and unresposiveness of application. Im am using Tomcat 8.5, Tomcat connection pool, Java 8. Trace for blocking thread:
***"http-nio-80-exec-433" #4207 daemon prio=5 os_prio=0 tid=0x00007fd9d8042000 nid=0x503b runnable [0x00007fd839f04000]
java.lang.Thread.State: RUNNABLE
at java.util.Hashtable.get(Hashtable.java:363)
- locked <0x000000070193caa0> (a java.util.Hashtable)
at java.lang.ConditionalSpecialCasing.lookUpTable(ConditionalSpecialCasing.java:151)
at java.lang.ConditionalSpecialCasing.toUpperCaseEx(ConditionalSpecialCasing.java:123)
at java.lang.String.toUpperCase(String.java:2775)
at java.lang.String.toUpperCase(String.java:2833)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1638)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4401)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:4482)
- locked <0x000000074cd7d868> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:6272)
at sun.reflect.GeneratedMethodAccessor400.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210)
at com.sun.proxy.$Proxy637.executeQuery(Unknown Source)
at sun.reflect.GeneratedMethodAccessor400.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114)
at com.sun.proxy.$Proxy637.executeQuery(Unknown Source)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1953)
at org.hibernate.loader.Loader.doQuery(Loader.java:802)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2533)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)***
Here is a trace for one of the 10+ BLOCKED threads
***"http-nio-80-exec-271" #2777 daemon prio=5 os_prio=0 tid=0x00007fd9c8941800 nid=0x19c3 waiting for monitor entry [0x00007fd8356ca000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.Hashtable.get(Hashtable.java:363)
- waiting to lock <0x000000070193caa0> (a java.util.Hashtable)
at java.lang.ConditionalSpecialCasing.lookUpTable(ConditionalSpecialCasing.java:151)
at java.lang.ConditionalSpecialCasing.toUpperCaseEx(ConditionalSpecialCasing.java:123)
at java.lang.String.toUpperCase(String.java:2775)
at java.lang.String.toUpperCase(String.java:2833)
at oracle.jdbc.driver.CharCommonAccessor.init(CharCommonAccessor.java:164)
at oracle.jdbc.driver.VarcharAccessor.<init>(VarcharAccessor.java:88)
at oracle.jdbc.driver.T4CVarcharAccessor.<init>(T4CVarcharAccessor.java:108)
at oracle.jdbc.driver.T4CTTIdcb.fillupAccessors(T4CTTIdcb.java:431)
at oracle.jdbc.driver.T4CTTIdcb.receiveCommon(T4CTTIdcb.java:209)
at oracle.jdbc.driver.T4CTTIdcb.receive(T4CTTIdcb.java:145)
at oracle.jdbc.driver.T4C8Oall.readDCB(T4C8Oall.java:963)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:447)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:235)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:543)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:239)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:1246)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1500)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1717)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4401)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:4482)
- locked <0x000000074d203f60> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:6272)
at sun.reflect.GeneratedMethodAccessor400.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210)
at com.sun.proxy.$Proxy637.executeQuery(Unknown Source)
at sun.reflect.GeneratedMethodAccessor400.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114)
at com.sun.proxy.$Proxy637.executeQuery(Unknown Source)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)***
I have no idea why toUpperCase() would lock something (it is some Integer object being locked, as far as i found) for 30+ seconds, but this keeps occuring multiple times per day. Thread dump analysers did not found any deadlocks in dump. Tomcat pools logs that query for blocking thread http-nio-80-exec-433 took 5 minutes to complete.
Could this be a problem with jvm, memory or something else? Like jdbc driver or connection pool configuration problem?

It appears the problem was not code related. We had 10GB size catalina.out log file, and 4 bash scripts which analyzed that file for specific errors every five minutes, and because of large file size, each such analysis (mostly tail/wc commands) took 3-4 minutes. I do not know if catalina.out was being locked, but CPU usage for "tail" and "wc" commands was quite significant. Memory usage did not increase significantly though.
After manually rolling catalina.out, the problem is gone. Admin has been tasked with figuring out why logrotate is failing.
Update: The problem kept reappearing under higher loads (>50 users), so after some testing, locale was changed from "lt" to "en". Together with fixing another MyFaces cache bug, response times from Tomcat decreased 10-20 times, and number of concurrent users that can use application increased >10 times.

Java thread hung in a native method

While parsing a date string the thread hung in the native method. The thread is stopped in the same state for the last 18 hours(Thread dumb in different times shows the same show the same stack for the thread in question).
Please suggest what might be the error
"DateParsingThread - 001" #228 daemon prio=5 os_prio=0 tid=0x00007f98e0029000 nid=0x9c2 runnable [0x00007f98016ee000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.doPrivileged(Native Method)
at sun.util.resources.LocaleData.getBundle(LocaleData.java:163)
at sun.util.resources.LocaleData.getDateFormatData(LocaleData.java:127)
at sun.util.locale.provider.LocaleResources.getCalendarNames(LocaleResources.java:325)
at sun.util.locale.provider.CalendarNameProviderImpl.getDisplayNamesImpl(CalendarNameProviderImpl.java:157)
at sun.util.locale.provider.CalendarNameProviderImpl.getDisplayNames(CalendarNameProviderImpl.java:139)
at sun.util.locale.provider.CalendarDataUtility$CalendarFieldValueNamesMapGetter.getObject(CalendarDataUtility.java:178)
at sun.util.locale.provider.CalendarDataUtility$CalendarFieldValueNamesMapGetter.getObject(CalendarDataUtility.java:154)
at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObjectImpl(LocaleServiceProviderPool.java:281)
at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObject(LocaleServiceProviderPool.java:265)
at sun.util.locale.provider.CalendarDataUtility.retrieveFieldValueNames(CalendarDataUtility.java:88)
at java.util.Calendar.getDisplayNames(Calendar.java:2178)
at java.text.SimpleDateFormat.getDisplayNamesMap(SimpleDateFormat.java:2366)
at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:1972)
at java.text.SimpleDateFormat.parse(SimpleDateFormat.java
at java.text.DateFormat.parse(DateFormat.java:364)
Any suggestions to find the solutions are welcome.

Java / Wildfly: Threads piling up / Eclipse shows one additional running thread for each request

I am using javas Thread to connect via SMTP to our mailprovider as this can take some time until it finishes and I dont want the request to wait.
But it looks like the threads are not closed after they are finished.
I noticed this in the debug mode of Eclipse:
For each time I create a new Thread(), it adds one running thread, but it is not closing it (at least I assume this, as eclipse still shows Running).
This is my code:
Thread mailThread = new Thread() {
public void run() {
System.out.println("Does it work?");
try {
Transport t = session.getTransport("smtp");
t.connect("user","pass");
t.sendMessage(message,message.getAllRecipients());
t.close();
System.out.println("SENT");
return;
} catch (MessagingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return;
}
}
};
mailThread.start();
Is this working as intended? Or does Running in eclipse mean something different?

I suggest not only to use the debugger to see, to see which threads you have at a certain point in time. Debuggers might display threads which are active during a break point but should not be there under normal conditions.
It is preferrable to use the command line tool jstack to create thread dumps. This will dump all the threads in a JVM at a certain point in time.
Here are some instructions on how to use it: https://helpx.adobe.com/uk/experience-manager/kb/TakeThreadDump.html
Another thing could help you debugging and finding threads in the dump: give threads a name using the string in one of the constructor.
new Thread("foo")
Then it becomes easier to find these in the thread dump.
If you call a thread "foo" then it will show up in a thread dump like this:
"foo" #16 prio=5 os_prio=0 tid=0x0000000041970800 nid=0x41f8 waiting on condition [0x000000004244e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base#9/Native Method)
at stackoverflow.ThreadReferenceTest$1.run(ThreadReferenceTest.java:14)
Locked ownable synchronizers:
- None
"Service Thread" #15 daemon prio=9 os_prio=0 tid=0x0000000041914000 nid=0x3d90 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None

How to figure out why thread is stuck?

I see this thread in my jstack that does not appear to be moving at all. Any pointers on how to figure out why it's stuck? I don't see any locks or anything, the only suspicious thing is the "Object.wait()" reference.
"main" prio=10 tid=0x00007f3a8000b000 nid=0x942 in Object.wait() [0x00007f3a89539000]
java.lang.Thread.State: RUNNABLE
at org.joda.time.DateTimeZone.<clinit>(DateTimeZone.java:95)
at org.joda.time.format.DateTimeFormatter.withZoneUTC(DateTimeFormatter.java:301)
at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at java.lang.Class.newInstance0(Class.java:372)
at java.lang.Class.newInstance(Class.java:325)
at com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:121)
at com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:107)
at com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:80)
at com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:311)
at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:284)
at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160)
Also, line 95 in DateTimeZone.java at the top of the stack is this:
public static final DateTimeZone UTC = new FixedDateTimeZone("UTC", "UTC", 0, 0);
There is another thread that's also stuck in a similar place:
"FeatureManagerService" daemon prio=10 tid=0x00007f3a8056a800 nid=0x94f in Object.wait() [0x00007f3a84151000]
java.lang.Thread.State: RUNNABLE
at com.amazonaws.util.DateUtils.<clinit>(DateUtils.java:35)
at com.amazonaws.services.s3.internal.ServiceUtils.<clinit>(ServiceUtils.java:59)
at com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:123)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:348)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:245)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3711)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3664)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:620)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:603)
And DateUtils.java:35 is:
private static final DateTimeZone GMT = new FixedDateTimeZone("GMT", "GMT", 0, 0);
I already tried looking into it with jvisualvm/jhat but didn't really get very far.
Note that this is a live process, not something I am running in my debugger locally and after restart it works fine so it appears to be intermittent.
Any help would be appreciated!
Thanks!
Update using the mixed mode in jstack seems to give some more insight - it's waiting on a pthread_cond_wait:
----------------- 2370 -----------------
0x00007f3a89115414 __pthread_cond_wait + 0xc4
0x00007f3a8833a03c _ZN13ObjectMonitor4waitElbP6Thread + 0x7dc
0x00007f3a88117fbb _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread + 0x36b
0x00007f3a881182ca _ZN13instanceKlass10initializeEP6Thread + 0x6a
0x00007f3a8814d3f3 _ZN18InterpreterRuntime4_newEP10JavaThreadP19constantPoolOopDesci + 0x143
0x00007f3a7d01d9ee * org.joda.time.DateTimeZone.<clinit>() bci:0 line:95 (Interpreted frame)
0x00007f3a7d0004f7 <StubRoutines>
...

Maybe it's not stuck. It's just calling new DateTimeZone() in a loop, and the constructor does some computations. Every time you look at this thread, it's inside DateTimeZone() - but it's a different DateTimeZone() each time.
Which then gets discarded. Happened to me quite a few times.

As found by #naumcho, this proved to be a bug (https://github.com/JodaOrg/joda-time/issues/171).
Based on the information provided (stack traces of two different threads + source line) one could have suspected a deadlock because both threads are trying to instantiate a new object of the same type FixedDateTimeZone.
The next step to confirm that would be to use GDB to inspect the stack frames around __pthread_cond_wait().

HttpURLConnection getting locked

I have a thread running under tomcat which creates a HttpUrlConnection and reads it through BufferedInputStream.
After fetching data for some urls, it stalls. I got the jstack of the process which says HttpUrlConnection is locked and BufferedInputStream is also locked.
"http-8080-1" daemon prio=10 tid=0x08683400 nid=0x79c9 runnable [0x8f618000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x956ef8c0> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1072)
- locked <0x956ef910> (a sun.net.www.protocol.http.HttpURLConnection)
Could somebody help here.
Thanks

You probably have a problem on the other end. read() on an InputStream is a blocking operation - from the javadoc (http://java.sun.com/javase/6/docs/api/): "This method blocks until input data is available, the end of the stream is detected, or an exception is thrown."
Is the server on the other end responding? Do you know if it's sent anything?
edit: To make it clearer, the thread is in RUNNABLE state, so you're not deadlocked - it sounds like that's what you're thinking that it is, but there's no evidence here of any deadlock.