Develop Reference

Deduce why threads have increased in java application

Context
I have a metric on a server that publishes the amount of threads that I have at any given time. With a recent deployment, I have noticed the number of threads increase by about 30 threads on average (originally stagnated around 370, now at 400 threads consistently).
What I've done
There are many packages/possibilities that could be the root cause for this increase. This is why I looked into analyzing threads. I learned how to get and got a thread dump but I can't see any useful information for me on why these threads were created/how they are used.
My service is not impacted negatively (latency/CPU/Memory) but I would still like to root cause this issue as it could be grounds for a memory leak.
My Question
If there is some resource to be able to get the class/package that created the thread, that would be very helpful (I have searched online for a while for a resource like that).
Any advice to root cause this is much appreciated!

Find The Source Of Threads
Use Java Executors instead of dealing with Threads directly if you aren't already. If you are using Executors you can define names for the threads in your thread pool. These names are included in thread dumps of your java process. So if a thread has one of your custom defined names you will know which thread pool created it. It's common practice to maintain different thread pools for different types of tasks in your application and to give the threads in each pool different names. This way the you can determine how many threads each part of your application is creating when you take a thread dump. See this question for how to define custom thread names.
How To Limit Threads
The ThreadPoolExecutor allows you define the maximum number of
Threads in the pool. So if you know all the pools you are using and what their maximum sizes are you should be able to define the maximum number concurrent Threads that can run in your application.
You may have to pay special attention to server, client, and networking libraries though since they probably create their own threads. Libraries typically name their threads as well, so you can probably google an unfamiliar name or stack trace in your thread dump to figure out where it's coming from.

Wildfly 8 CPU, Memory issue

While perform a load testing on a application in Wildfly-8.0, both memory and cpu are hiked up. After stopped the testing both memory and cpu went down to 50% but the server fails to accept any request even from the server hosted machine facing the same problem with wildfly console
No clue has been found when we monitor the object created in server through Visual VM, so is this issue with wildfly-8.0 version since we wonder why the application server doesn't accept any request even after the resource consumption went below 50%.

First, check the log files. Look for any unexplained exceptions. (OOME's in particular can lead to lockups.)
Next, use jstack or similar to get a dump of the thread stacks. Check that the listener thread is still alive and that there idle worker threads ready to process requests.
There are a variety of things that can cause lockups under heavy load. Common syndromes include:
OOMEs causing threads to die, leaving data structures locked, or other threads waiting for notify events or similar that are never going to arrive.
Synchronization or similar problems triggered by the load.

Problems caused due to explicitly creating threads in java

I am dealing with below OutOfMemory exception in WAS 6.1.
Exception in thread "UnitHoldingsPolicySummary" java.lang.OutOfMemoryError: unable to create new native thread.
I have done a lot of research on this to prevent this. After Googling, I have found that, this happens when the Native memory gets exhausted due to creation of lots of threads concurrently.
Now, after analysing the below logs, we can figure out that, inside the application, the threads are created explicitely, which I read is a very very bad practice. (Can experts please confirm this?)
07/07/14 08:50:38:165 BST] 0000142c SystemErr R Exception in thread "xxxxxx" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at com.fp.sv.controller.business.thread.xxxxxxxxxexecute(Unknown Source)
at com.fp.sv.controller.business.thread.xxxxxxxxx.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)
I am more into WAS administration and doesn't possess much knowledge on Java and thread creation in Java. Now I need to discuss this with developer, but before that I want to be 100% confirmed that my findings are correct and developers should correct the code by not explicitely creating the threads.
What all things that I need to check on application server side before blaming this on code?
On solaris, I am firing the command pmap -x 9547|grep -i stack|wc -l to check how many threads are getting created on that instance of time. I could see during the 'OutOfMemory' issue, this number is very high.
Could you please confirm whether this command is the good way to checknumber of threads currently active?
Editing the question with my latest findings
Also, when this issue happens, at the same one of the MQ queue gets piled up as WAS doesn't pick up the messages from the queue. I could see below error in the application specific logs.
Non recoverable Exception detected whilst connecting to queue manager or response queue
Underlying reason = MQJE001: Completion Code 2, Reason 2102
Can this issue related to MQ as well?Which in turn causes OutOfMemory issue?
Regards,
Rahul

There are different possibilities of implementing a threading system for a virtual machine. The two extreme forms are:
Green threads: All Java Thread instances are managed within one native OS thread. This can cause problems if a method blocks within a native invocation what makes this implementation complex. In the end, implementers need to introduce renegade threads for holding native locks to overcome such limitations.
Native threads: Each Java Thread instance is backed by a native OS thread.
For the named limitations of green threads, all modern JVM implementations, including HotSpot, choose the later implementation. This implies that the OS needs to reserve some memory for each created thread. Also, there is some runtime overhead for creating such a thread as it needs direct interaction with the underlying OS. At some point, these costs accumulate and the OS refuses the creation of new threads to prevent the stability of your overall system.
Threads should therefore be pooled for resue. Object pooling is normally considered bad practice as many programers used it to ease the JVM's garbage collector. This is not longer useful as modern garbage collectors are optimized for handling short-living objects. Today, by pooling objects you might in contrary slow down your system. However, if an object is backed by costly native resources (as a Thread), pooling is still a recommended practice. Look into the ExecutorService for a canonical way of pooling threads in Java.
In general, consider that context switches of threads are expensive. You should not create a new thread for small tasks, this will slow your application down. Rather make your application less concurrent. You only have a number of cores which can work concurrently in the first place, creating more threads than your (non-virtual) cores will not improve runtime performance. Are you implementing some sort of divide-and-conquer algorithm? Look into Java's ForkJoinPool.

Yes, it's a bad practice. Normally, you don't manage threads inside a Java EE server. By "normally" I mean "while developing business applications".
According to http://www.oracle.com/technetwork/java/restrictions-142267.html:
Why is thread creation and management disallowed?
The EJB specification assigns to the EJB container the responsibility
for managing threads. Allowing enterprise bean instances to create and
manage threads would interfere with the container's ability to control
its components' lifecycle. Thread management is not a business
function, it is an implementation detail, and is typically complicated
and platform-specific. Letting the container manage threads relieves
the enterprise bean developer of dealing with threading issues.
Multithreaded applications are still possible, but control of
multithreading is located in the container, not in the enterprise
bean.
However, I don't think your logs demonstrate that threads are being created explicitly. If you want to be 100% sure, decompile the deployables and look at the code in those lines.
Also take a look at this:
"java.lang.OutOfMemoryError : unable to create new native Thread"
And this:
https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread
Concerning the number of threads used by your app, I'd try to use a monitoring tool like JConsole, or VisualVm.

Thread status TimedWait. How to debug?

My application run some complex threads that fetch maps in a background thread and draw them. Sometimes if I run the app for a couple hours on a slow network I seem to be getting it into a weird state where all my threads status are showing TimedWait or Wait (except the ones that are Native such as main).
What is the cause of this? How can I debug it? I am absolutely lost and I know this is a bit of a general question but I would appreciate it if someone could point me to the right direction. EG:
How to pin point the cause of the problem.
What king of issues generally cause all the threads to lock up?
Anybody seen anything similar?
Thanks

A timed wait is simply a thread which is blocked on some O/S level call which has a timeout specified, such as a simple wait primitive (Object.wait()) or a socket operation (Socket read()/write()), a thread queue etc. It's quite normal for any complex program to have several or many of these - I have an application server which routinely has hundreds, even thousands.
Your threads may be backing up on non-responsive connections and may not be misbehaving at all, per se. It may simply be that you need to program them to detect and abort an idle connection.
Click on each of the threads which you are concerned about and analyze their stack trace for how they got there.
Most decent profiling tools (and application containers) will have the option of printing a full stack trace, and more modern ones will do a dead-lock and live-lock analysis for you. The JVisualVM tool distributed with Sun's JDK and available on the net as VisualVM will do this and it's very effective. Most decent profilers will also show lock acquisition in the stack trace (yours, above, is not in that view).
Otherwise, you are looking for two or more threads contending for the same lock or acquiring the same locks in a different order. You may need to do this manually by actually examining the source and annotating your stack trace, but you should be able to whittle down likely candidates if your tool doesn't point right to the conflicting threads.

How to get stack trace of a thread

I have a multithreaded application. Several messages are coming to the application and are processed in separated threads. For this I am using classes ThreadPoolExecutor and FutureTask from package java.util.concurrent.
Occasionally I have some deadlocks in the application. When a deadlock occurs I want to interrupt the blocking thread and I want to log the stack trace of this thread so that I can later resolve the deadlock.
Is there any way how can we find the stack trace of a thread outside of that thread in Java?

See here for how to generate stack traces, including how to do this programatically. From the console, Ctrl+Break will dump the stack traces to stdout. See also this SO question for more details.

You could log the stack traces of all thread from time to time (or before killing the process) from within your application. To do that use:
Map<Thread, StackTraceElement[]> m = Thread.getAllStackTraces();
for(Map.Entry<Thread, StackTraceElement[]> e : m.entrySet()) {
log(e.getKey().toString());
for (StackTraceElement s : e.getValue()) {
log(" " + s);
}
}
When running nightly automated tests, sometimes some one of the test cases gets into a deadlock. I added a "TimeBomb" daemon thread that waits 30 minutes, and if then logs all stack traces as above.

Before entering the deadlock region, set a field like,
thread = Thread.currentThread();
In your monitoring thread you can perform thread.getStackTrace(); to get the stack trace of that thread at any time.

You use JStack. Here is a nice blog entry that details how to get stack traces.

I wasn't sure if you wish to obtain the stacktrace from within the same JVM or externally, but if you wish to obtain the stack trace with external tools, the following will help:
The Java VisualVM tool can be used to connect to the running JVM, where the thread stack can be dumped. This is usually the preferred approach for most people using Java 6. Once VisualVM is launched, the thread dump of the process can be obtained by selecting the process in the Application tab. A Threads tab is now made available to view the threads in the running process, and in the same tab you'll find the "Thread Dump" button to extract the required information.
The jstack utility within the JDK can also be used to produce thread stacktraces.

When a deadlock occurs I want to interrupt the blocking thread ...
You could implement a periodic task to check for deadlocks (where deadlocks are java intrinsic or Lock based) and call interrupt on all threads involved in the scenario. However, this has no guarantees that it will solve your problem. Its likely the scenario that will just happen again. See Dr Heinz's article on a deadlock detector for details.
If fact, there is no guarantee that interrupt will even free up a blocked process like this. Its a far better approach to avoid the deadlock scenario in the first place by, for example, using locks with timeouts and retry strategies or 'try before you buy' approaches.
and I want to log the stack trace of this thread...
If you want to do this programatically, again, follow Dr Heinz's example. If not, just generate the thread dump when you've spotted the problem.
Is there any way how can we find the stack trace of a thread outside of that thread in Java?
Yes and no. You can dump the threads from other VMs but their stack traces may not be as useful as you might think to determining the causes of your deadlock. If a genuine deadlock has been detected (by the JVM itself on thread dump of your applications VM) you should have everything you need to debug the cause (more or less).