Reviewing a stack trace of a non-responsive web-app, I realized that some of the data did not match how-I-thought-tomcat-works.
Context
The application is getting hit in rapid succession for a slow url. These requests seem to pile up, i.e. form a traffic jam in the app server. Logging shows that the number of threads/http connectors have maxed out (number of busy threads has crept up to to the maxThreads value of 120).
Jboss 4.2.2
Uses a variation of tomcat 6.0 called 'jboss-web'
The question
Many of the threads are "doing something"--i.e. reading from the database, writing to the output stream etc...Yet over 50 of the threads are "waiting on the connector to provide a new socket" (From the comments)
What exactly does this mean to the non-socket programmer?
My prior assumptions: wrong
I had assumed that each http thread would 'do its own work'--get the request, do some work, and write the response -- and not need to wait for anything.
so...
What's going on? Could someone clarify the socket-ish stuff?
What implication does such a bottleneck for the tomcat settings? (i.e. increase this setting, decrease that one, etc.)
** Stack Trace **
"http-0.0.0.0-80-90" daemon prio=6 tid=0x695e1400 nid=0x24c in Object.wait() [0x6e8cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x09c34480> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
- locked <0x09c34480> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
Code fragment from Tomcat's org.apache.tomcat.util.net.JIOEndpoint
/**
* Process an incoming TCP/IP connection on the specified socket. Any
* exception that occurs during processing must be logged and swallowed.
* <b>NOTE</b>: This method is called from our Connector's thread. We
* must assign it to our own thread so that multiple simultaneous
* requests can be handled.
*
* #param socket TCP socket to process
*/
synchronized void assign(Socket socket) {
// Wait for the Processor to get the previous Socket
while (available) {
try {
wait();
} catch (InterruptedException e) {
}
}
// Store the newly available Socket and notify our thread
this.socket = socket;
available = true;
notifyAll();
}
thanks
The maxNumThreads setting doesn't affect the servlet container's performance rather than requiring bigger heap and more cpu cycles (if your threads get activated). However, every time you modify that setting to a number bigger than 150, you might think of a bottleneck in your application.
The web server is not designed to handle more than 100 simultaneous requests. If you do find yourself in such a situation, consider clustering. I see you are using jbossweb and there is a really nice article here:
http://refcardz.dzone.com/refcardz/getting-started-jboss
However, as I don't think you have more than 100 simultaneous requests, I think it is a bottleneck in your applications. Things to check are your jdbc driver, the version of jdk you use, the tomcat version (in your case 6.0). Requests to your application should finish in less than 1 second - network latency (and even that is a way too huge delay ) and if you find they take more, it is probably somewhere in your code. Do you manually close/open your database connections, do you use efficient threading in the background, do you use JMS. That are the things to look at usually. The other might be a bug in your particular servlet container version.
P.S If you do decide to use a higher number of max threads, it might make sense to decrease/increase the thread stack size and see how it affects performance. If you have long lived threads( which should not be), you might want to increase the stack size. If you have short - lived threads, try decreasing the stack size to preserve a bit memory.
-Xss is the flag.
Also, i just saw the jboss AS version you are using. Check that also. Now that I look at your symptoms, I believe your problem is somewhere in the configuration files.
Related
I'm trying to see difference between DirectMessageListener and SimpleMessageListener. I have this drawing just to ask if it is correct.
Let me try to describe how I understood it and maybe you tell me if it is correct.
In front of spring-rabbit there is rabbit-client java library, that is connecting to rabbit-mq server and delivering messages to spring-rabbit library. This client has some ThreadPoolExecutor (which has in this case I think - 16 threads). So, it does not matter how many queues are there in rabbit - if there is a single connection, I get 16 threads. These same threads are reused if I use DirectMessageListener - and this handler method listen is executed in all of these 16 threads when messages arrive. So if I do something complex in handler, rabbit-client must wait for thread to get free in order to get next message using this thread. Also if I increase setConsumersPerQueue to lets say 20, It will create 20 consumer per queue, but not threads. These 20*5 consumers in my case will all reuse these 16 threads offered by ThreadPoolExecutor?
SimpleMessageListener on the other hand, would have its own threads. If concurrent consumers == 1 (I guess default as in my case) it has only one thread. Whenever there is a message on any of secondUseCase* queues, rabbit-client java library will use one of its 16 threads in my case, to forward message to single internal thread that I have in SimpleMessageListener. As soon as it is forwarded, rabbit-client java library thread is freed and it can go back fetching more messages from rabbit server.
Your understanding is correct.
The main difference is that, with the DMLC, all listeners in all listener containers are called on the shared thread pool in the amqp-client (you can increase the 16 if needed). You need to ensure the pool is large enough to handle your expected concurrency across all containers, otherwise you will get starvation.
It's more efficient because threads are shared.
With the SMLC, you don't have to worry about that, but at the expense of having a thread per concurrency. In that case, a small pool in the amqp-client will generally be sufficient.
I have questions regarding the performance tuning.
I'm using Linux 64bit server, java 1.8 with wildfly 10.0.0.final . I developed a webservice which uses thread factory and managed executor service through the wildfly configuration.
the purpose of my webervice is to receive the request which has large data, save data, and then create a new thread to process this data, then return response to request. This way the webservice can return response quickly without waiting for data processing to finish.
The configured managed-executor-service holds a thread pool config specifically for this purpose.
for my understanding in configuration, the core-thread defines how many threads will be alive in the thread pool. when core-thread is full, new requests will be put in queue, when queue is full, then new threads will be created, but these newly created thread will be terminated after some time.
I'm trying to figure out what is the best combination to set the thread pool. The following is my concerns:
if this core-thread is set too small (like 5), maybe the responding time will be long because only 5 active threads are processing data, the rest are put in queue until queue is full. the response time won't look good at heavy load time
if I set core-thread to be big, (like 100 maybe), that means even the system is not busy, there still will be 100 live threads in the pool. I don't see any configuration that can allow these threads to be terminated. I'm concerned it is too many live threads idle.
Does anyone have any suggestions on how to set parameters to handle both heavy load and light load situation without too many idle threads left in pool? I'm actually not familiar with this area, like how many idle threads means too many, how to measure it.
the following is the configuration for thread factory and managed-executor-service.
<managed-thread-factory name="UploadThreadFactory" jndi-name="java:jboss/ee/concurrency/factory/uploadThreadFactory"/>
<managed-executor-service name="UploadManagedExecutor" Jodi-name="java:jboss/ee/concurrency/executor/uploadManagedExecutor" context-service="default" thread-factory="UploadThreadFactory" hung-task-threshold="60000" core-thread="5" max-thread="100" keep-alive-time="5000" queue-length="500"/>
Thanks a lot for your help,
Helen
I am performance testing an application in WebLogic and at some point the performance starts degrading and I see all the threads that are serving http requests are stuck in this code
sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
weblogic.socket.NIOSocketMuxer$NIOOutputStream.writeInternal(NIOSocketMuxer.java:1090)
weblogic.socket.NIOSocketMuxer$NIOOutputStream.write(NIOSocketMuxer.java:983)
weblogic.socket.JSSEFilterImpl.writeToNetwork(JSSEFilterImpl.java:773)
weblogic.socket.JSSEFilterImpl.wrapAndWrite(JSSEFilterImpl.java:734)
I believe the above means WebLogic does not have enough channel to send the responses back but I am not sure what setting is related to that. I have checked the file limit on the OS and it is set to unlimited. I am not sure what else I need to check. Any ideas? The code is running on a Solaris box running SunOS.
EDIT: Thread pool has been configured to initialize with minimum of 250 threads and can grow to 400 threads. At the point of the problem we are at 250 threads and only 60 of them are serving requests.
Threads hang on selecting a channel
No.
I believe the above means WebLogic does not have enough channel to send the responses back
No.
It is one thread blocking while selecting on a network event, and it means there is no incoming data, no space in the socket send buffers, etc., depending on what events are being selected on.
In this case is it being invoked by a write further down the stack, which indicates lack of space in the socket send buffer, which indicates that the peer is slow reading.
There's nothing you can do about that at this end.
I encounter a problem with java threads and I am not sure if its related to my approach or if thread pooling is going to resolve what I am trying to achieve.
for (int i = 0; i < 100; i++) {
verifier[i]=new Thread();
verifier[i].start();
}
I initialize 100 threads and start them. In the threads the code that gets executed is just
con = (HttpURLConnection) website.openConnection(url);
// gets only the header
con.setRequestMethod("HEAD");
con.setConnectTimeout(2000); // set timeout to 2 seconds
These threads repeat the process above over a long list of url/data.
The first 50 threads execute almost instantly then they just stop for 60 seconds or so and then there is another spike of execution that 20 of them or so finish at the same time and so on. The same deadlock occurs even if there is 4 of them.
My first guess was a deadlock. I am not sure how to resolve the issue and maintain a constant execution pace, without deadlocks and stops.
I am looking for an explanation of why this occurs and how it can be resolved.
By DeadLock I reefer to the Java Virtual Machine and how it handles thread. Not deadlock caused by my threads.
SCREENSHOT OF THREAD EXECUTION:
It looks like the threads are dying for no reason and I don't know why?!
It could be that the operating system configurable limit of tcp/ip connections gets hit, which causes the JVM to block waiting for a new TCP/IP connection to be created, which will only happen if a connection already used get's closed.
This could help to find what is going on:
profile the run with visualvm which comes with the JVM itself (run it on the command line with jvisualvm). There should be indication of how many threads are created and why are they blocked, deadlocks, etc.
Wait for it to block and take thread dumps of the JVM process to check for deadlocks in the thread stack traces using jstack or visualvm, search for the deadlock keyword.
Check with netstat -nao the state of your TCP connections, to see if the operating system limit is getting hit, if there are many connections in CLOSE_WAIT at the times the blocking occurs
Are you behind a corporate proxy/firewall, you could be hitting some other sort of security limit that prevents you from opening more TCP connections, not necessarily the limit of the operating system
If none of this helps you can always edit the question with further findinds, but based on the description of the code other limits are getting hit that on a first look don't seem related to JVM thread deadlocks, hope this helps.
I've a Java client which accesses our server side over HTTP making several small requests to load each new page of data. We maintain a thread pool to handle all non UI processing, so any background client side tasks and any tasks which want to make a connection to the server. I've been looking into some performance issues and I'm not certain we've got our threadpool set up as well as possible. Currently we use a ThreadPoolExecutor with a core pool size of 8, we use a LinkedBlockingQueue for the work queue so the max pool size is ignored. No doubt there's no simple do this certain thing in all situations answer, but are there any best practices. My thinking at the moment is
1) I'll switch to using a SynchronousQueue instead of a LinkedBlockingQueue so the pool can grow to the max pool size figure.
2) I'll set the max pool size to be unlimited.
Basically my current fear is that occasional performance issues on the server side are causing unrelated client side processing to halt due to the upper limit on the thread pool size. My fear with unbounding it is the additional hit on managing those threads on the client, possibly just the better of 2 evils.
Any suggestions, best practices or useful references?
Cheers,
Robin
It sounds like you'd probably be better of limiting the queue size: does your application still behave properly when there are many requests queued (is it acceptable for all task to be queued for a long time, are some more important to others)? What happens if there are still queued tasks left and the user quits the application? If the queue growing very large, is there a chance that the server will catch-up (soon enough) to hide the problem completely from the user?
I'd say create one queue for requests whose response is needed to update the user interface, and keep its queue very small. If this queue gets too big, notify the user.
For real background tasks keep a separate pool, with a longer queue, but not infinite. Define graceful behavior for this pool when it grows or when the user wants to quit but there are tasks left, what should happen?
In general, network latencies are easily orders of magnitude higher than anything that can be happening in regards to memory allocation or thread management on the client side. So, as a general rule, if you are running into a performance bottle neck, look first and foremost to the networking link.
If the issue is that your server simply can not keep up with the requests from the clients, bumping up the threads on the client side is not going to help matters: you'll simply progress from having 8 threads waiting to get a response to more threads waiting (and you may even aggravate the server side issues by increasing its load due to higher number of connections it is managing).
Both of the concurrent queues in JDK are high performers; the choice really boils down to usage semantics. If you have non-blocking plumbing, then it is more natural to use the non-blocking queue. IF you don't, then using the blocking queues makes more sense. (You can always specify Integer.MAX_VALUE as the limit). If FIFO processing is not a requirement, make sure you do not specify fair ordering as that will entail a substantial performance hit.
As alphazero said, if you've got a bottleneck, your number of client side waiting jobs will continue to grow regardless of what approach you use.
The real question is how you want to deal with the bottleneck. Or more correctly, how you want your users to deal with the bottleneck.
If you use an unbounded queue, then you don't get feedback that the bottleneck has occurred. And in some applications, this is fine: if the user is kicking off asynchronous tasks, then there's no need to report a backlog (assuming it eventually clears). However, if the user needs to wait for a response before doing the next client-side task, this is very bad.
If you use LinkedBlockingQueue.offer() on a bounded queue, then you'll immediately get a response that says the queue is full, and can take action such as disabling certain application features, popping a dialog, whatever. This will, however, require more work on your part, particularly if requests can be submitted from multiple places. I'd suggest, if you don't have it already, you create a GUI-aware layer over the server queue to provide common behavior.
And, of course, never ever call LinkedBlockingQueue.put() from the event thread (unless you don't mind a hung client, that is).
Why not create an unbounded queue, but reject tasks (and maybe even inform the user that the server is busy (app dependent!)) when the queue reaches a certain size? You can then log this event and find out what happened on the server side for the backup to occur, Additionally, unless you are connecting to a multiple remote servers there is probably not much point having more than a couple of threads in the pool, although this does depend on your app and what it does and who it talks to.
Having an unbounded pool is usually dangerous as it generally doesn't degrade gracefully. Better to log the problem, raise an alert, prevent further actions being queued and figure out how to scale the server side, if the problem is there, to prevent this happening again.