We have a callable class A which actually makes HttpCalls through HttpClient.executeMethod(GetMethod) with a lot of other pre-computations. HttpClient is initialized in the constructor with MultiThreadedHttpConnectionManager.
Another class B creates list of threads for class A through ExecutorService and submits task to the pool and expects future objects to be returned. We have following logic in class B:
for( Future f : futures ){
try{
String str = f.get(timeOut, TimeUnit.SECONDS);
}catch(TimeoutException te){
f.cancel(true);
}
}
This way, our thread gets terminated after a specified time and execution of the task will be terminated and this thread will be available for next task.
I want to confirm the following:
If an external connection is made though HttpClient, how does that get handled on future.cancel of the thread?
In above case or in general, does the http connection pool gets the connection back by properly releasing the previous one? We do release the connection in finally but I don't think interrupting the thread will hit that block.
Could it cause any kind of leak on client or extra resource consumption on the server?
Thanks!
It depends.
If the Http Client uses java.net.Socket, its I/O isn't interrruptible, so the cancel will have no effect.
If it uses NIO, the interrupt will close the channel and cause an exception. At the server this will cause a premature end of stream or an exception on write, either of which the server should cope with corectly.
Related
I am trying to diagnose a problem I am having using WSO2 identity management.
package org.wso2.carbon.identity.mgt;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
/**
* email sender this creates a new task in thread pool for each email sending request
*/
public class NotificationSender {
private static ExecutorService threadPool = Executors.newFixedThreadPool(5);
NotificationSendingModule module;
/**
* creates and submits a task to the thread pool
*
* #param module email sending module as task
*/
public void sendNotification(NotificationSendingModule module) {
threadPool.submit(module);
}
public NotificationSendingModule getModule() {
return module;
}
}
I am load testing the user creation process, and at the point where WSO2 sends a User credentials configuration mail it is sending multiple to the same email address even though they are unique.
I have never used Java but am familiar with C#, so have been able to read through the code without issue, and my questions is:
In the Java docs it mentions "If a thread terminates due to failure during execution and prior to shutdown, a new thread is created to take its place."
Does this mean that if the email send encounters an error then a new thread will begin the process again?
I'm thinking perhaps the send email is erroring so a new thread is created, but the logging with isn't tied into a result is performed anyway.
Also, is it ok to never call
threadPool.shutdown()
When a thread that is part of a thread pool throws an exception it is indeed replaced with a new fresh thread. However it will not retry the same operation. The replacement only occurs so the thread pool can continue do its work when more tasks needs to be executed.
Normally when a thread is terminated in such a fashion, a stack trace is logged, but it is possible the exception is swallowed somewhere. You could try adding a try-catch block around the sending code and logging any exception explicitly to analyze the problem further.
Not calling shutdown is fine.
I am load testing the user creation process, and at the point where WSO2 sends a User credentials configuration mail it is sending multiple to the same email address even though they are unique.
Well, when I hear a Java framework/app server doing identity management + thread pools + strange behavior, what immediatly comes to mind is that most frameworks use a thread per user model (that is : the user identity is tied to the thread. if you switch threads, the user authentication data is lost). Now I do not know if that is the case with SO2, but refer to the documentation. It is the "usual suspect" : thread local authentication mechanisms are everywhere.
In the Java docs it mentions "If a thread terminates due to failure during execution and prior to shutdown, a new thread is created to take its place."
Does this mean that if the email send encounters an error then a new thread will begin the process again?
No. It means that a new thread will be created to handle other unit of works as they are / have been submitted. But the failed unit of work will not be attempted again. As far as the thread pool is concerned, the task completed (with an exception), and it is done with it.
Also, is it ok to never call threadPool.shutdown()
It is not. You should either make your NotificationSender class have a close() or end() method of some sort. Or maybe tie it with some of WSO2 lifecycle callbacks (e.g. in a servlet context, you have listeners for lifecycle events, in a Spring container, you have other create/destroy callbacks, ... whatever works in your context). Failure to shut down a thread pool implies that some threads will hang around, and their resources never freed. Threads are coming pretty cheap nowadays, but they may still pile up and bit you in the long run. It may only be kind of ok if you are sure you only create one NotificationSender in your whole app, and that the lifecycle of this object is the same as your app. Then, essentially, shutting it down is the same as shutting the app down and so nothing bad really happens.
I have a servlet, and it is a singleton.
In the init, I create the producer and the consumers.
consumer = new MyThread();
consumer.start();
and the method run is like this:
#Override
public void run() {
while (!isShutdown()) {
if (queue.isEmpty())
queue.wait();
else
...
}
}
then i have the destroy method.
that set shutdown to true and I use .join
consumer.setShutdown(true);
consumer.join();
but the method .join got lock.. and never finishes.
If I change the queue.wait to Thread.sleep(), it works... but I don't want to change... what I'm doing wrong?
if I don't finish the thread, I got it:
SEVERE: The web application [/test] appears to have started a thread named [Comsumer0] but has failed to stop it. This is very likely to create a memory leak.
Using wait/notify methods is the old way to create a Consumer-Producer pattern.
It is better to use BlockingQueue.
BlockingQueue.put() - for Producer, and BlockingQueue.poll() - for Consumer.
BlockingQueue hides wait method underneath BlockingQueue.poll, BlockingQueue.put and will block a Consumer thread until Producer adds value to queue.
To fix the example above, you should make it wait for a finite amount of time like queue.wait(1000), or wake consumer thread explicitly calling queue.notify().
Most likely the queue is empty and since you have shutdown you never send anything into the queue to cause it to wake up.
Fix this either by sending a dummy message to the queue after setting the shutdown flag or by just removing the shutdown flag completely and sending a shutdown command to the queue.
Also be aware that if your shutdown flag is not volatile (or AtomicBoolean) then one thread may not see it being set by the other due to caching inside the CPU.
This is what the BlockingQueue is for:
BlockingQueue queue = new ArrayBlockingQueue(5);
// ...
while (!isShutdown()) {
Object poll = queue.poll(5, TimeUnit.SECONDS);
}
you should change the design as advised above, or if you don't want to change your code, change your setShutdown() method body and statement to notify the queue.
public void setShutdown(boolean shutdown) {
this.shutdown = shutdown;
this.queue.notifyAll();
}
As the topic suggests I have a server and some clients.
The server accepts I/O connections concurrently (no queueing in socket connections) but I have this troubling issue and I do not know how to bypass it!
If I force a client to throw an I/O Exception the server detects it and terminates the client thread correctly (verified from Task Manager (Windows) and System Monitor (Ubuntu) ). But If I emulate an I/O that is "hanging" like i.e. Thread.sleep(60*1000);or
private static Object lock = new Object();
synchronized(lock) {
while (true) {
try {
lock.wait();
} catch (InterruptedException e) {
/* Foo */
}
}
}
then all subsequent I/O operations (connection & data transfer) seem to block or wait until the "hanging" client is terminated. The applications makes use of the ExecutorService so if the "hanging" client does not complete the operations in the suggested time limit then the task will time out and the client is forced to exit. The subsequent "blocked" I/Os will resume but I wonder why the server doesn't accept any I/O connections or performs any I/O operations when a client "hangs"?
NOTE:The client threading takes place in the server main like this:
while (true) {
accept client connection;
submit client task;
||
\ /
\/
// ExecutorService here in the form
// spService.submit(new Callable<Tuple<String[], BigDecimal[]>>() {
// ... code ... }}).get(taskTimeout, taskTimeUnit);
check task result & perform cleanup if result is null;
otherwise continue;
}
The Problem :
This may very well indicate that your server ACCEPTS client connections concurrently, however, it only handles these connections synchronously. That means that even if a million clients connect, successfully, at any given time, if anyone of them takes a long time (or hangs), it will hold up the others.
The TEST:
To verify this : I would toggle the amount of time a client takes to connect by adding Thread.sleep statments(1000) in your clients.
Expected result :
I believe you will see that even adding a single Thread.sleep(1000) statement in your client delays all other connecting clients by 1000.
I think I have found the source of my problems!
I do use one thread-per-client model but I run my tests locally i.e. in the same machine which means all of them have the same IP! So each client is assigned the same IP with the server! I guess that this leaves server and clients to differ only in port number but since each client is mapped to a different localport for each server connection then the server shouldn't block. I have confirmed that each client and server use different I/Os (compared references) and I wrap their sockets using <Input/Output>Streams to BufferedReaders & PrintWriters but still when a client hangs all other clients hang too (so maybe the I/O channels are indeed the same???)!I will test this on another machine and check the results back with you! :)
EDIT: Confirmed the erratic behaviour. It seems that even with remote clients if one hangs the other clients seem to hang too! :/
Don't know but I am determined to fix this. It's just that it's pretty weird since I am pretty sure I use one thread-per-client (I/Os differ, client sockets differ, IPs seem to be not a problem, I even map each client in the server to a localport of my choice ...)
May be I'll switch to NIO if I don't find a solution soon enough.
SOLUTION: Solved the problem! It seemed that the ExecutorService had to be run in a seperate thread otherwise if an I/O in a client blocked, all I/Os would block! That's strange given the fact that I've tried both an Executors.newFixedThreadPool(<nThreads>); and Executors.newCachedThreadPool(); and the client actions (aka I/Os) should take place in a new Thread for each client.
In any case, I used a method and wrapped the calls so each client instace would use a final ExecutorService baseWorker = Executors.newSingleThreadExecutor(); and created a new Thread explicitly each time using <Thread instance>.start(); so each thread would run in the background :)
I am using org.apache.http and I've this code:
DefaultHttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet(url);
HttpResponse resp = client.execute(get);
HttpEntity entity = resp.getEntity();
InputStream input = entity.getContent();
...
//Read the bytes from input stream
This is the code I am using to download files over Http, I want to cancel the connection(may be user chooses to) What is the graceful way to close the connection. I found 2 ways, Both cancels the download.
Closing inputsteram, input.close(); which causes IOException.
Aborting HttpGet object, get.abort() causes SocketException.
I have try catch, so no erros, but without throwing exception,
is there a way to cancel or abort the connection?
What is the right way to go about it ?
The proper way doing this is sending FIN value to the server side.
How ever in android you do not have the option to be involved in this level, so you can implement by your self using C, or use one of the methods you mention in your question.
Using HttpUriRequest#about is the right way in my opinion. This will cause immediate termination of the underlying connection and its eviction from the connection pool. Newer versions of HttpClient (4.2 and newer) intercept SocketExceptions caused by premature request termination by the user. The problem is that Google ships a fork of HttpClient based on an extremely outdated version (pre-beta1). If you are not able or willing to use a newer version of HttpClient your only option is to catch and discard SocketException in your code.
Use this
client.getConnectionManager().closeExpiredConnections();
client.getConnectionManager().shutdown();
Now you can decide where would you like to write these 2 lines in code.. It will close the connection using the DefaultHttpClient object that you created.
Let me know if this helps you.
Try to cancel the task when you want to interrupt the connection:
task.cancel(true);
This will cancel the task and the threads running in it.
Check this for reference:
public final boolean cancel (boolean mayInterruptIfRunning)
Since: API Level 3
Attempts to cancel execution of this task. This attempt will fail if the task has already completed, already been cancelled, or could not be cancelled for some other reason. If successful, and this task has not started when cancel is called, this task should never run. If the task has already started, then the mayInterruptIfRunning parameter determines whether the thread executing this task should be interrupted in an attempt to stop the task.
Calling this method will result in onCancelled(Object) being invoked on the UI thread after doInBackground(Object[]) returns. Calling this method guarantees that onPostExecute(Object) is never invoked. After invoking this method, you should check the value returned by isCancelled() periodically from doInBackground(Object[]) to finish the task as early as possible.
Parameters
mayInterruptIfRunning true if the thread executing this task should be interrupted; otherwise, in-progress tasks are allowed to complete.
Returns
false if the task could not be cancelled, typically because it has already completed normally; true otherwise
I have a server thread with this code:
public void run() {
try {
ServerSocket server;
EneaLog.printLog("Server is running.");
server = new ServerSocket(this.portnumber);
while (true) {
new EneaServerConnection(server.accept(), this.project,stopped).start();
if (stopped) {
EneaLog.printLog("Server safe-shutdown completed.");
EneaLog.printLog("Hi!");
server.close();
return;
}
}
} catch (IOException ex) {
Logger.getLogger(EneaServer.class.getName()).log(Level.SEVERE, null, ex);
project.getExceptionHandler().handler(ex);
}
}
and a shutdown method like this:
public void shutdown() {
EneaLog.printLog("Server shutdown NOW!");
stopped = true;
}
I want that shutdown can unblock thread that are waiting on server.accept() otherwise I must wait for connection before server shutdown.
I can't do server.close() in shutdown() because I must signal to registered client that server is coming down.
Any ideas?
I try to design my code so that it can be "shutdown" with an interrupt. Mainly, this is because the Executor framework in Java's concurrency package uses interrupt to cancel running tasks. Also, the "shutdown" task doesn't have to know any internals of the task being killed.
However, a call to accept will not respond to an interrupt unless it is created from a ServerSocketChannel. A server created with a ServerSocket constructor will ignore interrupts, and I haven't found a way to reconfigure this.
If you can't change the code that creates the server, arrange for another thread to call close on the server socket. This will also raise an exception in thread blocked on accept, regardless of the method used to create the server socket.
This turns out to be a really big pain when using SSL. A JSSE socket is not created from an InterruptibleChannel, and won't respond to a simple interrupt on the thread.
I just noticed that the question says that the server can't be closed without notifying the client. Successfully interrupting a socket results in its closure.
On a call to accept this shouldn't be a problem, since the client is not connected if the server socket is blocked in accept. That should only be an issue for Socket instances, that represent current connections.
If that doesn't satisfy the notification requirements, a rework to use NIO's ServerSocketChannel in non-blocking mode may be necessary.
You should be able to close the socket from another thread.
Neither interrupt (that's dependent on interrupt points in the same way cancellation is dependent on cancellation points) nor close will do it (accept does not responsed to closing its file descriptor). You'll have to communicate with the accept (try sendto, with a shutdown notification) to notify it to not continue accepting. At least this is the case on linux; don't know what it's like on other platforms.
I've been faced to the same problem. My working solutions consists into closing the ServerSocket object (serverSocket.close()) ; doing this will cause the accept() method to throw a SocketException, which is what you want to do.
Vincent
Have you tried Thread.interrupt() ?
If this thread is blocked in an I/O
operation upon an interruptible
channel then the channel will be
closed, the thread's interrupt status
will be set, and the thread will
receive a ClosedByInterruptException.
If this thread is blocked in a
Selector then the thread's interrupt
status will be set and it will return
immediately from the selection
operation, possibly with a non-zero
value, just as if the selector's
wakeup method were invoked.