I'm indexing pdf files using Apache lucene with threads. Some threads take time more than 15 minutes. After 15 minutes of thread execution it will throw Thread interrupted Exception.Is there a way to increase the time limit to avoid this issue.
I got this exception when there is a single thread running and it indexed nearly 76% of its pdf files.
application server is Glassfish
List<Thread> threads = new ArrayList<Thread>();
Thread worker;
for (int a = 1;a <= count; a++) {
IndexManualRunnable indexManualRunnable =
new IndexManualRunnable(indexfileLocation, collections, manual, processId);
worker = new Thread(indexManualRunnable);
worker.setName(manual.getName());
worker.setPriority(Thread.MAX_PRIORITY);
worker.start();
threads.add(worker);
}
for (Thread thread : threads) {
try {
thread.join();
} catch (InterruptedException interruptedException) {
saveReport("", "", "Interrupted Exception", 1, processId, Category.INDEXING, thread.getName());
interruptedException.printStackTrace();
}
}
UPDATE:
I see that you are using Glassfish and are saying this interrupt is occurring every time at 15 minutes. It appears Glassfish is set to timeout at around 900 seconds which is 15 minutes by default - this throws an InterruptException.
Since your application viably needs to process for longer than 15 minutes, update the following server config to a time limit you see fit.
http.request-timeout-seconds
Here is an example asadmin command to update the property but I have not tested it:
# asadmin set server-config.network-config.protocols.protocol.<listener-name>.http.request-timeout-seconds=-1
- NOTE: <listener-name> is the name of the listener they wish to disable the timeout on.
- (-1) means unlimited
To deal with an interrupted thread you can catch and handle the exception yourself. If you want the thread to continue executing regardless you would theoretically do nothing - but to keep the code clean and proper I would implement it as below:
boolean interrupted = false;
try {
while (true) {
try {
return queue.take();
} catch (InterruptedException e) {
interrupted = true;
// fall through and retry
}
}
} finally {
if (interrupted)
Thread.currentThread().interrupt();
}
I like this example because it does not just leave an empty catch clause, it instead preserves the fact that it was called in a boolean. It does this so that when it finally does complete you can check if it was interrupted and if so respect it and interrupt the current thread.
For more information and where the example came from look into the following article:
http://www.ibm.com/developerworks/library/j-jtp05236/
Change the domain.xml in Glassfish
<thread-pools>
<thread-pool name="http-thread-pool" idle-thread-timeout-seconds="1800" />
<thread-pool max-thread-pool-size="200" name="thread-pool-1" idle-thread-timeout-seconds="1800" />
</thread-pools>
increase the idle-thread-timeout-seconds
Related
I have a message stream, where messages comes which I need to process and then store them in database. In Java, I've written polling code which polls stream and consumes messages every 20 seconds.
This is done inside an infinite for-loop, like below:
for (;;) {
try{
//1. Logic for polling.
//2. Logic for processing the message.
//3. Logic for storing the message in database.
Thread.sleep(20000 - <time taken for above 3 steps >);
} catch(Exception E){
//4. Exception handling.
}
}
This logic runs as expected and the stream is polled, but once in a while it hits an exception or something goes wrong and polling stops.
I want to have a mechanism, that as soon as polling stopped, let's say this for loop is not running for 60 seconds, I should receive a mail or ping.
What is the best way to invoke a method if this for loop is not running for 60 seconds?
I am thinking like, each for-loop execution will ping a heartbeat, and when that heartbeat pinging not received from for-loop then a mail sending is invoked.
There are two different reasons why polling stops making progress, and each needs a different approach:
If the logic throws a Throwable other than an Exception, for instance an Error, the catch does not match, and execution will leave the for-loop, and likely reach the thread's UncaughtExceptionHandler, the default implementation of which logs the exception to System.err and terminates the thread. To prevent this, you should catch Throwable rather than Exception.
The second possibility is that some step in your logic doesn't terminate, for instance due to an infinite loop, a deadlock, waiting for I/O operations, or whatever. In this case, you'll want to take a thread dump to see where the thread is stuck. You can automate this as follows:
class Watchdog {
final Duration gracePeriod;
final Thread watchedThread;
volatile Instant lastProgress;
public Watchdog(Duration gracePeriod) {
this.gracePeriod = gracePeriod;
watchedThread = Thread.currentThread();
everythingIsFine();
var t = new Thread(this::keepWatch);
t.setDaemon(true);
t.start();
}
public void everythingIsFine() {
lastProgress = Instant.now();
}
void keepWatch() {
while (true) {
var silence = Duration.between(lastProgress, Instant.now());
if (silence.compareTo(gracePeriod) > 0) {
System.err.println("Watchdog hasn't seen any progress for " + silence.toSeconds() + " seconds. The watched thread is currently at:");
for (var element : watchedThread.getStackTrace()) {
System.err.println("\tat " + element);
}
}
try {
Thread.sleep(gracePeriod);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
while you can use as follows:
public class Test {
void step() throws Exception {
System.in.read();
}
void job() {
var snoopy = new Watchdog(Duration.ofSeconds(2));
for (;;) {
try {
step();
snoopy.everythingIsFine();
Thread.sleep(1000);
} catch (Throwable t) {
System.err.println(t);
}
}
}
public static void main(String[] args) throws Exception {
new Test().job();
}
}
once the grace period elapses, the WatchDog will print something like:
Watchdog hasn't seen any progress for 2 seconds. The watched thread is currently at:
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:293)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:255)
at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:289)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:276)
at stackoverflow.Test.step(Test.java:48)
at stackoverflow.Test.job(Test.java:55)
at stackoverflow.Test.main(Test.java:65)
I have multi-threaded Spring Boot application in which I am reading data from table in batches (the table contains around 1 million records).
I am getting into Java heap memory issues, and I am unable to find a workaround. Below is the code sample.
I call the Spring Boot REST API which then calls this code. Here I am reading from db in the main thread in batches, then passing the batches to thread pool executorService and then finally processing the result in another thread pool resultProcessor.
The Worker class implements Callable<WorkerResult>
ExecutorService executorService = Executors.newFixedThreadPool(15);
Long workerCount = 0L;
ExecutorService resultProcessor = Executors.newFixedThreadPool(10);
List<CompletableFuture<WorkerResult>> futures = new ArrayList<>();
while (workerCount < totalData) {
List<Model> dbRecords = repo.getData(workerCount,workerCount+rp,date);
workerCount += rp + 1;
try {
futures.add(CompletableFuture.supplyAsync(() -> {
try {
return new Worker(dbRecords).call(); // Here for each record third party api is called
} catch (Exception ex) {
throw new CompletionException(ex);
}
// Or return default value
}, executorService).thenApplyAsync(result -> {
service.resultReceived(result); // update the results into db
return result;
}, resultProcessor));
} catch (RejectedExecutionException e) {
logData("Can't submit anymore tasks %s ", e.getMessage());
}
}
}
Outside the while loop once I have read all data from DB, then I call the CompletableFuture.allOf method to finish any remaining tasks.
Below is the code for that:
try {
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
executorService.shutdown();
executorService.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
resultProcessor.shutdown();
resultProcessor.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
Here, if I do not add the CompletableFuture.allOf, the result is returned from this method without completing all tasks in the queues.
Instead of calling CompletableFuture.allOf, I have tried futures.foreach(CompletableFuture::join) but my issue didn't resolve that way either.
Currently, I have assigned 1GB ram to the Tomcat server, therefore I face heap space error after some 100 thousand records are processed successfully.
What can I do here to get rid of this error and improve code efficiency as well, also the solution should be in Java 8 and not the latest versions if possible.
I don't know how much data will be in real this is a test environment data.
I have a java code which I want to run. If the job does not complete within, say, 2 hours, then it should be killed automatically (basically, some kind of timed batch).
How to achieve this in Java?
If you are on Java 9 or higher, you can do the timeout batching as below:-
CompletableFuture<Integer> future = CompletableFuture.supplyAsync(this::longRunningTask)
.orTimeout(2, TimeUnit.SECONDS);
future.get(); // j.u.c.ExecutionException after waiting for 2 second
If it completes within the timeout limit, it will return the value (here an Integer object in response to future.get() method)
And, this batching is asynchronous (If you don't call get method explicitly.)
NOTE: This does not prevent the thread from completing the task, it just completes a future in main thread with a Timeout Exception so that main thread can continue. The background task/thread is still continues to finish. (look #Andreas comment)
Some Samples:-
final CompletableFuture<Void> future =
CompletableFuture.supplyAsync(this::longRunningTask)
.orTimeout(2, TimeUnit.SECONDS);
future.get(); // j.u.c.ExecutionException after waiting for 2 second
And the longRunningTask() :-
private Void longRunningTask(){
log.info("Thread name" + Thread.currentThread().getName());
try {
log.info("Going to sleep for 10 sec...");
Thread.sleep(10*1000);
log.info("Sleep Completed. Task Completed.");
} catch (InterruptedException e) {
log.info("Exception Occurred");
}
finally{
log.info("Final Cleanup in progress.");
}
log.info("Finishing the long task.");
return null;
}
If you run above code, it will give Execution Exception in main thread (where future.get() is called) but the longRunningTask will still print Sleep Completed. Task Completed. after completing 10 sec sleep.
If you carefully notice, the longRunnigThread is never interrupted (does not enter in catch block) so continues normally, but main thread gets an exception on get().
Workaround/Solution:
Use ExecutorService and submit the longRunnigTask future with this Exceutor, if timeout occurs, shutdown the executor or else, shutdown after successful get() in case of no timeout exception.
Sample:
ExecutorService myWorkers = Executors.newFixedThreadPool(1);
final CompletableFuture<Void> longTask =
CompletableFuture.supplyAsync(this::longRunningTask, myWorkers)
.orTimeout(2, TimeUnit.SECONDS);
try {
longTask.get();
} catch (InterruptedException | ExecutionException e) {
log.info("EE... Kill the executor thread/s.");
myWorkers.shutdownNow(); // this will interrupt the thread, catch the IntrExcep in thread and return the execution there
}
and the slightly modified longRunnigTask
private Void longRunningTask(){
log.info("Thread name" + Thread.currentThread().getName());
try {
log.info("Going to sleep for 10 sec...");
Thread.sleep(10*1000);
log.info("Sleep Completed. Task Completed.");
} catch (InterruptedException e) {
log.info("Exception Occurred");
return null; // this will finish the thread exce releasing all locking resources. Can be GCed then.
}
finally{
log.info("Final Cleanup in progress.");
}
log.info("Finishing the long task.");
return null;
}
With this approach, it won't complete the task inf timeout is occurred (you won't see Sleep completed. Task completed. in logs..), and would see, exception occurred in the longRunningTask thread (because of interrupt caused by myWorker.shutdown).
The thread pool executor is being executed in parallel to the main thread. The main thread is not waiting until the shutdown of the executor.
public static void main(String[] args) {
Date jobStartTime = null;
LOGGER.info("MainApp::Job started");
try {
MainApp obj = new MainApp();
// Getting the job Id of the job
String jobName=args[0]; //batch name
String fileName=args[1]; //sqoop file
LOGGER.info("MainApp::jobName: "+jobName+" fileName "+fileName);
currentJobID = obj.getMaxJobId(jobName);
LOGGER.info("MainApp:Job Id is" + currentJobID);
// Getting the start time of the job
jobStartTime = commonDB.getTime();
LOGGER.info("MainApp:Job Start time is" + jobStartTime);
JobDetails job=new JobDetails(currentJobID,jobName,fileName);
// Reading and parsing the sqoop file and executing the sqoop commands
CommandGenerator exec=new CommandGenerator();
List<TableInfo> commandList = exec.parseAndExec(job);
ThreadPoolExecutor tp = (ThreadPoolExecutor) Executors.newFixedThreadPool(10);
for (final TableInfo table : commandList) {
ParallelExecutor pe = new ParallelExecutor(table);
tp.execute(pe);
}
tp.shutdown();
while(!tp.isShutdown()){
}
job=new JobDetails(currentJobID,jobName,fileName,jobStartTime);
//put everything in one method
StatusAndMailUtils status=new StatusAndMailUtils();
status.onJobCompletion(job);
} catch (Exception e) {
// TODO Auto-generated catch block
LOGGER.info("MainApp::Exception");
e.printStackTrace();
}
}
I have used the while loop to keep the main thread waiting mean while the executor threads are in progress. But, it is not helping. Please let me know how to make the main thread wait.
while(!tp.isShutdown()){
}
After having called shutdown(), you can use awaitTermination(long timeout, TimeUnit unit) to block the calling thread until all tasks have completed execution.
As timeout you can use a value excessively big if you want to wait as long as it is needed for the tasks to complete however as it could make your thread wait forever if a task never ends, it is always better to set a reasonable timeout in order to execute some tasks if it is abnormally too long.
For example:
tp.shutdown();
tp.awaitTermination(Long.MAX_VALUE, TimeUnit.DAYS);
Of course it's not waiting. That's the whole idea of creating a threadpool, so your main thread can perform other tasks while the threadpool is executing additional tasks.
You can use the awaitTermination(long timeout, TimeUnit unit) method to have your main thread pause while the threadpool finishes its tasks.
You can also submit these Runnables and wait them to complete. It is also possible to specify a timeout to wait for the threads to execute before throwing an exception.
List<Future<ParallelExecutor>> tasks = new ArrayList<>();
ExecutorService tp = Executors.newFixedThreadPool(10);
for (final TableInfo table : commandList) {
ParallelExecutor pe = new ParallelExecutor(table);
tasks.add(tp.submit(pe));
}
for (Future<ParallelExecutor > p : tasks) {
p.get(); // with timeout p.get(10, TimeUnit.SECONDS);
}
tp.shutdown();
I faced a strange problem. I hope someone would find out the reason.
code:
public static void asyncSend(final RoomNotification notification, final int retryTimes) {
Thread thread = new Thread(new Runnable() {
#Override
public void run() {
boolean finish = false;
try {
ObjectMapper mapper = new ObjectMapper();
String messageString = mapper.writeValueAsString(notification);
logger.info("json to send to Hipchat :{}", messageString);
#1 Content content = Request.Post("https://api.hipchat.com/v2/room/<hidden>/notification?auth_token=<hidden>")
.bodyString(messageString, ContentType.APPLICATION_JSON)
.execute().returnContent();
#2 logger.info("Hipchat return:{}", content.asString());
finish = true;
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
} finally {
#3 logger.info("send Hipchat {}:\n{}", finish ? "successfully" : "unsuccessfully", notification.message);
if (finish) {
return;
}
//如果失败，且还有重试次数就重新发送
if (retryTimes > 0) {
logger.info("retry sending Hipchat, retry times remain:{}\nmessage:{}", retryTimes, notification.getMessage());
try {
Thread.sleep(10000); //重试前暂停10秒
} catch (InterruptedException e) {
e.printStackTrace();
}
asyncSend(notification, retryTimes - 1);
} else {
logger.info("no retry times remain, finally send Hipchat unsuccessfully.\nmessage:{}", notification.getMessage());
}
}
}
});
thread.start();
}
The problem is that, sometimes the line at neither #2 nor #3 would be run. is that mean the thread crashed without exception, or the line at #1 never finished running?
The code at line #1 is reference from Apache Httpcomponents.
I analysised the log just now, the output shows below:
[wwwuser#mixi-mantou ~]$ grep "json to send to Hipchat" tomcat-mixi/logs/catalina.out | wc -l
216
[wwwuser#mixi-mantou ~]$ grep "send Hipchat successfully" tomcat-mixi/logs/catalina.out | wc -l
197
[wwwuser#mixi-mantou ~]$ grep "send Hipchat unsuccessfully" tomcat-mixi/logs/catalina.out | wc -l
14
So this problem occured 5(216-197-14) times.
Wish someone could help me!
Unless you received an Error so severe that the finally block could not execute, most likely the problem is that the Request.Post never returned.
There are several ways you can find out what is happening. All of these should be done while the program is still running, but all threads doing the call should have ended.
Check for open TCP connections: under both Windows or Unix use "netstat" in a shell or command prompt. This command shows you all current TCP connections between your computer and the server. Check whether there are more open connection than there should be.
Create a full stack dump: under unix find out the PID of the Java process (using "ps"), then issue a "kill -QUIT [pid]" (replacing [pid] with the PID you found). Under Windows it can be more complicated: If you started the Java program from a command line, press CTRL+BREAK. Or you could use jstack (see https://blogs.oracle.com/pcmreddy/entry/using_jstack_on_windows). All this writes to stdout a stack trace of all running threads. Check whether for threads that are still in the Request.Post call.
You can do this also while your application is still executing. But then you have to distinguish the hanging threads from those running ok and just happen to be in the Request.Post call. It might help to check the TCP connections and/or stack traces a few times, with some seconds in between, to see what has changed. You might also give your threads names containing the start time, to easier detect long-running threads in the thread dumps.