I've to do a massive upload to a certain server - my program do each upload in about 1 second - but i will upload around 10 thousand documents each time.
To do this, i thought in use parallelism to send the documents in a good time. (My server already accept a lot of requests simultaneously).
So, i created a Task (implementing a Callable) that upload my document.
And put it in a ExecutorService.
ExecutorUpload.java
public void execute(){
ExecutorService executor = Executors.newWorkStealingPool();
//this code create the InputStream objects from
//real Files from disk using java.io.InputStream and java.io.File
List<CallbackCreateDocumentTask> tasks = createTasks(...);
//this list holds the Future objects to try to terminate the executorService
List<Future<DocumentDTO>> futures = new CopyOnWriteArrayList<>();
//iterate the list and call the Tasks
tasks.stream().forEach((task) -> futures.add(executor.submit(task)));
executor.shutdown();
//here i was trying to stop the executor,
//but this makes me lose de asynchronous upload because
//the application stops to wait this unique executoService to terminate,
//and i've more than one executorService doing upload
//executor.awaitTermination(10, TimeUnit.MINUTES);
}
App.java
public static void main (String[] args){
new ExecutorUpload().execute();
}
This code is allright. All my documents were uploaded by the Tasks, in all instances of ExecutorService that i created.
But, my application never ends. It's like it stay waiting for more unfinished executorServices;
Does anybody knows why my application never ends ?
I suspect a few things :
Java 8 never close the main method
My executorService's run in a never-ending thread, or something like
that
My InputStream's doesnt are been closed, making the main method
wait for it
Something with the File, not with the InputStream, related with the Files not been closed too...
Related
I have a JSP application in which a webpage calls five methods one by one (all of them fetch data from different sources) and display charts based on data.
To load the webpage fastly, I planned to call all the five methods in parallel with the help of FixedThreadPool Executor.
Should I shut down my executor once I get the result from all five methods? Shutting down the executor is a bad idea according to me, since if someone opens the webpage a second time it will require the executor to initialize again in order to call the five methods parallelly.
However, I'm not sure about the consequences of leaving the executor open so not sure how to proceed.
Leaving it open is the normal way to use a thread pool. That's the whole point of thread pools: It's to prevent your application from having to create and then destroy however many new threads every time it needs to load a page. Instead, it can just use the same threads again and again.
In chapter 7 of "Java Concurrency in Practice" there is an example just like this, where a so called one-shot execution service is proposed:
If a method needs to process a batch of tasks and does not return until all the
tasks are finished, it can simplify service lifecycle management by using a private
Executor whose lifetime is bounded by that method.
Its code example:
boolean checkMail(Set<String> hosts, long timeout, TimeUnit unit)
throws InterruptedException {
ExecutorService exec = Executors.newCachedThreadPool();
final AtomicBoolean hasNewMail = new AtomicBoolean(false);
try {
for (final String host : hosts)
exec.execute(new Runnable() {
public void run() {
if (checkMail(host))
hasNewMail.set(true);
}
});
} finally {
exec.shutdown();
exec.awaitTermination(timeout, unit);
}
return hasNewMail.get();
}
I'd suggest simplifying your code using this approach.
I am creating a JSP page which has one Upload button(to upload XLS and later update this data in DB). As soon as user will click on Upload button it will read the XLS file and prepare a list of objects that will be passed to a method (m1(List a)) to execute SQL queries.
Now the problem is that i have around 100 sql queries in this method(m1(List a)), that takes around 30 min to get completed.
So I don't want user to wait until this method completes DB process.
Is there any way i can call my method to update DB and without waiting for the response of this DB operation, i can respond to user that file has been uploaded and DB process has been initiated that will be completed after some time.
Hand off the work to be done outside of the request-response cycle to an ExecutorService.
private void doDatabaseWork(Input input) {
BackgroundWorkTask task = new BackgroundWorkTask(input);
executorService.submit(task);
// since the work is now handed off to a separate pool of
// threads, the current HTTP-handling thread will continue
// here and return a response to the user
}
public class BackgroundWorkTask implements Runnable {
public void run() {
// put all of your database querying operations in here
}
}
Make sure that, since this is a webapp, you have a way to shut down the ExecutorService when the webapp is stopped - which will also give the ExecutorService a chance to finish any in-progress work before allowing the container to stop.
we use JDK 7 watchservice to watch directory which can have xml or csv files. These files are put in threadpool and later on processed and pushed into database.This application runs for ever watching the directory and keeps processing files as and when available. XML file are small and does not take time, however each csv file can contain more than 80 thousand records so processing takes time to put in database. Java application give us outofmemory error when there are 15 csv files getting processed from threadpool. Is there any way where when csv files comes into threadpool, it can be serially processed i.e only one at a time.
Java application give us outofmemory error when there are 15 csv files getting processed from threadpool. Is there any way where when csv files comes into threadpool, it can be serially processed i.e only one at a time.
If I'm understanding, you want to stop adding to the pool if you are over some threshold. There is an easy way to do that which is by using a blocking-queue and the rejected execution handler.
See the following answer:
Process Large File for HTTP Calls in Java
To summarize it, you do something like the following:
// only allow 100 jobs to queue
final BlockingQueue<Runnable> queue = new ArrayBlockingQueue<Runnable>(100);
ThreadPoolExecutor threadPool =
new ThreadPoolExecutor(nThreads, nThreads, 0L, TimeUnit.MILLISECONDS, queue);
// we need our RejectedExecutionHandler to block if the queue is full
threadPool.setRejectedExecutionHandler(new RejectedExecutionHandler() {
#Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
try {
// this will block the producer until there's room in the queue
executor.getQueue().put(r);
} catch (InterruptedException e) {
throw new RejectedExecutionException(
"Unexpected InterruptedException", e);
}
}
});
This will mean that it will block adding to the queue and should not exhaust memory.
I would take a different route to solve your problem, I guess you have everything right except when you start reading too much data into memory.
Not sure how are you reading csv files, would suggest to use a LineReader and read e.g. 500 lines process them and then read next 500 lines, all large files should be handled this way only, because no matter how much you increase your memory arguments, you will hit out of memory as soon as you will have a bigger file to process, so use an implementation that can handle records in batches. This would require some extra coding effort but will never fail no matter how big file you have to process.
Cheers !!
You can try:
Increase the memory of JVM using the -Xmx JVM option
Use a different executor to reduce the number of processed files at a time. A drastical solution is to use a SingleThreadExecutor:
public class FileProcessor implements Runnable {
public FileProcessor(String name) { }
public void run() {
// process file
}
}
// ...
ExecutorService executor = Executors.newSingleThreadExecutor();
// ...
public void onNewFile(String fileName) {
executor.submit(new FileProcessor(fileName));
}
How can I create a timeout for each command that is running in parallel using java.util.concurrent.ExecutorService?
My code is something like this:
For example in the code below I need obj1 run for maximum 1 min, and obj2 for 2 mins and others 5 mins.
ExecutorService exService;
exService = Executors.newCachedThreadPool();
exService.execute(obj1);
exService.execute(obj2);
exService.execute(obj3);
exService.execute(obj4);
exService.shutdown();
boolean finshed = exService.awaitTermination(5, TimeUnit.MINUTES);
if (finshed) {
//Doing something
}
EDIT:
Unfortunately the class of obj1 - obj4 is scraping some web pages using WebHarvest that uses jakarta HttpClient for reading web pages and HttpClient (And neither WebHarvest itself) doesn't have any feature for timeout on entire page reading and/or scraping job.
This is my time consuming task and I thought about killing ExecutorService thread after a timeout to handle this problem.
In general, there is no reliable way to make a separate thread quit. In particular, there is no reliable way to interrupt and stop your task after a timeout from outside that task. What you need to do is make the tasks themselves responsible for stopping after their time runs out. Depending on what they do, you might be able to abstract this behaviour into a superclass something like:
public abstract class TimeoutRunnable implements Runnable {
private final long timeLimitMillis;
private long startTimeMillis;
public TimeoutRunnable(long timeLimitMillis) {
this.timeLimitMillis = timeLimitMillis;
}
public final void run() {
startTimeMillis = System.currentTimeMillis();
while (System.currentTimeMillis() - startTimeMillis < timeLimitMillis) {
runIteration();
}
}
protected abstract void runIteration();
}
Then in your subclass override, runIteration() and perform a single "step" of the task.
The only reasonably reliable way to kill a task is to run it in a separate process and kill that process if it times out. Using any other approach with a library which does not support timeouts is likely to be error prone at best.
from my point of view I think that such stuff requires some more robust foundations than plain Java standard classes , that 's why I would suggest to use any scheduler infrastructure (Quartz or any other project) which may gives you handles (job identifiers) to kill your time consuming tasks .
You may have something like this :
Main Thread launches the Quartz Scheduler , receive subscriptions from different jobs
saying : Job1,Job 2 and TimeCheckerJob
TimeCheckerJob would be a forever job ,notified to any new job and would check for living time for each any new job... In this job you would have to deal with start time of each job, beware of the OS clocks and don't try to setup too hard constraints (nanoseconds is pure fiction).
HTH
My 2 cents
Jerome
I am using the JavaMail API , and there is a method in the Folder class called "search" that sometimes take too long to execute. What i want is to execute this method over a maximum period of time( say for example 15 seconds in maximum) , that way i am sure that this method will not run up more than 15 seconds.
Pseudo Code
messages = maximumMethod(Folder.search(),15);
Do I have to create a thread just to execute this method and in the main thread use the wait method ?
The best way to do this is create a single threaded executor which you can submit callables with. The return value is a Future<?> which you can get the results from. You can also say wait this long to get the results. Here is sample code:
ExecutorService service = Executors.newSingleThreadExecutor();
Future<Message[]> future = service.submit(new Callable<Message[]>() {
#Override
public Message[] call() throws Exception {
return Folder.search(/*...*/);
}
});
try {
Message[] messages = future.get(15, TimeUnit.SECONDS);
}
catch(TimeoutException e) {
// timeout
}
You could
mark current time
launch a thread that will search in the folder
while you get the result (still in thread) don't do anything if current time exceeds time obtained in 1 plus 15 seconds. You won't be able to stop the connection if it is pending but you could just disgard a late result.
Also, if you have access to the socket used to search the folder, you could set its timeout but I fear it's gonna be fully encapsulated by javamail.
Regards,
Stéphane
This SO question shows how to send a timeout exception to the client code: How do I call some blocking method with a timeout in Java?
You might be able to interrupt the actual search using Thread.interrupt(), but that depends on the method's implementation. You may end up completing the action only to discard the results.