I have a file upload functionality, in which user can upload mutiple file at a same time, for good result I used Thread processing like this
Thread.start{
// do file processing 'cause it is a long running process
}
Now the problem is, for each file upload system will create a new Thread so that will lead to system thrashing and someother issues, so now I am looking for a solution where I can create a Queue to store all the received files and create minimum number(say 5 nos) of Thread at a time and process it and again create a set of Thread and process.
So for I am looking into GPars, Java Thread and Queue and soon but no idea which is efficient way and what is the existing good solution
You are looking for a thread pool or - in Java terms - for an Executor:
Executor executor = anExecutor();
executor.execute(aRunnable());
The method anExecutor should return a new Executor instance:
Executor anExecutor() {
return Executors.newFixedThreadPool(42); // just an example ...
}
Related
Hi Guys can you help me about the error that I've encountered on my Java program. I do the Callable implementation with a loop, Basically I need to send a request to another webservice and I will be collecting the response ID. Based on my testing, my aim to process it asynchronously is working using the below implementation. One time I try to run again my program then I got this kind of error "Error 500: java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11". What I did is I just restarted the server then it becomes ok.
So I want to know if there is something wrong with my implementation? like is there any additional line of code that I need to add or remove just to prevent again to experience those kind of error? I only experience it once. Hope you could help me
//pccsSurvList is a list of details coming from the database.
ExecutorService executorService = null;
List<Callable<SyncFlagEntity>> lst = new ArrayList<Callable<SyncFlagEntity>>();
if(pccsSurvList != null && pccsSurvList.size() > 0){
executorService = Executors.newFixedThreadPool(pccsSurvList.size());
for(PCCSSurveyInfoEntity user: pccsSurvList){
NotifyEmailTransactionImpl emailTransact = new NotifyEmailTransactionImpl(user);
lst.add(emailTransact);
}
}
// returns a list of Futures holding their status and results when all complete
List<Future<SyncFlagEntity>> tasks = new ArrayList<Future<SyncFlagEntity>>();
tasks = executorService.invokeAll(lst);
executorService.shutdown();
java.lang.OutOfMemoryError: Failed to create a thread
This OutOfMemoryError indicates that Java is out of memory.
Most likely the problem lies on this line ...
executorService = Executors.newFixedThreadPool(pccsSurvList.size());
You are getting a lot of rows and there's no enough RAM for the JVM to create a thread for each one. Try logging the number of rows you get and see what's happening.
I think you are running too many processes at once. You should try to set a limit and use a thread pool.
Maybe you can do something like:
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(pccsSurvList.size());
Then inside for loop you can do:
executor.submit(() -> emailTransact);
There is more here: https://www.baeldung.com/thread-pool-java-and-guava
Also take a look at reactive programming. Can be of more help in your case: https://www.vogella.com/tutorials/RxJava/article.html
I've to do a massive upload to a certain server - my program do each upload in about 1 second - but i will upload around 10 thousand documents each time.
To do this, i thought in use parallelism to send the documents in a good time. (My server already accept a lot of requests simultaneously).
So, i created a Task (implementing a Callable) that upload my document.
And put it in a ExecutorService.
ExecutorUpload.java
public void execute(){
ExecutorService executor = Executors.newWorkStealingPool();
//this code create the InputStream objects from
//real Files from disk using java.io.InputStream and java.io.File
List<CallbackCreateDocumentTask> tasks = createTasks(...);
//this list holds the Future objects to try to terminate the executorService
List<Future<DocumentDTO>> futures = new CopyOnWriteArrayList<>();
//iterate the list and call the Tasks
tasks.stream().forEach((task) -> futures.add(executor.submit(task)));
executor.shutdown();
//here i was trying to stop the executor,
//but this makes me lose de asynchronous upload because
//the application stops to wait this unique executoService to terminate,
//and i've more than one executorService doing upload
//executor.awaitTermination(10, TimeUnit.MINUTES);
}
App.java
public static void main (String[] args){
new ExecutorUpload().execute();
}
This code is allright. All my documents were uploaded by the Tasks, in all instances of ExecutorService that i created.
But, my application never ends. It's like it stay waiting for more unfinished executorServices;
Does anybody knows why my application never ends ?
I suspect a few things :
Java 8 never close the main method
My executorService's run in a never-ending thread, or something like
that
My InputStream's doesnt are been closed, making the main method
wait for it
Something with the File, not with the InputStream, related with the Files not been closed too...
My web application (on Tomcat) provides “on the fly” logic execution functionality.
The problem is the “on the fly” logic can contains infinite loop , or something long duration.
My solution is timeout: to run the “on the fly” logic in a new daemon thread and go back main thread in timeout, p-code as below:
ExecutorService executor = Executors.newSingleThreadExecutor(new ThreadFactory(){
#Override
public Thread newThread(Runnable r) {
Thread t = new Thread(r);
result.setDaemon(true);
return t;
}});
Future<Object> future = executor.submit(callable/* 'on the fly' callable: it can be infinite loop, the callable is out of my control */);
//Back to main thread
return future.get(timeout, TimeUnit.MILLISECONDS);
However, the daemon thread is still running, although future.get() returns in timeout. The daemon is terminated until Tomcat stops.
Now my latest solution is create a new Java process Runtime.getRuntime().exec("java MyProgram"). MyProgram contains future.get() shown before. The daemon is terminated once main thread quits as expected.
I am here to ask more elegant solution to terminate thread in web application. The new Java process is heavy and out control of web application.
Thanks!
threading in a managed environment is generally a bad idea. why not use some sort of abstraction like JMS to start a background handler every time someone sends a request ? that way you can control the number of active threads (jms pool size)
I am working on a project in which I will be having different Bundles. Let's take an example, Suppose I have 5 Bundles and each of those bundles will have a method name process.
Now currently, I am calling the process method of all those 5 bundles in parallel using multithread code below.
But somehow, everytime when I am running the below multithread code, it always give me out of memory exception. But if I am running it sequentially meaning, calling process method one by one, then it don't give me any Out Of memory exception.
Below is the code-
public void callBundles(final Map<String, Object> eventData) {
// Three threads: one thread for the database writer, two threads for the plugin processors
final ExecutorService executor = Executors.newFixedThreadPool(3);
final Map<String, String> outputs = (Map<String, String>)eventData.get(Constants.EVENT_HOLDER);
for (final BundleRegistration.BundlesHolderEntry entry : BundleRegistration.getInstance()) {
executor.submit(new Runnable () {
public void run() {
try {
final Map<String, String> response = entry.getPlugin().process(outputs);
//process the response and update database.
System.out.println(response);
} catch (Exception e) {
e.printStackTrace();
}
}
});
}
}
Below is the exception, I am getting whenever I am running above Multithreaded code.
JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
JVMDUMP032I JVM requested Heap dump using 'S:\GitViews\Stream\goldseye\heapdump.20130904.175256.12608.0001.phd' in response to an event
JVMDUMP010I Heap dump written to S:\GitViews\Stream\goldseye\heapdump.20130904.175256.12608.0001.phd
JVMDUMP032I JVM requested Java dump using 'S:\GitViews\Stream\goldseye\javacore.20130904.175256.12608.0002.txt' in response to an event
UTE430: can't allocate buffer
UTE437: Unable to load formatStrings for j9mm
JVMDUMP010I Java dump written to S:\GitViews\Stream\goldseye\javacore.20130904.175256.12608.0002.txt
JVMDUMP032I JVM requested Snap dump using 'S:\GitViews\Stream\goldseye\Snap.20130904.175256.12608.0003.trc' in response to an event
UTE001: Error starting trace thread for "Snap Dump Thread": -1
JVMDUMP010I Snap dump written to S:\GitViews\Stream\goldseye\Snap.20130904.175256.12608.0003.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
ERROR: Bundle BullseyeModellingFramework [1] EventDispatcher: Error during dispatch. (java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 12)
java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 12
JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
JVMDUMP032I JVM requested Heap dump using 'S:\GitViews\Stream\goldseye\heapdump.20130904.175302.12608.0004.phd' in response to an event
JVMDUMP010I Heap dump written to S:\GitViews\Stream\goldseye\heapdump.20130904.175302.12608.0004.phd
JVMDUMP032I JVM requested Java dump using 'S:\GitViews\Stream\goldseye\javacore.20130904.175302.12608.0005.txt' in response to an event
I am using JDK1.6.0_26 as the installed JRE's in my eclipse.
Each call of callBundles() will create a new threadpool by creating an own executor. Each thread has its own stack space! So if you say you start the JVM, the first call will create three threads with a sum of 3M heap (1024k is the default stack size of a 64-bit JVM), the next call another 3M etc. 1000 calls/s will need 3GB/s!
The second problem is you never shutdown() the created executor services, so the thread will live on until the garbage collector removes the executor (finalize() also call shutdown()). But the GC will never clear the stack memory, so if the stack memory is the problem and the heap is not full, the GC will never help!
You need to use one ExecutorService, lets say with 10 to 30 threads or a custom ThreadPoolExecutor with 3-30 cached threads and a LinkedBlockingQueue. Call shutdown() on the service before your application stops if possible.
Check the physical RAM, load and response time of your application to tune the parameters heap size, maximum threads and keep alive time of the threads in the pool. Have a look on other locking parts of the code (size of a database connection pool, ...) and the number of CPUs/cores of your server. An staring point for a thread pool size may be number of CPUs/core plus 1., with much I/O wait more become useful.
The main problem is that you aren't really using the thread pooling properly. If all of your "process" threads are of equal priority, there's no good reason not to make one large thread pool and submit all of your Runnable tasks to that. Note - "large" in this case is determined via experimentation and profiling: adjust it until your performance in terms of speed and memory is what you expect.
Here is an example of what I'm describing:
// Using 10000 purely as a concrete example - you should define the correct number
public static final LARGE_NUMBER_OF_THREADS = 10000;
// Elsewhere in code, you defined a static thread pool
public static final ExecutorService EXECUTOR =
Executors.newFixedThreadPool(LARGE_NUMBER_OF_THREADS);
public void callBundles(final Map<String, Object> eventData) {
final Map<String, String> outputs =
(Map<String, String>)eventData.get(Constants.EVENT_HOLDER);
for (final BundleRegistration.BundlesHolderEntry entry : BundleRegistration.getInstance()) {
// "Three threads: one thread for the database writer,
// two threads for the plugin processors"
// so you'll need to repeat this future = E.submit() pattern two more times
Future<?> processFuture = EXECUTOR.submit(new Runnable() {
public void run() {
final Map<String, String> response =
entry.getPlugin().process(outputs);
//process the response and update database.
System.out.println(response);
}
}
// Note, I'm catching the exception out here instead of inside the task
// This also allows me to force order on the three component threads
try {
processFuture.get();
} catch (Exception e) {
System.err.println("Should really do something more useful");
e.printStackTrace();
}
// If you wanted to ensure that the three component tasks run in order,
// you could future = executor.submit(); future.get();
// for each one of them
}
For completeness, you could also use a cached thread pool to avoid repeated creation of short-lived Threads. However, if you're already worried about memory consumption, a fixed pool might be better.
When you get to Java 7, you might find that Fork-Join is a better pattern than a series of Futures. Whatever fits your needs best, though.
we use JDK 7 watchservice to watch directory which can have xml or csv files. These files are put in threadpool and later on processed and pushed into database.This application runs for ever watching the directory and keeps processing files as and when available. XML file are small and does not take time, however each csv file can contain more than 80 thousand records so processing takes time to put in database. Java application give us outofmemory error when there are 15 csv files getting processed from threadpool. Is there any way where when csv files comes into threadpool, it can be serially processed i.e only one at a time.
Java application give us outofmemory error when there are 15 csv files getting processed from threadpool. Is there any way where when csv files comes into threadpool, it can be serially processed i.e only one at a time.
If I'm understanding, you want to stop adding to the pool if you are over some threshold. There is an easy way to do that which is by using a blocking-queue and the rejected execution handler.
See the following answer:
Process Large File for HTTP Calls in Java
To summarize it, you do something like the following:
// only allow 100 jobs to queue
final BlockingQueue<Runnable> queue = new ArrayBlockingQueue<Runnable>(100);
ThreadPoolExecutor threadPool =
new ThreadPoolExecutor(nThreads, nThreads, 0L, TimeUnit.MILLISECONDS, queue);
// we need our RejectedExecutionHandler to block if the queue is full
threadPool.setRejectedExecutionHandler(new RejectedExecutionHandler() {
#Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
try {
// this will block the producer until there's room in the queue
executor.getQueue().put(r);
} catch (InterruptedException e) {
throw new RejectedExecutionException(
"Unexpected InterruptedException", e);
}
}
});
This will mean that it will block adding to the queue and should not exhaust memory.
I would take a different route to solve your problem, I guess you have everything right except when you start reading too much data into memory.
Not sure how are you reading csv files, would suggest to use a LineReader and read e.g. 500 lines process them and then read next 500 lines, all large files should be handled this way only, because no matter how much you increase your memory arguments, you will hit out of memory as soon as you will have a bigger file to process, so use an implementation that can handle records in batches. This would require some extra coding effort but will never fail no matter how big file you have to process.
Cheers !!
You can try:
Increase the memory of JVM using the -Xmx JVM option
Use a different executor to reduce the number of processed files at a time. A drastical solution is to use a SingleThreadExecutor:
public class FileProcessor implements Runnable {
public FileProcessor(String name) { }
public void run() {
// process file
}
}
// ...
ExecutorService executor = Executors.newSingleThreadExecutor();
// ...
public void onNewFile(String fileName) {
executor.submit(new FileProcessor(fileName));
}