Let's say we've got an SQL database (hsqldb) and want to run a number of queries on it which do not modify the content.
This takes a long time for some queries and I would like to run the queries in multiple threads.
So my question is: what is the best way to implement this?
I did not find any good samples to do this so I came up with the following (which I would love to get some comments on).
First, very briefly in words:
I use thread-safe collections to access the queries and to put the results in. The queries are executed in a number of worker threads. The results are processed in the main thread which checks for new results until all threads are finished.
Now the code:
Create thread-safe collections of queries and results:
ConcurrentLinkedQueue<String> queries = new ConcurrentLinkedQueue<String>()
ConcurrentLinkedQueue<ResultSet> sqlResults = new ConcurrentLinkedQueue<ResultSet>();
Create a number of threads and start them (edited):
ExecutorService executorService = Executors.newFixedThreadPool(4);
for(int i=0; i<4; i++){
executorService.execute(new QueryThread(sqlResults, queries));
}
Within the thread class QueryThread a connection is opened and queries are executed as long as there are any left:
private class QueryThread implements Runnable {
private ConcurrentLinkedQueue<ResultSet> sqlResults;
private ConcurrentLinkedQueue<String> queries;
public QueryThread(ConcurrentLinkedQueue<ResultSet> sqlResults, ConcurrentLinkedQueue<String> queries){
this.sqlResults = sqlResults;
this.queries = queries;
}
#Override
public void run(){
Connection connThr = null;
try{
try {
connThr = DriverManager.getConnection(dbModeSave, "sa", "");
connThr.setAutoCommit(false);
} catch (SQLException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
String currentQuery;
do {
currentQuery = queries.poll(); // get and remove element from remaining queries
if (currentQuery != null) { // only continue if element was found
try {
Statement stmnt = connThr.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,ResultSet.CONCUR_UPDATABLE);
try {
ResultSet resultSet = stmnt.executeQuery(currentQuery);
sqlResults.add(resultSet);
} catch (SQLException e) {
// (Do something specific)
} finally {
stmnt.close();
}
} catch (SQLException e) {
// (Do something specific)
}
}
} while (currentQuery != null);
} finally {
if (connThr != null) {
try {
connThr.close();
} catch (SQLException e) {
// Nothing we can do?
}
}
}
}
}
From the original thread I check, if the threads are all finished and therefore all queries were processed (edited).
while (!executorService.isTerminated()) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
while (!sqlResults.isEmpty()) {
ResultSet result = sqlResults.poll();
//process result and close it in the end
}
}
Java standard sulution for parallel processing is ThreadPoolExecutor. Try it.
Related
Recently I begin to learn concurrency based on Java, I run the following code on windows (jdk 11)
import java.util.*;
import java.util.concurrent.*;
class TaskWithResult implements Callable<String>{
private int id;
public TaskWithResult(int id){
this.id = id;
}
public String call(){
return "Result of TaskWithResult "+id;
}
}
public class TestCallable{
public static void main(String[] args){
ExecutorService exec = Executors.newCachedThreadPool();
ArrayList<Future<String>> results =
new ArrayList<Future<String>>();
for(int i = 0;i<10;i++){
results.add(exec.submit(new TaskWithResult(i)));
for(Future<String> fs:results){
try{
System.out.println(fs.get());
}catch(InterruptedException e){
System.out.println(e);
return;
}catch(ExecutionException e){
System.out.println(e);
}finally{
exec.shutdown();
}
}
}
}
}
The sanme Exception occurs everytime I run it：
\\output：
Result of TaskWithResult 0
Exception in thread "main" java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask#380fb434[Not completed, task = me.rexjz.a.TaskWithResult#21bcffb5] rejected from java.util.concurrent.ThreadPoolExecutor#3cda1055[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
at javaBin/me.rexjz.a.TestCallable.main(TestCallable.java:22)
The code is excerpted from Thinging in Java(4th)，I initially guess the implicit thread which drives main() execute shutdown() before all the tasks are submitted because the first task is succeessfully executed and the Exeception information indicates that pool size= 1, but it is impossible because everything in main() is sequentially executed. All the Callable objects should be submitted before shutdown.
Then I change the type of ThreadPool to Executors.newFixedThreadPool(10), the Exception stil occured and the pool size is still 1.
How did this happen?
If you look at your for loops a little more closely you'll see the problem (especially once the code is conventionally indented):
for (int i = 0; i < 10; i++) {
results.add(exec.submit(new TaskWithResult(i)));
for (Future<String> fs : results) {
try {
System.out.println(fs.get());
} catch (InterruptedException e) {
System.out.println(e);
return;
} catch (ExecutionException e) {
System.out.println(e);
} finally {
exec.shutdown();
}
}
}
Notice that the for loop which queries each Future is nested within the for loop which submits the tasks. That means you submit one task, wait for the result, shutdown the executor, and then attempt to submit another task. The following should fix your problem:
for (int i = 0; i < 10; i++) {
results.add(exec.submit(new TaskWithResult(i)));
}
executor.shutdown(); // shutdown() allows already-submitted tasks to execute
for (Future<String> fs : results) {
try {
System.out.println(fs.get());
} catch (InterruptedException e) {
e.printStackTrace();
return;
} catch (ExecutionException e) {
e.printStackTrace();
}
}
I moved the executor.shutdown() call since that only needs to happen once, after you've submitted the last task. Of course, if you're going to keep reusing the executor then you would not want to shut it down.
I also changed System.out.println(e) to e.printStackTrace(). It's typically better to print the stack trace rather than just the exception type and message (which is what Throwable#toString() returns, by default). It may not be obvious in a short program like your example, but the stack trace is extremely valuable in more complicated applications since it points you directly to where the exception was thrown. See What is a stack trace, and how can I use it to debug my application errors? for more information.
package com.springboot.testapplication;
import java.util.*;
import java.util.concurrent.*;
class TaskWithResult implements Callable<String> {
private int id;
public TaskWithResult(int id) {
this.id = id;
}
public String call() {
return "Result of TaskWithResult " + id;
}
}
public class TestCallable {
public static void main(String[] args) {
ExecutorService exec = Executors.newCachedThreadPool();
ArrayList<Future<String>> results = new ArrayList<Future<String>>();
for (int i = 0; i < 10; i++) {
results.add(exec.submit(new TaskWithResult(i)));
}
for (Future<String> fs : results) {
try {
System.out.println(fs.get());
} catch (InterruptedException e) {
System.out.println(e);
return;
} catch (ExecutionException e) {
System.out.println(e);
} finally {
exec.shutdown();
}
}
}
}
I am using executorsevice in JAVA to execute some threads, let’s say ten threads, number of threads may vary. Each thread is executing a SQL server query. I am using Future and Callable classes to submit the tasks. I am getting the results [using future.get()] once each thread is finished.
Now my requirement is that I need to know the query which is executed by each thread once its result is returned, even if the result is an empty set.
Here is my code:
List<Future<List>> list = new ArrayList<Future<List>>();
int totalThreads = allQueriesWeight.size();
ExecutorService taskExecutor = Executors.newFixedThreadPool(totalThreads);
for (String query : allQueriesWeight) {//allQueriesWeight is an arraylist containing sql server queries
SearchTask searchTask = new SearchTask(query);
Future<List> submit = taskExecutor.submit(searchTask);
list.add(submit);
}
Here is my call function:
#Override
public List<SearchResult> call() throws Exception {
java.sql.Statement statement = null;
Connection co = null;
List<SearchResult> allSearchResults = new ArrayList();
try {
//executing query and getting results
while (r1.next()) {
...
allSearchResults.add(r);//populating array
}
} catch (Exception e) {
Logger.getLogger(GenericResource.class.getName()).log(Level.SEVERE, null, e);
} finally {
if (statement != null) {
statement.close();
}
if (co != null) {
co.close();
}
}
return allSearchResults;
}
Here is how I am getting the results:
for (Future<List> future : list) {
try {
System.out.println(future.get().size());
List<SearchResult> sr = future.get();
} catch (InterruptedException ex) {
Logger.getLogger(GenericResource.class.getName()).log(Level.SEVERE, null, ex);
} catch (ExecutionException ex) {
Logger.getLogger(GenericResource.class.getName()).log(Level.SEVERE, null, ex);
}
}
In this above for loop, I need to identify the query of which the result is returned. I am a newbie and any help/suggestion is highly appreciated.
Thanks.
Alternative 1:
You have both the lists in the same order and of same size, so you can simple do as below
for (int i = 0; i < allQueriesWeight.size(); i++) {
allQueriesWeight.get(i);
futureList.get(i);
}
Alternative 2:
If all the queries are different, you can use a map as shown below but this approach will lose the order of execution.
int totalThreads = allQueriesWeight.size();
Map<String,Future<List>> map = new HashMap<>;
ExecutorService taskExecutor = Executors.newFixedThreadPool(totalThreads);
for (String query : allQueriesWeight) {//allQueriesWeight is an arraylist containing sql server queries
SearchTask searchTask = new SearchTask(query);
Future<List> submit = taskExecutor.submit(searchTask);
map.put(query ,submit );
}
And then iterate the map
for (Entry<String,Future<List>> future : map.) {
System.out.println("query is:" +map.getKey());
List<SearchResult> sr = map.getValue().get();
}
Alternative 3
If you want to keep the order, create a class with Future and query as the attributes and then put that class in list
public class ResultWithQuery {
private final Future<List<?>> future;
private final String query;
public ResultWithQuery(Future<List<?>> future, String query) {
this.future = future;
this.query = query;
}
public Future<List<?>> getFuture() {
return future;
}
public String getQuery() {
return query;
}
}
And
List<ResultWithQuery > list = new ArrayList<ResultWithQuery >();
int totalThreads = allQueriesWeight.size();
ExecutorService taskExecutor = Executors.newFixedThreadPool(totalThreads);
for (String query : allQueriesWeight) {//allQueriesWeight is an arraylist containing sql server queries
SearchTask searchTask = new SearchTask(query);
Future<List> submit = taskExecutor.submit(searchTask);
list.add(new ResultWithQuery (submit, query));
}
And iterate the list
for (ResultWithQuery resQuery: list) {
try {
resQuery.getQuery();
List<SearchResult> sr = resQuery.getFuture.get();
} catch (InterruptedException ex) {
Logger.getLogger(GenericResource.class.getName()).log(Level.SEVERE, null, ex);
} catch (ExecutionException ex) {
Logger.getLogger(GenericResource.class.getName()).log(Level.SEVERE, null, ex);
}
}
Guys I'm facing a similar situation
like
This junit case on another thread
though i don't have a junit case. I tried everything that i know of.. including suggestion on that link page, keeping a countdown and thread sleep but the results don't change. if i run through debug and give it some time it shows me all the results from all the thread but if i run it normally it invariably gives me less results.
My code is as belows
`
AtomicInteger atomicInteger = new AtomicInteger(employeeids.size());
CountDownLatch latch = new CountDownLatch(employeeids.size());
Iterable<List<String>> batchList = createBatches(employeeids, batchSize);
Set<Future<List<GradeSearchDTO>>> set = new HashSet<Future<List<GradeSearchDTO>>>();
for(List<String> employeeidsList: batchList) {
Callable<List<GradeSearchDTO>> callable = new ScheduleCallable( employeetype, employeedetails, employeeidsList, dept, seeker, atomicInteger,latch );
Future<List<GradeSearchDTO>> future = pool.submit(callable);
set.add(future);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
try {
latch.await(getTimeOutInMillis(), TimeUnit.MILLISECONDS);
} catch (InterruptedException e1) {
// TODO Auto-generated catch block
throw new EmployeeException("Building of Schedules didn't finish in time of ["+getTimeOutInMillis()+"] millis. ");
}
long timeLeft = getTimeOutInMillis();
boolean check=true;
while (check){
logger.debug("Waiting for building asset. countdown value is[" + timeLeft+"]");
try {
Thread.sleep(TIME_TO_PAUSE);
timeLeft = timeLeft - TIME_TO_PAUSE;
if(timeLeft == 0 || timeLeft < 0){
throw new EmployeeException("Building of Schedules didn't finish in time of ["+getTimeOutInMillis()+"] millis. ");
}
for (Future<List<GradeSearchDTO>> future : set) {
if(!future.isDone()){
check=true;
break;
}
else{check=false;}
}
} catch (InterruptedException e) {
logger.error("Error waiting for asset to build to bulid");
}
}
for (Future<List<GradeSearchDTO>> future : set) {
try {
EmployeeScheduleList.addAll(future.get());
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ExecutionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public static class ScheduleCallable implements Callable
{
private String employeetype;
private List<Employee> employeedetails;
private List<String> employeeidsList;
private String dept;
private EmployeeSeekerHelper seeker;
private AtomicInteger atomicInteger;
private CountDownLatch latch;
public ScheduleCallable(String employeetype,List<Employee> employeedetails,
list<String> employeeidsList, String dept,EmployeeSeekerHelper seeker,AtomicInteger
atomicInteger,CountDownLatch latch )
{
this.employeetype = employeetype;
this.employeedetails = employeedetails;
this.employeeidsList = employeeidsList;
this.dept = dept;
this.seeker = seeker;
this.atomicInteger=atomicInteger;
this.latch=latch;
}
public List<GradeSearchDTO> call()
{
List<GradeSearchDTO> EmployeeScheduleList = new ArrayList<GradeSearchDTO>(0) ;
int counter=1;
for(String scheduleId : employeeidsList)
{
latch.countDown();
EmployeeScheduleList.addAll(searchEmployeeRulesForSchedule(employeetype,employeedetails,scheduleId,dept,seeker,latch));
System.out.println("Thread COUNTER "+counter);
atomicInteger.decrementAndGet();
counter++;
// latch.countDown();
}
return EmployeeScheduleList;
}
}
`
So the above code is perfectly fine... nothing wrong at all. The problem that i faced with the random results was because the method searchEmployeeRulesForSchedule(employeetype,employeedetails,scheduleId,dept,seeker,latch)
which does the business logic under the call() was internally calling a rules engine which was not returning proper results because of the usage of same instance of a class instead of a new instance for each thread.
I try to make multithreading program on Java print results that returned finished threads.
The thing is, when i run this code it simply gets stuck on second value that was in queue:
System.out.println("[!] Creaing pool");
int max_threads = 50;
ExecutorService threadPool = Executors.newFixedThreadPool(max_threads);
CompletionService<String> taskCompletionService =
new ExecutorCompletionService<String>(threadPool);
String url;
while(our_file.hasNext()){
url = our_file.next();
if (url.length()>0){
futures.add(
taskCompletionService.submit(
new GoGo(url)
)
);
}
int total_tasks = futures.size();
while(total_tasks>0){
for (int i=0; i<futures.size(); i++){
try{
Future result = taskCompletionService.poll();
if(result!=null && result.isDone()){
System.out.println(result.get());
total_tasks--;
}
}
catch (InterruptedException e) {
// Something went wrong with a task submitted
System.out.println("Error Interrupted exception");
e.printStackTrace();
} catch (ExecutionException e) {
// Something went wrong with the result
e.printStackTrace();
System.out.println("Error get() threw exception");
}
}
}
}
threadPool.shutdown();
try {
threadPool.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
}
catch (InterruptedException e ) {
}
...
class GoGo implements Callable{
private String url;
public GoGo(String received_url){
this.url = received_url;
}
public String call(){
String URL = this.url;
return url;
}
}
output is like this:
[!] Creaing pool
http://www.www1.com/
http://www.www2.ch/
and at this point program just stucks.
I tried to move loop that iterates futures array out of main loop that submits threads, and it worked fine, but in case if i will go through very large file i need real-time output.
Please help me figure out where is the bottleneck, I was unable to find any suitable piece of code that using non-blocking poll() method from CompletionService.
Thanks for any answer or reference.
The problem is you are trying to do two things simultaneously (submit work, and read work results) in one thread.
That doesn't make sense - for simultaneous tasks, you need multiple threads.
So create another thread to read the results. Or another thread to submit the tasks. It doesn't matter which way you do it; either way, you end up with 2 threads instead of one.
Thanks to Robin Green for advice, putting future harvester class to separate thread solved the problem! So, i just start the endless loop thread that pops argument with poll() check if the popped future object indicates that thread isDone() and write output. And after shutting down the fixedThreadPool, output writer class is stopped. Here's the code (except the GoGo class):
public class headScanner {
public static List<Future<String>> gloabal_futures = new ArrayList<Future<String>>();
public static void main(String args[]){
Scanner our_file = null;
ArrayList<String> our_urls = new ArrayList<String>();
List<Future<String>> futures = new ArrayList<Future<String>>();
ArrayList<String> urls_buffer = new ArrayList<String>();
try {
our_file = new Scanner (new File ("list.txt"));
}
catch(IOException e){
System.out.println("[-] Cant open the file!");
System.exit(0);
}
System.out.println("[!] Creaing pool");
int max_threads = 50;
ExecutorService threadPool = Executors.newFixedThreadPool(max_threads);
CompletionService<String> taskCompletionService =
new ExecutorCompletionService<String>(threadPool);
String url;
Thread result_thread = new Thread(new ResultHarvester(futures.size(), taskCompletionService));
result_thread.start();
while(our_file.hasNext()){
url = our_file.next();
if (url.length()>0){
futures.add(
taskCompletionService.submit(
new GoGo(url)
)
);
}
}
threadPool.shutdown();
try {
threadPool.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
}
catch (InterruptedException e ) {
}
result_thread.stop();
}
}
class ResultHarvester implements Runnable {
private int size;
private CompletionService<String> all_service;
public ResultHarvester (int size, CompletionService<String> service){
this.size = size;
this.all_service = service;
}
public void run(){
int future_size = 1;
CompletionService<String> this_service = this.all_service;
while(true){
Future result = this_service.poll();
try {
if(result!=null && result.isDone()){
String output = result.get().toString();
if(output.length()>1){
System.out.println(output);
}
}
}
catch (InterruptedException e) {
// Something went wrong with a task submitted
System.out.println("Error Interrupted exception");
e.printStackTrace();
} catch (ExecutionException e) {
// Something went wrong with the result
e.printStackTrace();
System.out.println("Error get() threw exception");
}
}
}
}
How to handle the thread pool where one is polling while the other should update new incoming data after processing.
The program execution beings in a controller class which has a main method and thread pool:
The main class Controller
public static void main(String[] args) throws InterruptedException {
RunnableController controller = new RunnableController();
Accumulator acque = new Accumulator();
controller.initializeDb();
controller.initialiseThreads(acque);
controller.initialUpdate(acque);
}
The Run method for Polling class:
public void run() {
int seqId = 0;
List<KpiMessage> list = null;
while(true) {
try{
list = fullPoll(seqId);
if (!list.isEmpty()) {
accumulator.manageIngoing(list);
}
} catch (Exception e){
e.printStackTrace();
}
}
}
public List<KpiMessage> fullPoll(int lastSeq) throws Exception {
Statement st = dbConnection.createStatement();
System.out.println("Polling");
ResultSet rs = st.executeQuery("Select * from msg_new_to_bde where ACTION = 804 and SEQ >" +
lastSeq + "order by SEQ DESC");
return pojoCol;
}
Run method for processing:
public void run() {
try {
generate(accumulator.outgoingQueue);
accumulator.manageOutgoing(accumulator.outgoingQueue, dbConnection);
} catch (Exception e) {
e.printStackTrace();
}
}
}
Method for updating into Database
public void updateDb(Collection<KpiMessage> updatedQueue, Connection dbConnection) throws
SQLException{
for(KpiMessage pojoClass : updatedQueue){
Statement stmtupd = dbConnection.createStatement();
System.out.println("Updating");
String query = "UPDATE msg_new_to_bde SET KEYINFO1= 'Processed', KEYINFO2 = 'Updated'
WHERE ACTION = 804";
stmtupd.executeUpdate(query);**My Execution stops here**
Finally an accumulator class for maintaing all these queues:
public boolean isUsed = false;
public synchronized void manageIngoing(List<KpiMessage> list){
if(this.isUsed){
try {
wait();
System.out.println("first wait");
} catch (Exception e1) {
e1.printStackTrace();
}
}
System.out.println("recived pass after update");
this.getIncomingQueue().addAll(list);
//incoming queue copied to outgoing queue
this.setOutgoingQueue(this.getIncomingQueue());
System.out.println("waiting");
System.out.println("new incoming message");
this.isUsed = false;
notifyAll();
}
/**
* Method which handles synchronization using wait and notify for outgoing messages after
polling
* #param outgoingQueue
* #param dbConnection
*/
public synchronized void manageOutgoing(Collection<KpiMessage> outgoingQueue, Connection
dbConnection){
if(!this.isUsed)
{
try {
System.out.println("second wait");
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
this.isUsed = true;
DBhandler dbhandler = new DBhandler();
try {
dbhandler.updateDb(getOutgoingQueue(), dbConnection);
} catch (SQLException e) {
e.printStackTrace();
}
notifyAll();
}
}
My task and Question is :
1.The controller should handle both the threads Poller & processor and accumulator handles the incoming and outgoing queues, finally fed into to updated queue for updating DB after processing
2.My class here just does polling once, is not able to update ,execution stops at
3.Is my wait(), notifyALL() handle correct here.
How to achieve repeated polling and updation here?
Chances are, in this complex setting with five different questions, there will be no complete answer for everything. While waiting for those, you should read up on what java.util.concurrent has to offer, especially the concurrent collections with support for blocking reads and writes. Use wait() and notify() only if the JDK classes are not enough for you.