I am using executorsevice in JAVA to execute some threads, let’s say ten threads, number of threads may vary. Each thread is executing a SQL server query. I am using Future and Callable classes to submit the tasks. I am getting the results [using future.get()] once each thread is finished.
Now my requirement is that I need to know the query which is executed by each thread once its result is returned, even if the result is an empty set.
Here is my code:
List<Future<List>> list = new ArrayList<Future<List>>();
int totalThreads = allQueriesWeight.size();
ExecutorService taskExecutor = Executors.newFixedThreadPool(totalThreads);
for (String query : allQueriesWeight) {//allQueriesWeight is an arraylist containing sql server queries
SearchTask searchTask = new SearchTask(query);
Future<List> submit = taskExecutor.submit(searchTask);
list.add(submit);
}
Here is my call function:
#Override
public List<SearchResult> call() throws Exception {
java.sql.Statement statement = null;
Connection co = null;
List<SearchResult> allSearchResults = new ArrayList();
try {
//executing query and getting results
while (r1.next()) {
...
allSearchResults.add(r);//populating array
}
} catch (Exception e) {
Logger.getLogger(GenericResource.class.getName()).log(Level.SEVERE, null, e);
} finally {
if (statement != null) {
statement.close();
}
if (co != null) {
co.close();
}
}
return allSearchResults;
}
Here is how I am getting the results:
for (Future<List> future : list) {
try {
System.out.println(future.get().size());
List<SearchResult> sr = future.get();
} catch (InterruptedException ex) {
Logger.getLogger(GenericResource.class.getName()).log(Level.SEVERE, null, ex);
} catch (ExecutionException ex) {
Logger.getLogger(GenericResource.class.getName()).log(Level.SEVERE, null, ex);
}
}
In this above for loop, I need to identify the query of which the result is returned. I am a newbie and any help/suggestion is highly appreciated.
Thanks.
Alternative 1:
You have both the lists in the same order and of same size, so you can simple do as below
for (int i = 0; i < allQueriesWeight.size(); i++) {
allQueriesWeight.get(i);
futureList.get(i);
}
Alternative 2:
If all the queries are different, you can use a map as shown below but this approach will lose the order of execution.
int totalThreads = allQueriesWeight.size();
Map<String,Future<List>> map = new HashMap<>;
ExecutorService taskExecutor = Executors.newFixedThreadPool(totalThreads);
for (String query : allQueriesWeight) {//allQueriesWeight is an arraylist containing sql server queries
SearchTask searchTask = new SearchTask(query);
Future<List> submit = taskExecutor.submit(searchTask);
map.put(query ,submit );
}
And then iterate the map
for (Entry<String,Future<List>> future : map.) {
System.out.println("query is:" +map.getKey());
List<SearchResult> sr = map.getValue().get();
}
Alternative 3
If you want to keep the order, create a class with Future and query as the attributes and then put that class in list
public class ResultWithQuery {
private final Future<List<?>> future;
private final String query;
public ResultWithQuery(Future<List<?>> future, String query) {
this.future = future;
this.query = query;
}
public Future<List<?>> getFuture() {
return future;
}
public String getQuery() {
return query;
}
}
And
List<ResultWithQuery > list = new ArrayList<ResultWithQuery >();
int totalThreads = allQueriesWeight.size();
ExecutorService taskExecutor = Executors.newFixedThreadPool(totalThreads);
for (String query : allQueriesWeight) {//allQueriesWeight is an arraylist containing sql server queries
SearchTask searchTask = new SearchTask(query);
Future<List> submit = taskExecutor.submit(searchTask);
list.add(new ResultWithQuery (submit, query));
}
And iterate the list
for (ResultWithQuery resQuery: list) {
try {
resQuery.getQuery();
List<SearchResult> sr = resQuery.getFuture.get();
} catch (InterruptedException ex) {
Logger.getLogger(GenericResource.class.getName()).log(Level.SEVERE, null, ex);
} catch (ExecutionException ex) {
Logger.getLogger(GenericResource.class.getName()).log(Level.SEVERE, null, ex);
}
}
Related
I am using Mongo async driver 3.11.2 for the following code to get distinct values from a collection along with a filter query. The distinct method doesn't seem to return any values at all. I have tried after removing the filter but no luck. A similar code works with find.
public CompletableFuture<List<String>> findDistinctByCondition(JsonNode condition, String collectionName, String distinctField) {
MongoCollection<Document> collection = mongoDatabase.getCollection(collectionName);
List<String> tripNumberList = new ArrayList<>();
CompletableFuture<List<String>> finalResult = new CompletableFuture<>();
Block<String> printDocumentBlock = new Block<String>() {
#Override
public void apply(String tripNumber) {
tripNumberList.add(tripNumber);
}
};
SingleResultCallback<Void> callbackWhenFinished = new SingleResultCallback<Void>() {
#Override
public void onResult(final Void result, final Throwable t) {
logger.info("Operation Finished!");
finalResult.complete(tripNumberList);
}
};
DistinctIterable<String> distinct = collection.
distinct(distinctField, Document.parse(condition.toString()), String.class);
distinct.forEach(printDocumentBlock , callbackWhenFinished);
try {
logger.info("Printing distinct tripNumbers");
finalResult.get().forEach(s -> {
logger.info(s);
});
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
return finalResult;
}
What might be going wrong here and how to make it work?
I have simple table with several required and nullable columns. My java application writes data into it via JsonStreamWriter. Most of time everything is ok, but sometimes it fails with error
java.util.concurrent.ExecutionException:
com.google.api.gax.rpc.PermissionDeniedException:
io.grpc.StatusRuntimeException: PERMISSION_DENIED: Permission
'TABLES_UPDATE_DATA' denied on resource
'projects/project-name/datasets/dataset-name/tables/table-name' (or it
may not exist).
Data the similar, I am just append it, without update and I have no idea what goes wrong.
private Queue<Map<String, Object>> queue = new ConcurrentLinkedQueue<>();
private JsonStreamWriter streamWriter;
#Autowired
private BigQueryManager manager;
#PostConstruct
private void initialize() {
WriteStream stream = WriteStream.newBuilder().setType(WriteStream.Type.COMMITTED).build();
TableName parentTable = TableName.of(project, dataset, table);
CreateWriteStreamRequest writeStreamRequest = CreateWriteStreamRequest.newBuilder().setParent(parentTable.toString()).setWriteStream(stream).build();
WriteStream writeStream = manager.getClient().createWriteStream(writeStreamRequest);
try {
streamWriter = JsonStreamWriter.newBuilder(writeStream.getName(), writeStream.getTableSchema(), manager.getClient()).build();
} catch (Exception ex) {
log.error("Unable to initialize stream writer.", ex);
}
}
#Override
public void flush() {
try {
List<Pair<JSONArray, Future>> tasks = new ArrayList<>();
while (!queue.isEmpty()) {
JSONArray batch = new JSONArray();
JSONObject record = new JSONObject();
queue.poll().forEach(record::put);
batch.put(record);
tasks.add(new Pair<>(batch, streamWriter.append(batch)));
}
List<AppendRowsResponse> responses = new ArrayList<>();
tasks.forEach(task -> {
try {
responses.add((AppendRowsResponse) task.getValue().get());
} catch (Exception ex) {
log.debug("Exception while task {} running: {}", task.getKey(), ex.getMessage(), ex);
}
});
responses.forEach(response -> {
if (!"".equals(response.getError().getMessage())) {
log.error(response.getError().getMessage());
}
});
} finally {
streamWriter.close();
}
}
#Override
public void addRow(Map<String, Object> row) {
queue.add(row);
}
This issue was fixed in v1.20.0. If you’re using a lower version, consider updating the library. I you’re using a higher version you could try constructing the JsonStreamWriter builder with BigQuery client being initialized by StreamWriter by default:
streamWriter = JsonStreamWriter.newBuilder(writeStream.getName(), writeStream.getTableSchema()).build();
Good afternoon, I'm trying to get the values from my database and add them to a list and I always get null. What is the cause and how could it save those values? When it comes to printing by console I have no problem. Best regards.
public List<JsonObject> getAll() {
List<JsonObject> listObject = new ArrayList<JsonObject>();
try {
this.connect();
this.client.query("SELECT * FROM user")
.execute().onSuccess(ar -> {
for (Row row : ar) {
listObject.add(row.toJson());
System.out.println("---- OK " + listObject.stream().collect(Collectors.toList()));
}
}
);
} catch (Exception e) {
System.out.println(e.getLocalizedMessage());
}
System.out.println("--- Null: " + listObject.stream().collect(Collectors.toList()));
return listObject;
}
You wouldn't use return since this is an asynchronous call.
Instead, pass a Consumer
public void getAll(Consumer<ArrayList<JsonObject>> callback) {
try {
this.connect();
this.client.query("SELECT * FROM user").execute().onSuccess(ar -> {
List<JsonObject> listObject = new ArrayList<>();
for (Row row : ar) {
listObject.add(row.toJson());
}
System.out.printf("Fetched %d objects%n", listObject.size());
callback.accept(listObject); // this replaces your return statement
});
} catch (Exception e) {
System.out.println(e.getLocalizedMessage()); // TODO: replace with a proper logger
}
}
Elsewhere, pass in an instance of a consumer
databaseClient.getAll(list -> {
// TODO: use the list to perform some action
})
Otherwise, you might want to look at R2DBC and how it can be used to wire subscribers to the result-set.
I had two seperate taskLists and want to common / generic method to execute both using invokeAll() in ExecutorService . I am able to get the response and invokeAll() for each TaskList .
But unable to write a common to execute and get the result .
LocateUser Tasks :
List<LocateUser> taskList = new ArrayList<>();
taskList.add(new BSLocateUserClient(url, locateName, username, token));
List<Future<LocateUserResponse>> locateResponse = executor.invokeAll(taskList);
locateResponse.forEach(response -> {
LocateUserResponse user;
try {
user = response.get();
} catch (InterruptedException | ExecutionException e) {
//
} finally {
executor.shutdown();
}
});
LoginResponse Tasks :
List<LoginUser> taskList = new ArrayList<>();
for (String url : urls) {
taskList.add(new BSWebserviceClient(url, username, password, isOciLogin22, ociWebServiceTemplateFactory));
}
List<Future<LoginResponse>> response = executor.invokeAll(taskList);
List<LoginResponse> loginResponses = new ArrayList<>();
response.forEach(loginResponse -> {
try {
LoginResponse loginDetails = loginResponse.get();
} catch (InterruptedException | ExecutionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
executor.shutdown();
}
});
Can you help me to resolve this ?
I achieved this one - by adding one interface ResponseTask and implements this interface to both response class and create a generic / common method for invokeAll and getResponse .
public interface ResponseTask extends Serializable {
}
invokeAll :
public <T extends ResponseTask> List<T> invokeAll(Set<Callable<ResponseTask>> callables, int threadCount) {
ThreadFactory customThreadfactory = new CustomThreadFactoryBuilder().setNamePrefix("MultiThreadExecutor-Calls")
.setDaemon(false).build();
ExecutorService executor = Executors.newFixedThreadPool(threadCount, customThreadfactory);
try {
List<Future<ResponseTask>> threadResponse = executor.invokeAll(callables);
return getResponse(threadResponse);
} catch (InterruptedException e) {
// Restore interrupted state...
Thread.currentThread().interrupt();
} finally {
executor.shutdown();
}
return Collections.emptyList();
}
getResponse :
public <T extends ResponseTask> List<T> getResponse(List<Future<ResponseTask>> threadResponse) {
List<BSLocateUserResponse> locateResponse = new ArrayList<>();
List<LoginUser> userResponse = new ArrayList<>();
for (Future<ResponseTask> response : threadResponse) {
ResponseTask result = null;
try {
result = response.get();
} catch (ExecutionException e) {
// Todo Need to capture the specific exception to ignore here
LOG.info("Exception : {} occurred when calling multithread ", e.getMessage());
continue;
} catch (InterruptedException e) {
// Restore interrupted state...
Thread.currentThread().interrupt();
}
if (result instanceof BSLocateUserResponse) {
locateResponse.add((BSLocateUserResponse) result);
} else if (result instanceof LoginUser) {
userResponse.add((LoginUser) result);
}
}
if (locateResponse.isEmpty()) {
return (List<T>) userResponse;
}
return (List<T>) locateResponse;
}
Let's say we've got an SQL database (hsqldb) and want to run a number of queries on it which do not modify the content.
This takes a long time for some queries and I would like to run the queries in multiple threads.
So my question is: what is the best way to implement this?
I did not find any good samples to do this so I came up with the following (which I would love to get some comments on).
First, very briefly in words:
I use thread-safe collections to access the queries and to put the results in. The queries are executed in a number of worker threads. The results are processed in the main thread which checks for new results until all threads are finished.
Now the code:
Create thread-safe collections of queries and results:
ConcurrentLinkedQueue<String> queries = new ConcurrentLinkedQueue<String>()
ConcurrentLinkedQueue<ResultSet> sqlResults = new ConcurrentLinkedQueue<ResultSet>();
Create a number of threads and start them (edited):
ExecutorService executorService = Executors.newFixedThreadPool(4);
for(int i=0; i<4; i++){
executorService.execute(new QueryThread(sqlResults, queries));
}
Within the thread class QueryThread a connection is opened and queries are executed as long as there are any left:
private class QueryThread implements Runnable {
private ConcurrentLinkedQueue<ResultSet> sqlResults;
private ConcurrentLinkedQueue<String> queries;
public QueryThread(ConcurrentLinkedQueue<ResultSet> sqlResults, ConcurrentLinkedQueue<String> queries){
this.sqlResults = sqlResults;
this.queries = queries;
}
#Override
public void run(){
Connection connThr = null;
try{
try {
connThr = DriverManager.getConnection(dbModeSave, "sa", "");
connThr.setAutoCommit(false);
} catch (SQLException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
String currentQuery;
do {
currentQuery = queries.poll(); // get and remove element from remaining queries
if (currentQuery != null) { // only continue if element was found
try {
Statement stmnt = connThr.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,ResultSet.CONCUR_UPDATABLE);
try {
ResultSet resultSet = stmnt.executeQuery(currentQuery);
sqlResults.add(resultSet);
} catch (SQLException e) {
// (Do something specific)
} finally {
stmnt.close();
}
} catch (SQLException e) {
// (Do something specific)
}
}
} while (currentQuery != null);
} finally {
if (connThr != null) {
try {
connThr.close();
} catch (SQLException e) {
// Nothing we can do?
}
}
}
}
}
From the original thread I check, if the threads are all finished and therefore all queries were processed (edited).
while (!executorService.isTerminated()) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
while (!sqlResults.isEmpty()) {
ResultSet result = sqlResults.poll();
//process result and close it in the end
}
}
Java standard sulution for parallel processing is ThreadPoolExecutor. Try it.