I am currently facing the following issue with Java concurrency. I want to parse a friend list on a website, then search for the friends of the friends etc. recursively.
Here it is represented visually:
0
/|\
0 0 0
/|\
0 0 0
I came up with a solution but it does not perform as well as I expected, I assume my logic might be somewhat faulty.
private ArrayList<String> getUserFriendsProfileURLs(final String uri, final int indexOfDeep, int foldenesLevel)
throws IOException {
var usersURIs = getUsers(uri); //long network call
ArrayList<String> uris = new ArrayList<>();
uris.addAll(usersURIs);
if (indexOfDeep != foldenesLevel) {
List<CompletableFuture<ArrayList<String>>> futures = new ArrayList<>();
usersURIs.forEach(useruri -> {
CompletableFuture<ArrayList<String>> future = CompletableFuture.supplyAsync(new Supplier<>() {
public ArrayList<String> get() {
var friendsOfUser = new ArrayList<String>();
try {
friendsOfUser = getUserFriendsProfileURLs(useruri, indexOfDeep, 1 + foldenesLevel);
} catch (IOException e) {
throw new IllegalStateException(e);
}
return friendsOfUser;
}
});
futures.add(future);
});
CompletableFuture<Void> allFuturesResult = CompletableFuture
.allOf(futures.toArray(new CompletableFuture[futures.size()]));
var res = allFuturesResult.thenApply(v -> futures.stream().map(CompletableFuture::join).toList());
try {
uris = (ArrayList<String>) res.get().stream().flatMap(Collection::stream).collect(Collectors.toList());
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
}
return uris;
}
I wrote 2nd version of function:
/**
* getUserFriendsProfileURLs returns list of friends of given steamid DFS
* algorithm is used
*
*/
private ArrayList<String> getUserFriendsProfileURLs(final int indexOfDeep, int foldenesLevel,
ArrayList<String> usersURIs) throws IOException {
ArrayList<String> uris = new ArrayList<>();
uris.addAll(usersURIs);
if (indexOfDeep != foldenesLevel) {
List<CompletableFuture<ArrayList<String>>> futures = new ArrayList<>();
usersURIs.forEach(useruri -> {
CompletableFuture<ArrayList<String>> future = CompletableFuture.supplyAsync(new Supplier<>() {
public ArrayList<String> get() {
ArrayList<String> ur = null;
try {
ur = getUsers(useruri); // long network call
} catch (IOException e1) {
e1.printStackTrace();
}
return ur;
}
});
futures.add(future);
});
CompletableFuture<Void> allFuturesResult = CompletableFuture
.allOf(futures.toArray(new CompletableFuture[futures.size()]));
var res = allFuturesResult.thenApply(v -> futures.stream().map(CompletableFuture::join).toList());
try {
uris = (ArrayList<String>) res.get().stream().flatMap(Collection::stream).collect(Collectors.toList());
uris = getUserFriendsProfileURLs(indexOfDeep, 1 + foldenesLevel, uris);
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
}
return uris;
}
Related
I am using Mongo async driver 3.11.2 for the following code to get distinct values from a collection along with a filter query. The distinct method doesn't seem to return any values at all. I have tried after removing the filter but no luck. A similar code works with find.
public CompletableFuture<List<String>> findDistinctByCondition(JsonNode condition, String collectionName, String distinctField) {
MongoCollection<Document> collection = mongoDatabase.getCollection(collectionName);
List<String> tripNumberList = new ArrayList<>();
CompletableFuture<List<String>> finalResult = new CompletableFuture<>();
Block<String> printDocumentBlock = new Block<String>() {
#Override
public void apply(String tripNumber) {
tripNumberList.add(tripNumber);
}
};
SingleResultCallback<Void> callbackWhenFinished = new SingleResultCallback<Void>() {
#Override
public void onResult(final Void result, final Throwable t) {
logger.info("Operation Finished!");
finalResult.complete(tripNumberList);
}
};
DistinctIterable<String> distinct = collection.
distinct(distinctField, Document.parse(condition.toString()), String.class);
distinct.forEach(printDocumentBlock , callbackWhenFinished);
try {
logger.info("Printing distinct tripNumbers");
finalResult.get().forEach(s -> {
logger.info(s);
});
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
return finalResult;
}
What might be going wrong here and how to make it work?
I'm writing a Java bot with https://github.com/rubenlagus/TelegramBots, and I have a problem, when I click inline keyboard button, this little clock:
appears and after some time it says that my bot is not responding. My bot is actually fully functional except this one thing. Here is how I receive callbacks:
#Override
public void onUpdateReceived(Update update) {
var messagesToSend = updateReceiver.handle(update);
if (messagesToSend != null && !messagesToSend.isEmpty()) {
messagesToSend.forEach(response -> {
if (response instanceof SendMessage) {
try {
execute((SendMessage) response);
} catch (TelegramApiException e) {
e.printStackTrace();
}
} else if (response instanceof SendPhoto) {
try {
execute((SendPhoto) response);
} catch (TelegramApiException e) {
e.printStackTrace();
}
} else if (response instanceof FileSaveRequest) {
FileSaveRequest request = (FileSaveRequest) response;
try {
saveFile(request);
} catch (TelegramApiException | IOException e) {
e.printStackTrace();
}
}
});
}
}
=
This is only part of the whole code
} else if (update.hasCallbackQuery()) {
final CallbackQuery callbackQuery = update.getCallbackQuery();
final long chatId = callbackQuery.getFrom().getId();
final User user = userRepository.findById(chatId)
.orElseGet(() -> userRepository.save(new User(chatId)));
AnswerCallbackQuery acceptCallback = new AnswerCallbackQuery();
acceptCallback.setShowAlert(false);
acceptCallback.setCallbackQueryId(String.valueOf(update.getCallbackQuery().getId()));
acceptCallback.setCacheTime(1000);
List<PartialBotApiMethod<? extends Serializable>> resultList =
new ArrayList<>(
getHandlerByCallBackQuery(callbackQuery.getData())
.handle(user, callbackQuery.getData()));
resultList.add(acceptCallback);
return resultList;
}
As you can see, I still attach AnswerCallbackQuery but it still doesent work. What's wrong?
you must use answercallbackquery
I just already solve that issue. It's not problem on Library but it could error in some exceptions.
var messagesToSend = updateReceiver.handle(update);
if (messagesToSend != null && !messagesToSend.isEmpty()) {
I dont have full your code but I think there's some confused written and happen exception before if (update.callbackQuery())...
Here is my sample:
#Override
public void onUpdateReceived(Update update) {
// I have error, cannot getCallbackQuery because of print which call method getMessage.getText() is null -> happen exception error on the println
// -> System.out.println(update.getMessage.getText());
if (update.hasMessage() && !update.getMessage().getText().isEmpty()) {
String chat_id = update.getMessage().getChatId().toString();
if (update.getMessage().getText().equals("/start")) {
SendMessage sendMessage = new SendMessage();
sendMessage.setText("Here is option:");
sendMessage.setChatId(chat_id);
sendMessage.setParseMode(ParseMode.MARKDOWN);
InlineKeyboardMarkup inlineKeyboardMarkup = new InlineKeyboardMarkup();
List<List<InlineKeyboardButton>> listInlineButton = new ArrayList<>();
List<InlineKeyboardButton> reportSaleBtn = new ArrayList<>();
List<InlineKeyboardButton> reportBuyBtn = new ArrayList<>();
List<InlineKeyboardButton> reportPriceBtn = new ArrayList<>();
InlineKeyboardButton saleBtn = new InlineKeyboardButton();
InlineKeyboardButton buyBtn = new InlineKeyboardButton();
InlineKeyboardButton priceBtn = new InlineKeyboardButton();
saleBtn.setText(Constant.SALE_REPORT_TEXT);
saleBtn.setCallbackData(Constant.SALE_REPORT);
buyBtn.setText(Constant.BUY_REPORT_TEXT);
buyBtn.setCallbackData(Constant.BUY_REPORT);
priceBtn.setText(Constant.PRICE_TEXT);
priceBtn.setCallbackData(Constant.PRICE_REPORT);
reportSaleBtn.add(saleBtn);
reportBuyBtn.add(buyBtn);
reportPriceBtn.add(priceBtn);
listInlineButton.add(reportSaleBtn);
listInlineButton.add(reportBuyBtn);
listInlineButton.add(reportPriceBtn);
inlineKeyboardMarkup.setKeyboard(listInlineButton);
sendMessage.setReplyMarkup(inlineKeyboardMarkup);
try {
execute(sendMessage);
} catch (TelegramApiException e) {
e.printStackTrace();
}
}
}
else if (update.hasCallbackQuery()) {
CallbackQuery callbackQuery = update.getCallbackQuery();
String data = callbackQuery.getData();
String chat_id = callbackQuery.getMessage().getChat().getId().toString();
SendChatAction sendChatAction = new SendChatAction();
if (data.equals(Constant.SALE_REPORT)) {
sendChatAction.setChatId(chat_id);
SendMessage sendMessage = new SendMessage();
sendMessage.setText("Generating report, please wait!");
sendMessage.setChatId(chat_id);
try {
sendChatAction.setAction(ActionType.TYPING);
execute(sendChatAction);
execute(sendMessage);
} catch (TelegramApiException e) {
e.printStackTrace();
}
}
}
Why it got an error. Click we click on /start Bot will display all inlinekeyboard.
In the button you only setText() and setCallbackData(). So update.GetMessage() is null.
In while update.getMessage().getText() is null cannot print out. So it is error and it skip the else if (update.hasCallbackQuery()) {...}
I think you can check again your code below:
#Override
public void onUpdateReceived(Update update) {
//check carefully before if may there's exception error before if
}
I hope what I explain may solve your problems too.
I'm trying to use futures to make concurrent api calls. Code:
private void init() throws ExecutionException, InterruptedException {
Long start = System.currentTimeMillis();
List<ApiResponse> responses = fetchAllUsingFuture(ids, 3);
log.info(responses.toString());
Long finish = System.currentTimeMillis();
log.info(MessageFormat.format("Process duration: {0} in ms", finish-start));
}
private List<ApiResponse> fetchAllUsingFuture(List<String> ids, int threadCount) throws ExecutionException, InterruptedException {
ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
List<List<String>> chunks = Utils.splitToChunks(ids, threadCount);
List<Future<List<ApiResponse>>> futures = new ArrayList<>();
chunks.forEach(chunk -> {
futures.add(wrapFetchInFuture(chunk));
});
Future<List<ApiResponse>> resultFuture = executorService.submit(() -> {
List<ApiResponse> responses = new ArrayList<>();
futures.forEach(future -> {
try {
responses.addAll(future.get());
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
});
return responses;
});
executorService.shutdown();
return resultFuture.get();
}
private Future<List<ApiResponse>> wrapFetchInFuture(List<String> ids) {
return new FutureTask<>(() -> {
List<ApiResponse> responses = new ArrayList<>();
ids.forEach(id -> {
responses.add(fetchData(id));
});
return responses;
});
}
private ApiResponse fetchData(String id) {
ResponseEntity<ApiResponse> response = restTemplate.getForEntity(id, ApiResponse.class);
log.info(MessageFormat.format("Fetching from {0}", id));
ApiResponse body = response.getBody();
log.info(MessageFormat.format("Retrieved {0}", body));
return body;
}
It doesn't execute, the app starts and then just pends. Futures don't get fulfilled. All advices are appreciated.
P.S. I'm aware this is much more easily done using CompletableFuture, I was just wondering how to do this with Futures
In the original version of the question, you are creating a list of FutureTasks but never send them to the ExecutorService to run them. The tasks never complete, so Future.get blocks forever.
In the updated version of the question, you have put the code that does the waiting into the executor service as a task. The FutureTasks never run, so FutureTask.get will still block forever.
I would suggest you change the code in fetchAllUsingFuture to:
List<Callable<List<ApiResponse>>> tasks = new ArrayList<>();
chunks.forEach(chunk -> {
tasks.add(wrapFetchInCallable(chunk));
});
List<Future<List<ApiResponse>>> futures = executorService.invokeAll(tasks);
where wrapFetchInCallable creates a Callable instead of FutureTask:
private static Callable<List<ApiResponse>> wrapFetchInCallable(List<String> ids) {
return () -> {
List<ApiResponse> responses = new ArrayList<>();
ids.forEach(id -> {
responses.add(fetchData(id));
});
return responses;
};
}
It looks like you are creating a list of FutureTasks but never send them to the ExecutorService to run them.
I have implemented ExecutorService with Future Object as below, i hope it helps you:
Service layer:
public List<MovieDTO> searchMoviesParallel(String limit, String offset, String searchPhrase) throws Exception {
ExecutorService executor = Executors.newFixedThreadPool(1);
Future<List<MovieDTO>> digitoonResult = executor.submit(new DigitoonSearchTask(limit, offset, searchPhrase));
List<MovieDTO> movieDTOList = digitoonResult.get();
executor.shutdown();
return movieDTOList;
}
And my Search task(DigitoonSearchTask class) is as below:
public class DigitoonSearchTask implements Callable<List<MovieDTO>> {
private String limit;
private String offset;
private String searchPhrase;
private final static String digitoonSearchBaseUrl = "http://apitwo.xxx.com/partner/search/?q=";
public DigitoonSearchTask(String limit, String offset, String searchPhrase) {
this.limit = limit;
this.offset = offset;
this.searchPhrase = searchPhrase;
}
#Override
public List<MovieDTO> call() throws Exception {
List<MovieDTO> movieDTOList = new ArrayList<>();
ObjectMapper mapper = new ObjectMapper();
try {
String uri = digitoonSearchBaseUrl + URLEncoder.encode(searchPhrase, "utf-8") + "&limit=" + limit + "&offset=" + offset;
URL url = new URL(uri);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
conn.setRequestProperty("authorization", "xxxxxxxxxx");
if (conn.getResponseCode() != 200) {
throw new RuntimeException("Failed : HTTP error code : "
+ conn.getResponseCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn.getInputStream())));
String output;
while ((output = br.readLine()) != null) {
movieDTOList = Arrays.asList(mapper.readValue(output, MovieDTO[].class));
}
br.close();
conn.disconnect();
} catch (UnknownHostException e) {
call();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return movieDTOList;
}}
consider that now I have just one API and after getting others they can be added as another Search task in service layer by increasing the thread number.
I had two seperate taskLists and want to common / generic method to execute both using invokeAll() in ExecutorService . I am able to get the response and invokeAll() for each TaskList .
But unable to write a common to execute and get the result .
LocateUser Tasks :
List<LocateUser> taskList = new ArrayList<>();
taskList.add(new BSLocateUserClient(url, locateName, username, token));
List<Future<LocateUserResponse>> locateResponse = executor.invokeAll(taskList);
locateResponse.forEach(response -> {
LocateUserResponse user;
try {
user = response.get();
} catch (InterruptedException | ExecutionException e) {
//
} finally {
executor.shutdown();
}
});
LoginResponse Tasks :
List<LoginUser> taskList = new ArrayList<>();
for (String url : urls) {
taskList.add(new BSWebserviceClient(url, username, password, isOciLogin22, ociWebServiceTemplateFactory));
}
List<Future<LoginResponse>> response = executor.invokeAll(taskList);
List<LoginResponse> loginResponses = new ArrayList<>();
response.forEach(loginResponse -> {
try {
LoginResponse loginDetails = loginResponse.get();
} catch (InterruptedException | ExecutionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
executor.shutdown();
}
});
Can you help me to resolve this ?
I achieved this one - by adding one interface ResponseTask and implements this interface to both response class and create a generic / common method for invokeAll and getResponse .
public interface ResponseTask extends Serializable {
}
invokeAll :
public <T extends ResponseTask> List<T> invokeAll(Set<Callable<ResponseTask>> callables, int threadCount) {
ThreadFactory customThreadfactory = new CustomThreadFactoryBuilder().setNamePrefix("MultiThreadExecutor-Calls")
.setDaemon(false).build();
ExecutorService executor = Executors.newFixedThreadPool(threadCount, customThreadfactory);
try {
List<Future<ResponseTask>> threadResponse = executor.invokeAll(callables);
return getResponse(threadResponse);
} catch (InterruptedException e) {
// Restore interrupted state...
Thread.currentThread().interrupt();
} finally {
executor.shutdown();
}
return Collections.emptyList();
}
getResponse :
public <T extends ResponseTask> List<T> getResponse(List<Future<ResponseTask>> threadResponse) {
List<BSLocateUserResponse> locateResponse = new ArrayList<>();
List<LoginUser> userResponse = new ArrayList<>();
for (Future<ResponseTask> response : threadResponse) {
ResponseTask result = null;
try {
result = response.get();
} catch (ExecutionException e) {
// Todo Need to capture the specific exception to ignore here
LOG.info("Exception : {} occurred when calling multithread ", e.getMessage());
continue;
} catch (InterruptedException e) {
// Restore interrupted state...
Thread.currentThread().interrupt();
}
if (result instanceof BSLocateUserResponse) {
locateResponse.add((BSLocateUserResponse) result);
} else if (result instanceof LoginUser) {
userResponse.add((LoginUser) result);
}
}
if (locateResponse.isEmpty()) {
return (List<T>) userResponse;
}
return (List<T>) locateResponse;
}
This code is returning only 100 results, even if there are tens of thousands of entries in the redis database:
RedisFuture<List<Object>> future = redis.dispatch(CommandType.SCAN, new ArrayOutput<String, String>(StringCodec.UTF8), new CommandArgs<String, String>(StringCodec.UTF8).add("aircraft"));
List<Object> result = Collections.emptyList();
try {
result = future.get();
} catch (InterruptedException | ExecutionException e) {
// TODO: Do something
}
if(result.size() > 1) {
#SuppressWarnings("unchecked")
List<List<String>> aircraft = (List<List<String>>) result.get(1);
Map<String, Aircraft> retval = aircraft.stream().collect(Collectors.toMap(it -> it.get(0), it -> {
try {
JsonNode positionJson = objectMapper.readTree(it.get(1)).get("coordinates");
return new Aircraft(positionJson.get(0).asDouble(), positionJson.get(1).asDouble());
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}));
return retval;
} else {
return Collections.emptyMap();
}