Develop Reference

BigQuery JsonStreamWriter sometimes throw PERMISSION_DENIED error

I have simple table with several required and nullable columns. My java application writes data into it via JsonStreamWriter. Most of time everything is ok, but sometimes it fails with error
java.util.concurrent.ExecutionException:
com.google.api.gax.rpc.PermissionDeniedException:
io.grpc.StatusRuntimeException: PERMISSION_DENIED: Permission
'TABLES_UPDATE_DATA' denied on resource
'projects/project-name/datasets/dataset-name/tables/table-name' (or it
may not exist).
Data the similar, I am just append it, without update and I have no idea what goes wrong.
private Queue<Map<String, Object>> queue = new ConcurrentLinkedQueue<>();
private JsonStreamWriter streamWriter;
#Autowired
private BigQueryManager manager;
#PostConstruct
private void initialize() {
WriteStream stream = WriteStream.newBuilder().setType(WriteStream.Type.COMMITTED).build();
TableName parentTable = TableName.of(project, dataset, table);
CreateWriteStreamRequest writeStreamRequest = CreateWriteStreamRequest.newBuilder().setParent(parentTable.toString()).setWriteStream(stream).build();
WriteStream writeStream = manager.getClient().createWriteStream(writeStreamRequest);
try {
streamWriter = JsonStreamWriter.newBuilder(writeStream.getName(), writeStream.getTableSchema(), manager.getClient()).build();
} catch (Exception ex) {
log.error("Unable to initialize stream writer.", ex);
}
}
#Override
public void flush() {
try {
List<Pair<JSONArray, Future>> tasks = new ArrayList<>();
while (!queue.isEmpty()) {
JSONArray batch = new JSONArray();
JSONObject record = new JSONObject();
queue.poll().forEach(record::put);
batch.put(record);
tasks.add(new Pair<>(batch, streamWriter.append(batch)));
}
List<AppendRowsResponse> responses = new ArrayList<>();
tasks.forEach(task -> {
try {
responses.add((AppendRowsResponse) task.getValue().get());
} catch (Exception ex) {
log.debug("Exception while task {} running: {}", task.getKey(), ex.getMessage(), ex);
}
});
responses.forEach(response -> {
if (!"".equals(response.getError().getMessage())) {
log.error(response.getError().getMessage());
}
});
} finally {
streamWriter.close();
}
}
#Override
public void addRow(Map<String, Object> row) {
queue.add(row);
}

This issue was fixed in v1.20.0. If you’re using a lower version, consider updating the library. I you’re using a higher version you could try constructing the JsonStreamWriter builder with BigQuery client being initialized by StreamWriter by default:
streamWriter = JsonStreamWriter.newBuilder(writeStream.getName(), writeStream.getTableSchema()).build();

Collecting results from a list of Futures in java

I'm trying to use futures to make concurrent api calls. Code:
private void init() throws ExecutionException, InterruptedException {
Long start = System.currentTimeMillis();
List<ApiResponse> responses = fetchAllUsingFuture(ids, 3);
log.info(responses.toString());
Long finish = System.currentTimeMillis();
log.info(MessageFormat.format("Process duration: {0} in ms", finish-start));
}
private List<ApiResponse> fetchAllUsingFuture(List<String> ids, int threadCount) throws ExecutionException, InterruptedException {
ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
List<List<String>> chunks = Utils.splitToChunks(ids, threadCount);
List<Future<List<ApiResponse>>> futures = new ArrayList<>();
chunks.forEach(chunk -> {
futures.add(wrapFetchInFuture(chunk));
});
Future<List<ApiResponse>> resultFuture = executorService.submit(() -> {
List<ApiResponse> responses = new ArrayList<>();
futures.forEach(future -> {
try {
responses.addAll(future.get());
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
});
return responses;
});
executorService.shutdown();
return resultFuture.get();
}
private Future<List<ApiResponse>> wrapFetchInFuture(List<String> ids) {
return new FutureTask<>(() -> {
List<ApiResponse> responses = new ArrayList<>();
ids.forEach(id -> {
responses.add(fetchData(id));
});
return responses;
});
}
private ApiResponse fetchData(String id) {
ResponseEntity<ApiResponse> response = restTemplate.getForEntity(id, ApiResponse.class);
log.info(MessageFormat.format("Fetching from {0}", id));
ApiResponse body = response.getBody();
log.info(MessageFormat.format("Retrieved {0}", body));
return body;
}
It doesn't execute, the app starts and then just pends. Futures don't get fulfilled. All advices are appreciated.
P.S. I'm aware this is much more easily done using CompletableFuture, I was just wondering how to do this with Futures

In the original version of the question, you are creating a list of FutureTasks but never send them to the ExecutorService to run them. The tasks never complete, so Future.get blocks forever.
In the updated version of the question, you have put the code that does the waiting into the executor service as a task. The FutureTasks never run, so FutureTask.get will still block forever.
I would suggest you change the code in fetchAllUsingFuture to:
List<Callable<List<ApiResponse>>> tasks = new ArrayList<>();
chunks.forEach(chunk -> {
tasks.add(wrapFetchInCallable(chunk));
});
List<Future<List<ApiResponse>>> futures = executorService.invokeAll(tasks);
where wrapFetchInCallable creates a Callable instead of FutureTask:
private static Callable<List<ApiResponse>> wrapFetchInCallable(List<String> ids) {
return () -> {
List<ApiResponse> responses = new ArrayList<>();
ids.forEach(id -> {
responses.add(fetchData(id));
});
return responses;
};
}

It looks like you are creating a list of FutureTasks but never send them to the ExecutorService to run them.
I have implemented ExecutorService with Future Object as below, i hope it helps you:
Service layer:
public List<MovieDTO> searchMoviesParallel(String limit, String offset, String searchPhrase) throws Exception {
ExecutorService executor = Executors.newFixedThreadPool(1);
Future<List<MovieDTO>> digitoonResult = executor.submit(new DigitoonSearchTask(limit, offset, searchPhrase));
List<MovieDTO> movieDTOList = digitoonResult.get();
executor.shutdown();
return movieDTOList;
}
And my Search task(DigitoonSearchTask class) is as below:
public class DigitoonSearchTask implements Callable<List<MovieDTO>> {
private String limit;
private String offset;
private String searchPhrase;
private final static String digitoonSearchBaseUrl = "http://apitwo.xxx.com/partner/search/?q=";
public DigitoonSearchTask(String limit, String offset, String searchPhrase) {
this.limit = limit;
this.offset = offset;
this.searchPhrase = searchPhrase;
}
#Override
public List<MovieDTO> call() throws Exception {
List<MovieDTO> movieDTOList = new ArrayList<>();
ObjectMapper mapper = new ObjectMapper();
try {
String uri = digitoonSearchBaseUrl + URLEncoder.encode(searchPhrase, "utf-8") + "&limit=" + limit + "&offset=" + offset;
URL url = new URL(uri);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
conn.setRequestProperty("authorization", "xxxxxxxxxx");
if (conn.getResponseCode() != 200) {
throw new RuntimeException("Failed : HTTP error code : "
+ conn.getResponseCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn.getInputStream())));
String output;
while ((output = br.readLine()) != null) {
movieDTOList = Arrays.asList(mapper.readValue(output, MovieDTO[].class));
}
br.close();
conn.disconnect();
} catch (UnknownHostException e) {
call();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return movieDTOList;
}}
consider that now I have just one API and after getting others they can be added as another Search task in service layer by increasing the thread number.

Unable to catch thrown exception from async method in Spring

I am unable to catch thrown exceptions from an async method in Spring. I have written an uncaught exception handler to catch but was unsuccessful.
The application will enable to start any number of forever running asynchronous jobs.
I think my async method needs to return Future so that I can store it in hashmap and check its status or stop the job. I also can get all running jobs by storing it.
I think I can't use get method of future because if the input is correct it blocks and my job will be forever running. I need to send status as started if the input is fine. Whenever an exception occurs in the Async method it is thrown but I am unable to catch it. How can I do that?
Here is my complete code.
Application.java
#SpringBootApplication
#EnableAsync
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
AsyncConfig.java
#EnableAsync
#Configuration
public class AsyncConfig implements AsyncConfigurer {
#Override
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5);
executor.setMaxPoolSize(5);
executor.setQueueCapacity(100);
executor.setThreadNamePrefix("MyExecutor-");
executor.initialize();
return executor;
}
#Override
public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler() {
return new AsyncExceptionHandler();
}
}
AsyncExceptionHandler.java
public class AsyncExceptionHandler implements AsyncUncaughtExceptionHandler {
#Override
public void handleUncaughtException(Throwable throwable, Method method, Object... obj) {
System.out.println("Exception Cause - " + throwable.getMessage());
System.out.println("Method name - " + method.getName());
for (Object param : obj) {
System.out.println("Parameter value - " + param);
}
}
}
createBucket.java
#Service
public class createBucket {
#Async
public Future<String> start(String config){
try {
JSONObject map = new JSONObject(config);
Jedis jedis = new Jedis(map.getString("jedisip"));
jedis.auth(map.getString("password"));
// code to make a kafka consumer subscribe to a topic given in config input
while(true) {
//forever running code which polls using a kafka consumer
}
}
catch(JedisException j) {
throw new JedisException("Some msg");
}
}
}
Endpoint.java
#Controller
public class Endpoint {
#Autowired
private createBucket service;
private Future<String> out;
private HashMap<String, Future<String>> maps = new HashMap<>();
#PostMapping(value = "/start", consumes = "application/json", produces = "application/json")
public ResponseEntity<String> starttask(#RequestBody String conf) {
try {
out = service.start(conf);
maps.put(conf, out);
}
catch (Exception e) {
return new ResponseEntity<>("exception", HttpStatus.BAD_REQUEST);
}
return new ResponseEntity<>("{\"started\":\"true\"}", HttpStatus.CREATED);
}
}

As stated in official doc, AsyncUncaughtExceptionHandler is used for void return value.
https://docs.spring.io/spring/docs/5.1.10.RELEASE/spring-framework-reference/integration.html#spring-integration
In your scenario, I recommend using CompletableFuture and DeferredResult:
#Async
public CompletableFuture<String> start(String config) {
CompletableFuture completableFuture = new CompletableFuture();
try {
JSONObject map = new JSONObject(config);
Jedis jedis = new Jedis(map.getString("jedisip"));
jedis.auth(map.getString("password"));
completableFuture.complete("started!");
}
catch(JedisException j) {
completableFuture.completeExceptionally(j);
}
return completableFuture;
}
#PostMapping(value = "/start", consumes = "application/json", produces = "application/json")
public DeferredResult<ResponseEntity> starttask(#RequestBody String conf) {
CompletableFuture<String> start = service.start(conf);
DeferredResult<ResponseEntity> deferredResult = new DeferredResult<>();
start.whenComplete((res, ex) -> {
if (ex == null) {
ResponseEntity<String> successEntity = new ResponseEntity<>("{\"started\":\"true\"}", HttpStatus.CREATED);\
deferredResult.setResult(successEntity);
} else {
// handle ex here!
ResponseEntity<String> exEntity = new ResponseEntity<>("exception", HttpStatus.BAD_REQUEST);
deferredResult.setResult(exEntity);
}
});
return deferredResult;
}
There is another serious problem. The following code is not thread safe.
private Future<String> out;
private HashMap<String, Future<String>> maps = new HashMap<>();

How socket.io get response with synchronous call on android?

I'm writing a android chat application with socket.io-client-java.I want to check whether the client user exist at first.So I need to send a command like "user/exist" to server url and get the response from server.I need to wait the server response then can go to next step.But the socket.io use the asynchronous callback.For getting the response synchronous I known the Furture and Callable only.So I tried the way using code as below:
//this is request method using socket.io
public JSONObject request(final String method,final String url,final JSONObject data){
final JSONObject responseObj = new JSONObject();
if (mSocket.connected()) {
mSocket.emit(method, reqObj, new Ack() {
#Override
public void call(Object... objects) {
System.out.println("get Ack");
try {
responseObj.put("body", (JSONObject) objects[0]);
}catch (JSONException e){
e.printStackTrace();
}
}
})
}
}
//this is Callable call implement
#Override
public JSONObject call(){
return request("get","https://my-chat-server/user/exist",new JSONObject());
}
//this is call method in activity
ExecutorService executor = Executors.newCachedThreadPool();
Future<JSONObject> response = executor.submit(mApiSocket);
executor.shutdown();
JSONObject respObj = new JSONObject();
JSONObject respBody = new JSONObject();
try {
respObj = response.get();
respBody = respObj.getJSONObject("body");
}catch (ExecutionException e){
}catch(InterruptedException e1){
}catch(JSONException e2){
}
But it dose not work.The respObj is null.
How can i get the reponse synchronous?
I am a green hand on java and forgive my poor chinese english.
Any help would be appreciated!

I known the js can use Promise and await like below:
//request method
static request(method, url, data) {
return new Promise((resolve, reject) => {
this.socket.emit(method,
{
url: url,
method,
data,
},
async (res) => {
if (res.statusCode == 100) {
resolve(res.body, res);
} else {
throw new Error(`${res.statusCode} error: ${res.body}`);
reject(res.body, res);
}
}
)
})
}
//call method
response = await mSocket.request('get','https://my-chat-server/user/exist', {
first_name: 'xu',
last_name: 'zhitong',
});

I'm not sure this is the best way but we can wait for the callback as follows:
#Nullable
Object[] emitAndWaitForAck(#NotNull String event, #Nullable Object[] args,
long timeoutMillis) {
Object[][] response = new Object[1][1];
Semaphore lock = new Semaphore(0);
socketClient.emit(event, args, ackArgs -> {
response[0] = ackArgs;
lock.release();
});
try {
boolean acquired = lock.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
if (acquired) {
return response[0];
}
} catch (InterruptedException ignored) {
}
return null;
}
Assuming your socket.io server returns one argument containing the body (or null) you would call it something like this:
String method = "get";
String url = "https://my-chat-server/user/exist";
long timeoutMillis = 5000;
Object[] args = emitAndWaitForAck(method, new String[]{url}, timeoutMillis);
JSONObject response = (JSONObject) args[0];

Apache's HttpAsyncClient unreliable with multiple requests

I'm currently in a project where I have to do multiple, concurrent http requests to a rest service which returns a JSON response. This is a batch operation and the number of requests at any time could range from several hunderd to several thousend.
That's why I thought it would be a good idea to have an async http client so I could have concurrent requests, which dramatically could speed up the process. I first tried ning's async-http-client. Maybe I was doing something wrong, because it was kind of slow for me. About 10 seconds for 1000 requests.
After which I tried Apache's implementation which was much faster at about 4 seconds for 1000 requests. But I can't seem to get the requests to get stable. Most of the time I will get a List with a 1000 responses (like I expect), but sometimes I am just missing a few responses, like 1 or 2.
This is currently my code:
public class AsyncServiceTest {
public AsyncServiceTest(String serviceURI) {
this.httpClient = HttpAsyncClients.custom().setMaxConnPerRoute(100).setMaxConnTotal(20)
.setDefaultRequestConfig(RequestConfig.custom().build()).build();
this.objectMapper = new ObjectMapper();
this.serviceURI = serviceURI;
}
private List<Object> getResults(List<String> queryStrings) throws Exception {
try {
httpClient.start();
final List<HttpGet> requests = new ArrayList<>(addresses.size());
for (String str : queryStrings) {
requests.add(new HttpGet(buildUri(str))); // In this method we build the absolute request uri.
}
final CountDownLatch latch = new CountDownLatch(requests.size());
final List<Object> responses = new ArrayList<>(requests.size());
final List<String> stringResponses = new ArrayList<>(requests.size());
for (final HttpGet request : requests) {
httpClient.execute(request, new FutureCallback<HttpResponse>() {
#Override
public void completed(HttpResponse response) {
try {
stringResponses.add(IOUtils.toString(response.getEntity().getContent(), "UTF-8"));
latch.countDown();
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public void failed(Exception e) {
latch.countDown();
}
#Override
public void cancelled() {
latch.countDown();
}
});
}
latch.await();
for (String r : stringResponses) {
responses.add(mapToLocation(r)); // Mapping some Strings to JSON in this method.
}
return responses;
} finally {
httpClient.close();
}
}
}
So, in essence, I am wondering if there is something wrong with my code (probably) or is it just because of the way the library works? Because the CountDownLatch is at zero all the time. Or does anyone have a pointer in the right direction (maybe with another library)?

It seemed to be a concurrency problem (thanks to #vanOekel) in my code. The answer is to replace the ArrayList<E> with a Vector<E>, which is in fact thread-safe. Example code:
public class AsyncServiceTest {
public AsyncServiceTest(String serviceURI) {
this.httpClient = HttpAsyncClients.custom().setMaxConnPerRoute(100).setMaxConnTotal(20)
.setDefaultRequestConfig(RequestConfig.custom().build()).build();
this.objectMapper = new ObjectMapper();
this.serviceURI = serviceURI;
}
private List<Object> getResults(List<String> queryStrings) throws Exception {
try {
httpClient.start();
final CountDownLatch latch = new CountDownLatch(queryStrings.size());
final Vector<Object> responses = new Vector<>(queryStrings.size());
for (String str : queryStrings) {
// buildUri: In this method we build the absolute request uri.
httpClient.execute(new HttpGet(buildUri(str)), new FutureCallback<HttpResponse>() {
#Override
public void completed(HttpResponse response) {
try {
// mapToLocation: Mapping some Strings to JSON in this method.
responses.add(mapToLocation(IOUtils.toString(response.getEntity().getContent(), "UTF-8")));
latch.countDown();
} catch (IOException e) {
failed(e);
}
}
#Override
public void failed(Exception e) {
logger.error(e.getLocalizedMessage(), e);
latch.countDown();
}
#Override
public void cancelled() {
logger.error("Request cancelled.");
latch.countDown();
}
});
}
latch.await();
return responses;
} finally {
httpClient.close();
}
}
}
Thanks for all the helpful responses. If anyone has any suggestions regarding optimization of the above code, I will be glad to hear so.