Develop Reference

Jetty for Junit - java.net.ConnectException: Connection refused: no further information

TL;DR Am using org.mortbay.jetty:jetty:6.1.26 in my JUnits to Stub a server. I was hoping to get predetermined content in output. However am getting exception java.net.ConnectException: Connection refused: no further information. When I hit the endpoint from web browser, I do not get response immediately but after all JUnits are completed and Jetty shut the server (I get expected response in browser as well as Socket closed exception in app logs). I searched few stackoverflow posts but couldn't progress much and hence need help
Details: Application code we have is (successfully) getting required information from external clients by reading that data, parsing it and finally saving that data. Below is how code looks like
HTTPUtilities.performGet(uri)
.thenApply(HTTPUtilities::httpResponseToReader)
.thenApply(reader -> parseContent(reader, id))
.thenAccept(idDetailsList -> idRepository.saveAll(idDetailsList))
.whenComplete((resp, err) -> {
if (err == null) {
log.info("Data for {} is successfully saved", id);
} else {
log.error("Exception encountered for URI {} / id : {}. Message: -> {} ", uri, id, err.getMessage());
}
});
My intent is to Stub first method of above CompletableFuture which is HTTPUtilities.performGet(uri). It's a static method and its code looks like below
HttpUriRequest request = RequestBuilder.get()
.setUri(uri)
.setHeader(HttpHeaders.ACCEPT, HEADER_ACCEPT_CONSTANT)
.setHeader(HttpHeaders.USER_AGENT, HEADER_USER_AGENT_CONSTANT)
.build();
CompletableFuture<HttpResponse> resp = new CompletableFuture<>();
executorService.execute(() -> {
try (CloseableHttpClient httpclient = HttpClients.custom().setDefaultRequestConfig(config).build();
CloseableHttpResponse response = httpclient.execute(request)) {
resp.complete(response);
log.info("Successfully returning response for URL: {}", uri);
} catch (NullPointerException ex) {
log.error("NullPointerException encountered when performing GET against URL {} with message {}", uri, ex.getCause());
resp.completeExceptionally(ex);
} catch (IOException ex) {
log.error("IOException encountered when performing GET against URL {} with message {}", uri, ex.getCause());
resp.completeExceptionally(ex);
}
});
return resp;
This code works fine for actual calls. However, it fails with below stack trace when run from JUnits where HTTP calls are stubbed using Jetty. Below stacktrace is generated when code reaches CloseableHttpResponse response = httpclient.execute(request) line in above call.
java.net.ConnectException: Connection refused: no further information
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
Below is code from JUnit Test class
#ExtendWith(MockitoExtension.class)
public class IdRetreieveAndSaveTest {
#InjectMocks
UpdateIdData updateIdData;
#BeforeAll
static void initial() throws Exception {
//Stub to replace actual call
Server server = new Server(stubbedPort);
Context contentOkContext = new Context(server, stubbedContext);
contentOkContext.setHandler(new HttpCustomHandler());
server.setStopAtShutdown(true);
server.start();
}
#Test
void updateIdDataTest() {
List<IdData> resp = new ArrayList<>();
updateIdData.invokeHttpAndSave(id);
assertEquals(100, resp.size());
}
}
and finally helper class for JUnit
public class HttpCustomHandler extends AbstractHandler {
#Override
public void handle(String s, HttpServletRequest httpServletRequest, HttpServletResponse response, int i) throws IOException {
OutputStream out = response.getOutputStream();
try (ByteArrayISO8859Writer writer = new ByteArrayISO8859Writer()) {
writer.write(codeThatReturnsSampleDataFromExternalClient());
writer.flush();
response.setIntHeader(HttpHeaders.CONTENT_LENGTH, writer.size());
writer.writeTo(out);
out.flush();
};
}
}
I verified that port is up and listening during Junit execution. Below is response when at breakpoint during JUnit execution in debug mode and these lines disappear for same command once JUnit test execution completes
netstat -ano | Select-String -pattern "65500"
TCP 0.0.0.0:65500 0.0.0.0:0 LISTENING 27448
TCP [::]:65500 [::]:0 LISTENING 27448
Thank you in advance

How to wait for the HTTP request to get completed in java

I am writing Java code where i am downloading the file from a server and i have to copy the file in my local system when the file download is complete.
I am using the below code:-
OkHttpClient.Builder builder = new OkHttpClient.Builder();
OkHttpClient client = builder.readTimeout(600, TimeUnit.SECONDS).writeTimeout(600, TimeUnit.SECONDS)
.connectTimeout(600, TimeUnit.SECONDS).build();
Request downloadRequest = new Request.Builder().url(url + fileName).addHeader("cache-control", "no-cache")
.addHeader("Authorization", token).build();
try {
Response downloadResponse = client.newCall(downloadRequest).execute();
System.out.println(downloadResponse.message());
System.out.println("got response from blob " + downloadResponse.isSuccessful() + " " + fileName);
return downloadResponse;
} catch (IOException e1) {
e1.printStackTrace();
}
return null;
}
But the request is made asynchronously and before the request is completed then response is returned which is incomplete. Can anyone please help me how can i make a request and wait till the response is completed.
Any help is highly appreciated!

Looks like you're returning the response object (not the response body content).
try something like:
return downloadedResponse.body().string()
My experience with HttpClient is such that the headers return first. The content doesn't necessarily come across the wire unless/until you consume it.

To make a synchronous GET request we need to build a Request object based on a URL and make a Call. After its execution we get back an instance of Response:
#Test
public void whenGetRequest_thenCorrect() throws IOException {
Request request = new Request.Builder()
.url(BASE_URL + "/date")
.build();
Call call = client.newCall(request);
Response response = call.execute();
assertThat(response.code(), equalTo(200));
}

You are already using synchronous method calling.
client.newCall(downloadRequest).execute();
This is a synchronous way of requesting URL. If you want to do the aysynchronous call you need to use enqueue method of Call class.
call.enqueue(new Callback() {
public void onResponse(Call call, Response response)
throws IOException {
// ...
}
public void onFailure(Call call, IOException e) {
fail();
}
});
I think problem is somewhere else. Kindly give more details why you are suspecting the current one as an asynchronous call so that we can do RCA.

AsyncInvoker not releasing the Thread

I am using AsyncInvoker using Jersey 2.0. This works fine for me. However, thread is not ending after completion of the return. Please note that I don't expect any response for the services I call.
public Future<Response> fire(WebTarget webTarget) {
Future<Response> response = null;
try {
response = webTarget.request(MediaType.APPLICATION_JSON_TYPE)
.accept(MediaType.APPLICATION_JSON_TYPE)
.headers(headers)
.async().get();
}
catch (Exception e) {
e.printStackTrace();
}
return response;
}

As long as you don't do anything with the actual javax.ws.rs.core.Response that is provided to you once the future value resolves, the request response stream is kept open (and the thread associated with it the raw HTTP request as wel I guess). You should either:
Do something with the javax.ws.rs.core.Response object and close it (or it's stream).
Use .get(MyPojo.class) to have the response steam converted into an actual object. This will close the stream for you and resolve a MyPojo instance in the future instead.

You need to close your client object that you created in the calling method. In the calling method you would have something like below -
Client client = ClientBuilder.newClient();
WebTarget webTarget = client.target(SERVER_URL).path(API_PATH).path(String.valueOf(id));
fire(webTarget);
So you need to close your client after calling this method -
client.close()
However, the recommended way of closing client is after receiving response. Something like below -
public void fire(WebTarget webTarget) {
try {
webTarget.request(MediaType.APPLICATION_JSON_TYPE)
.accept(MediaType.APPLICATION_JSON_TYPE)
.headers(headers)
.async().get(new InvocationCallback<Response>() {
#Override
public void completed(Response response) {
// Do whatever your wish with response
client.close();
}
#Override
public void failed(Throwable throwable) {
throwable.printStackTrace();
client.close();
}
});
}
catch (Exception e) {
e.printStackTrace();
}
}

Handling multiple threads/outgoing http requests

I am working on a Java/Spring web application that for each incoming request does the following:
fires off a number of requests to third party web servers,
retrieves the response from each,
parses each response into a list of JSON objects,
collates the lists of JSON objects into a single list and returns it.
I am creating a separate thread for each request sent to the third party web servers. I am using the Apache PoolingClientConnectionManager. Here is an outline of the code I am using:
public class Background {
static class CallableThread implements Callable<ArrayList<JSONObject>> {
private HttpClient httpClient;
private HttpGet httpGet;
public CallableThread(HttpClient httpClient, HttpGet httpGet) {
this.httpClient = httpClient;
this.httpGet = httpGet;
}
#Override
public ArrayList<JSONObject> call() throws Exception {
HttpResponse response = httpClient.execute(httpGet);
return parseResponse(response);
}
private ArrayList<JSONObject> parseResponse(HttpResponse response) {
ArrayList<JSONObject> list = null;
// details omitted
return list;
}
}
public ArrayList<JSONObject> getData(List<String> urlList, PoolingClientConnectionManager connManager) {
ArrayList<JSONObject> jsonObjectsList = null;
int numThreads = urlList.size();
ExecutorService executor = Executors.newFixedThreadPool(numThreads);
List<Future<ArrayList<JSONObject>>> list = new ArrayList<Future<ArrayList<JSONObject>>>();
HttpClient httpClient = new DefaultHttpClient(connManager);
for (String url : urlList) {
HttpGet httpGet = new HttpGet(url);
CallableThread worker = new CallableThread(httpClient, httpGet);
Future<ArrayList<JSONObject>> submit = executor.submit(worker);
list.add(submit);
}
for (Future<ArrayList<JSONObject>> future : list) {
try {
if (future != null) {
if (jsonObjectsList == null) {
jsonObjectsList = future.get();
} else {
if (future.get() != null) {
jsonObjectsList.addAll(future.get());
}
}
}
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
executor.shutdown();
return jsonObjectsList;
}
}
This all works fine. My question is in relation to how well this code will scale as the traffic to my website increases? Is there a better way to implement this? For example, by implementing non-blocking I/O to reduce the number of threads being created. Are there libraries or frameworks that might help?
At the moment, I am using Java 6 and Spring Framework 3.1
Thanks in advance

I wouldn't recommend to implement this as a synchronous service. Do it asynchronously. Get your request, pool the callables, and return a resource location where the client can later request the result.
You've got to be pooling this callables in an executor. Poll the executor in a background process and make avalable the results in the location you returned at the first request. Doing it this way, it would be easier to control your available resuources, and deny cleanly a processing requests if there aren't any more resources available.
Non blocking IO won't reduce the number of threads, it just delegates the "job" to another thread, in order for the service thread not to be blocked and to be able to receive more requests.
use REST.
Receive a POST request, and answer with something like this:
HTTP/1.1 202 Accepted
Location: /result/to/consult/later
The client can then request the resutl at the given location. If the processing has not finished, then answer with:
HTTP/1.1 201 Created
If its done then return a HTTP/1.1 200 OK with the resulting JSON.

Handling Exception on Asynchronous Webservice Request

I am still new to implement web service request using Play!Framework 2.1 WS library. Now, I have problem on understanding the WS library behaviour.
Firstly, I have code like this :
public static Result espnRss() {
try {
// do request
return async(
WS.url("http://espnfc.com/rss/news?section=premierleague").get().map(
new F.Function<WS.Response, Result>() {
#Override
public Result apply(WS.Response response) throws Throwable {
return ok("Success!"); // success request
}
}
)
);
} catch (Exception e) {
// exception occured
return internalServerError("Oops, connect exception occured!");
}
}
When I try to request the espnRss action, I got SUCCESS response.
Then, I want to set WS timeout on the request. So, I changed my previous code like this :
public static Result espnRss() {
try {
// set request timeout for 1000 ms and do request
return async(
WS.url("http://espnfc.com/rss/news?section=premierleague").setTimeout(1000).get().map(
... // same as previous
)
);
} catch (Exception e) {
// exception occured
return internalServerError("Oops, connect exception occured!");
}
}
My internet connection is not fast (Download speed is about 40 KB/s) and I do that on purpose (set request time out for 1 second) to make exception handling code is executed.
But, I get default response from framework, not internalServerError response the code provided.
Execution Exception
[TimeoutException: No response received after 1000]
Can anyone explain me why the exception on WS request cannot be caught using my code above? How is the best way to handle exception using Play!Framework WS library?

To handle exception that occur on asynchronous request such as WS request with Play!Framework 2.1.0, there is method on Promise named recover(F.Function<java.lang.Throwable,A> function).
The method should be called when we want to handle all exception occured while requesting using WS library. So, I solved the problem using code that looked like following:
public static Result espnRss() {
// do request
return async(
WS.url("http://espnfc.com/rss/news?section=premierleague").setTimeout(100).get().map(
new F.Function<WS.Response, Result>() {
#Override
public Result apply(WS.Response response) throws Throwable {
return ok("Success!"); // success request
}
}
).recover( // to handle error occured on redeemed PROMISE
new F.Function<Throwable, Result>() {
#Override
public Result apply(Throwable throwable) throws Throwable {
// option to distinguish exception
if (throwable instanceof TimeoutException) {
return internalServerError("Oops, time out exception occured!");
} else {
return internalServerError("Oops, other exception occured!");
}
}
}
)
);
}

I am not familiar with the Play framework but async must be returning/using some kind of future. The request is actually performed in a separate Thread which Exceptions are obviously not caught by your try..catch handler.
There must be some function/method like onComplete that you can apply to async allowing you to test the result of running the request.