Im executing a http request like this:
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpGet httpRequest = new HttpGet("http://localhost:8080/test");
HttpResponse httpResponse = httpClient.execute(httpRequest);
What is the best way to log out everything about the request/response? I mean like what method was used for request and what url and all the headers and response code and response message body and so on.
Edit:
At the moment i worked out something like this:
private static void logHttpRequest(HttpMessage httpMessage) {
StringBuilder builder = new StringBuilder();
addHeaders(httpMessage, builder);
builder.append(httpMessage.toString() + "\n");
System.out.println("Request:\n" + builder.toString());
}
private static void logHttpResponse(HttpResponse httpResponse) throws IOException {
StringBuilder builder = new StringBuilder();
builder.append(httpResponse.getStatusLine() + "\n");
addHeaders(httpResponse, builder);
builder.append(EntityUtils.toString(httpResponse.getEntity()));
System.out.println("Response:\n" + builder.toString() + "\n");
}
private static void addHeaders(HttpMessage httpMessage, StringBuilder builder) {
Header[] headers = httpMessage.getAllHeaders();
if (headers != null) {
for (Header header : headers) {
builder.append(header.toString() + "\n");
}
}
}
Is there anything else that should be logged? Can anything be done better/more optimal?
I think this primarily depends on context and your requirements. Keep in mind though, that huge amounts of logging is also not smart. For example, you should log exceptions at a high log level, eg. SEVERE or ERROR (depends on the logging framework you're using). However, messages like, "trying to make a GET request to http://soAndSo/whatevz" could do with lower log level of, say DEBUG, so you could switch your code to log DEBUG level messages when something bad happens in production.
Something I have typically seen people do with the response is that they definitely log ERROR if there's no response at all but when there is a response, it's imperative to check the http status code of the response and if it's not what you expect it to be (typically 200/2xx), you log the response body, headers, etc at ERROR level.
When making POST or PUT requests with some body, try and see if it's useful to log the request body and headers if any at, say, INFO or DEBUG level. These things sometimes help in tracing out failure scenarios or lost requests/lost data.
At the consuming end too, your system should ideally have a monitor on the no.of/rate of ERROR level messages otherwise you'll come to know of something very bad only after it happens, which is not favourable.
Related
I am calling a File Server's REST API using POST method and sending it file content to be uploaded. The REST API should ideally save the file and send a response which contains fileName.
My code is something like this.
public String uploadFile() {
UploadResponse response = restTemplate.postForObject(
FILE_SERVER_URL/upload,
new HttpEntity<>(fileContent, headers),
UploadResponse.class);
return response.getFileName();
}
In the above code, the compiler complains that UploadResponse response could be null, and I should handle that.
I plan to handle it with the below code.
public String uploadFile() {
UploadResponse response = restTemplate.postForObject(
FILE_SERVER_URL/upload,
new HttpEntity<>(fileContent, headers),
UploadResponse.class);
if(response != null) {
return response.getFileServiceId();
}
else {
throw new RuntimeException("File upload failed");
}
}
However, I am not sure if it is the right way to handle this. I don't feel this is a Runtime Exception. Please guide me as how should I handle the case that response could be null.
You can use Optional to avoid null checks in your code.
You can read this really insightful Q/A here - https://softwareengineering.stackexchange.com/questions/309134/why-is-using-an-optional-preferential-to-null-checking-the-variable
Before writing something like "why don't you use Java HTTP client such as apache, etc", I need you to know that the reason is SSL. I wish I could, they are very convenient, but I can't.
None of the available HTTP clients support GOST cipher suite, and I get handshake exception all the time. The ones which do support the suite, doesn't support SNI (they are also proprietary) - I'm returned with a wrong cert and get handshake exception over and over again.
The only solution was to configure openssl (with gost engine) and curl and finally execute the command with Java.
Having said that, I wrote a simple snippet for executing a command and getting input stream response:
public static InputStream executeCurlCommand(String finalCurlCommand) throws IOException
{
return Runtime.getRuntime().exec(finalCurlCommand).getInputStream();
}
Additionally, I can convert the returned IS to a string like that:
public static String convertResponseToString(InputStream isToConvertToString) throws IOException
{
StringWriter writer = new StringWriter();
IOUtils.copy(isToConvertToString, writer, "UTF-8");
return writer.toString();
}
However, I can't see a pattern according to which I could get a good response or a desired response header:
Here's what I mean
After executing a command (with -i flag), there might be lots and lots of information like in the screen below:
At first, I thought that I could just split it with '\n', but the thing is that a required response's header or a response itself may not satisfy the criteria (prettified JSON or long redirect URL break the rule).
Also, the static line GOST engine already loaded is a bit annoying (but I hope that I'll be able to get rid of it and nothing unrelated info like that will emerge)
I do believe that there's a pattern which I can use.
For now I can only do that:
public static String getLocationRedirectHeaderValue(String curlResponse)
{
String locationHeaderValue = curlResponse.substring(curlResponse.indexOf("Location: "));
locationHeaderValue = locationHeaderValue.substring(0, locationHeaderValue.indexOf("\n")).replace("Location: ", "");
return locationHeaderValue;
}
Which is not nice, obviosuly
Thanks in advance.
Instead of reading the whole result as a single string you might want to consider reading it line by line using a scanner.
Then keep a few status variables around. The main task would be to separate header from body. In the body you might have a payload you want to treat differently (e.g. use GSON to make a JSON object).
The nice thing: Header and Body are separated by an empty line. So your code would be along these lines:
boolean inHeader = true;
StringBuilder b = new StringBuilder;
String lastLine = "";
// Technically you would need Multimap
Map<String,String> headers = new HashMap<>();
Scanner scanner = new Scanner(yourInputStream);
while scanner.hasNextLine() {
String line = scanner.nextLine();
if (line.length() == 0) {
inHeader = false;
} else {
if (inHeader) {
// if line starts with space it is
// continuation of previous header
treatHeader(line, lastLine);
} else {
b.append(line);
b.appen(System.lineSeparator());
}
}
}
String body = b.toString();
I'm trying to obtain data from Botify's REST API to use it inside a project, which is also a REST API. I'm using an instance of Spring's RestTemplate class to make the actual requests to Botify, specifically the .exchange method as I need to pass Botify's key as a header parameter.
My problem comes when I need to call to a method of the endpoint which takes a URL as a part of the request's URI (not a parameter). Documentation of this endpoint is in https://developers.botify.com/api/reference/#!/Analysis/getUrlDetail
Basically the structure of the requests is like this:
/analyses/{username}/{project_slug}/{analysis_slug}/urls/{url}
The last part of that URI is a URL address, which needs to be encoded in UTF-8 to make it possible to separate it from the actual request.
The problem is (I believe) that the .exchange method always encodes the request, so what I try to send like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com
...ends up like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%253A%252F%252Fwww.example.com'
Which obviously doesn't work. This is an excerpt from the method that makes the call to Botify:
public String callBotifyEndpoint(String reportType, String parameters) throws UnsupportedEncodingException {
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com"
HttpHeaders headers = new HttpHeaders();
headers.set("Authorization", "Token " + this.apiKey);
HttpEntity<String> entity = new HttpEntity<>(headers);
UriComponentsBuilder botifyQueryBuilder = UriComponentsBuilder.fromUriString(request);
String queryStringBuild = botifyQueryBuilder.build(true).toUriString();
String botifyResult = null;
try {
System.out.println("Calling Botify API: " + queryStringBuild);
ResponseEntity<String> response = botifyTemplate.exchange(queryStringBuild, HttpMethod.GET, entity, String.class);
if(response.hasBody()) {
botifyResult = response.getBody();
}
} catch(RestClientException ex) {
ex.printStackTrace();
}
try {
} catch (Exception e) {
// TODO: handle exception
}
return botifyResult;
}
In this line:
botifyQueryBuilder.build(true).toUriString();
The "true" parameter indicates whether the data is already encoded or not. I've tried to disable it but the result is the same.
I've removed actual request generation process (along with my user and project's name) to simplify things, but this should return a response from Botify with the existing data for that URL.
Instead, it returns a 400 bad request error (which makes sense, because the URL is not correct).
I'm feeling like this may be a bug in RestTemplate's .exchange method, but maybe I'm not using it properly. Any suggestions?
Don't encode prematurly as you do here:
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com";
Use parameter placeholders feature in RestTemplate instead of text concatenation.
Refer to:
Spring RestTemplate GET with parameters
I am writing a Solr custom analyzer to post a value of a field to Apache Stanbol for enhancement during indexing phase.
In my custom analyzer's incrementToken() method I have below code. I'm posting the value of the token to Stanbol enhancer endpoint using a Jersey REST client. Instead of the expected enhacement result I always get a HTTP 500 error response when running the analyzer.
But the same REST client logic works when executing it in a Java application main method.
Can someone please help me identify where the problem is? Could it be a Java permission problem, invoking a web endpoint within the Solr analyzer?
public boolean incrementToken() throws IOException {
if (!input.incrementToken()) {
return false;
}
char[] buffer = charTermAttr.buffer();
String content = new String(buffer);
Client client = Client.create();
WebResource webResource = client.resource("http://localhost:8080/enhancer");
ClientResponse response = webResource.type("text/plain").accept(new MediaType("application", "rdf+xml")).post(ClientResponse.class, content);
int status = response.getStatus();
if (status != 200 && status != 201 && status != 202) {
throw new RuntimeException("Failed : HTTP error code : "
+ response.getStatus());
}
String output = response.getEntity(String.class);
System.out.println(output);
charTermAttr.setEmpty();
char[] newBuffer = output.toCharArray();
charTermAttr.copyBuffer(newBuffer, 0, newBuffer.length);
return true;
}
This seems to be a weird intermittent issue when I use the Solr Analysis UI (http://localhost:8983/solr/#/collection1/analysis) for testing my Analyzer.
It works fine when I hard code the input value in the Analyzer and index. I gave the same input : "Tim Bernes Lee is a professor at MIT" hard coded in the Analyzer class and from the Solr Analysis UI. The UI response failed intermittently when I adjust the field value.
This could be a problem with character encoding of the field value it seems.
My goal is to rapidly make posts to a server from appengine(java). I am attempting to do this using UrlFetchService.fetchAsync. I have been basing my code after this blog post. I have been able to make the request using the code below, however I get some strange behavior:
private void futureRequests() {
URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();
URL url = new URL("https://someserver.com");
FetchOptions fetchOptions = FetchOptions.Builder.withDefaults();
fetchOptions.doNotValidateCertificate();
fetchOptions.setDeadline(60D);
ArrayList<Future<HTTPResponse>> asyncResponses = new ArrayList<Future<HTTPResponse>>();
for (int i = 0; i < postDatas.size(); i++) {
HTTPRequest request = new HTTPRequest(url, HTTPMethod.POST, fetchOptions);
request.setPayload(postDatas.get(i).getBytes(UTF8));
HTTPHeader header = new HTTPHeader("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8");
request.setHeader(header);
header = new HTTPHeader("Content-Length", Integer.toString(postDatas.get(i).getBytes().length));
request.setHeader(header);
header = new HTTPHeader("Authorization", "auth=" + authToken);
request.setHeader(header);
Future<HTTPResponse> responseFuture = fetcher.fetchAsync(request);
asyncResponses.add(responseFuture);
}
for (Future<HTTPResponse> future : asyncResponses) {
HTTPResponse response;
try {
response = future.get();
int responseCode = response.getResponseCode();
resp.getWriter().println("response: " + responseCode);
logger.warning("Response: " + responseCode);
} catch (Exception e) {
}
}
}
The strange behavior is that I get duplicate posts on the server, and according to my appstats page I use 10x-20x more urlFetches than what was added with the code above. Below is my appstats screen:
There are more urlFetch calls that could not fit on the screen. It appears that the requests are still completing in a synchronous fashion(circled items), but there are many urlFetches that appear to go on at the same time. My question is how am I getting all this calls to urlFetch when I only had 14 Future ?? Could the server be giving an error or 503 and urlFetch retrying until it goes through? And how can I be getting 2 posts for each request??
I understand that I could use the task queue to do asyc request, however I am dealing with a relatively low number of request(20-100) and the cold start time of ramping up another instance would probably make this not a good option for my situation. Can anyone explain this behavior or have experience with this?
This was simply a mistake in my code that was causing my app to make more request than I thought..