Develop Reference

Jersey Get Headers from Empty Response

I'm trying to update part of a legacy codebase I don't fully understand to work with a newer version of a REST API I also don't have access to the internals of. I do have a Swagger instance of it and can invoke it successfully via curl, but either Jersey is misbehaving or I'm misunderstanding how to read something.
If I execute the following curl command:
curl -v -k -X POST "[api endpoint]" -H "accept: application/json" -H "Authorization: Bearer [jwt token]" -H "Content-Type: multipart/form-data" -F "sender=[email address]" -F "recipient=[email address]" -F "fileType=file" -F "data=#[file]" -F "metaData=[other file]"
I get the following response:
> Content-Length: [#]
> Content-Type: multipart/form-data; boundary=------------------------9b1405ed70c2fd40
>
} [5 bytes data]
* We are completely uploaded and fine
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 201 Created
< Date: Wed, 26 Aug 2020 00:29:56 GMT
< Cache-Control: no-cache, no-store, must-revalidate
< Pragma: no-cache
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: JSESSIONID=[ID];Path=/;Secure
< Keep-Alive: timeout=600
< X-Frame-Options: SAMEORIGIN
< Server: WildFly
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, POST, DELETE, PUT
< Connection: Keep-Alive
< Access-Control-Allow-Headers: Content-Type
< Content-Type: application/json
< Location: [value I care about]
< Content-Length: 0
<
100 1667 0 0 100 1667 0 1825 --:--:-- --:--:-- --:--:-- 1825
* Connection #0 to host [proxy] left intact
This implies to me that POST responses which have empty bodies but nonempty headers are valid. However, when I try to effect the same thing via Jersey:
ClientConfig config = new ClientConfig();
config.register(MultiPartFeature.class);
Client client = ClientBuilder.newClient(config);
WebTarget endpoint = client.target([uri]);
FormDataMultiPart post = new FormDataMultiPart()
.field("sender", [email], MediaType.TEXT_PLAIN_TYPE)
.field("recipient", [email], MediaType.TEXT_PLAIN_TYPE)
.field("fileType", "file", MediaType.TEXT_PLAIN_TYPE)
.field("data", [InputStream], MediaType.APPLICATION_OCTET_STREAM_TYPE)
.field("metaData", [other InputStream], MediaType.APPLICATION_OCTET_STREAM_TYPE);
JsonNode response = endpoint.path([api path])
.request(MediaType.APPLICATION_JSON)
.header("Authorization", "Bearer " + [jwtTokenString])
.post(Entity.entity(post, MediaType.MULTIPART_FORM_DATA_TYPE), JsonNode.class);
I get a null response even though I know from inspecting server logs that the API command was received and processed successfully.
After a long time with the debugger I've determined that this is because Jersey will either return a null object or throw an exception if the data is null. (See here for more context. Though oddly enough I can't find the section they reference in any of the specification documents I can turn up via Google.)
This is probably fine as I'm not really interested in the empty body of the response, but I can't figure out how to get the HTTP headers returned as a result of my POST in Java.

You can't get the headers because you are using the method to only ask for the body as the response. What you want is the entire Response object. To get that, leave out the last argument (JsonNode.class) to the .post() method. That overloaded method you're using says that you don't care about anything but the body of the response. And that's what you will get.
When you use the overloaded post() method without the last body type parameter, the return type will be Response. You can get headers from it and the body.
Response response = endpoint.path([api path])
.request(MediaType.APPLICATION_JSON)
.header("Authorization", "Bearer " + [jwtTokenString])
.post(Entity.entity(post, MediaType.MULTIPART_FORM_DATA_TYPE));
URI locationUri = response.getLocation();
String locationHeader = response.getHeaderString("Location");
String body = response.readEntity(String.class);

Google HTTP Client Library for Java throws HttpResponseException: 301 Moved Permanently

I have a problem with Google HTTP Client Library for Java (1.22.0).
This is my code
String url = "http://gazetapraca.pl/ogl/2502758";
GenericUrl genericUrl = new GenericUrl(url);
ApacheHttpTransport apacheHttpTransport = new ApacheHttpTransport();
HttpRequest httpRequest = apacheHttpTransport.createRequestFactory().buildGetRequest(genericUrl);
httpRequest.setFollowRedirects(true);
HttpResponse httpResponse = httpRequest.execute();
and httpRequest.execute() throws
com.google.api.client.http.HttpResponseException: 301 Moved Permanently
Below is follow from Wireshark
GET /ogl/2502758 HTTP/1.1
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.22.0 (gzip)
Host: gazetapraca.pl
Connection: Keep-Alive
HTTP/1.1 301 Moved Permanently
Date: Sat, 26 Nov 2016 22:15:52 GMT
Server: Apache
Location: /ogl/2502758/pakowacz+-+mile+widziane+panie
Content-Length: 0
Set-Cookie: JSESSIONID_JOBS=2f1TffY6JYcb6zvBSrQ72fds7rfdsSnHM3sefw6D31Lfr434bnkDmdLQJLvLFZ6zkYBF!-12116034235597; path=/; HttpOnly
Content-Language: pl
P3P: CP="NOI DSP COR NID PSAo OUR IND"
Vary: User-Agent
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
GET /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie HTTP/1.1
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.22.0 (gzip)
Host: gazetapraca.pl
Connection: Keep-Alive
Cookie: JSESSIONID_JOBS=2f1TffY6JYcb6zvBSrQ72fds7rfdsSnHM3sefw6D31Lfr434bnkDmdLQJLvLFZ6zkYBF!-12116034235597
HTTP/1.1 301 Moved Permanently
Date: Sat, 26 Nov 2016 22:15:52 GMT
Server: Apache
Location: /ogl/2502758/pakowacz+-+mile+widziane+panie
Content-Length: 0
Content-Language: pl
P3P: CP="NOI DSP COR NID PSAo OUR IND"
Vary: User-Agent
Keep-Alive: timeout=2, max=99
Connection: Keep-Alive
and repeat a few times.
Maybe the problem is with url, because location is /ogl/2502758/pakowacz+-+mile+widziane+panie and next request method get is /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie.
In other software and library everything is working (google chrome browser, postman - addon to chrome, JSOUP - java library).
Does anyone have an idea how to solve the problem?

This is not your library's fault.
To understand why this problem is occurring, we must first understand the "error" message associated with your problem:
com.google.api.client.http.HttpResponseException: 301 Moved Permanently
So, what does this mean? Well, the last part of the error message, the description says "301 Moved Permanently". What that is referring to is an HTTP Status Code. An HTTP Status Code indicates what the outcome of a specific request is. In this case, the status code was 301, which according to RFC protocol means:
The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs.
So, this means that the URL that you are using is no longer valid, and that you have to use the new URL given to you by the Location response header. Now, it seems that the library that you're using is smart enough to detect this, and initializes a new request to the new URL. That's great and all, but your library that you are using, is incorrectly escaping the url provided by the Location header, and using that for the new request (turning /ogl/2502758/pakowacz+-+mile+widziane+panie into /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie), and the server receiving this request recognizes that those to paths are not the same (even though they should be. So, the server sends another 301 response, telling the client (the library in this case) to use the un-escaped URL instead of the escaped one, even though they should be the same.
Now, why is your library doing this? It turns out that, according to RFC spec, the '+' character is reserved for URIs. That means that that character, along with other characters are only intended for use in URIs for their intended purpose. Therefore, it is not standard to include the '+' character in URIs, unless it is used for a very specific purpose, which it looks like is not the case.
So, this all means that you cannot blame the library for this error, you can only blame the people who developed this site.
The reason that this works in your browser and other places is because those clients do not seem to be escaping the requested URL for you before sending it to the server.

how to get file name from the server request?

I'm trying http requests and create http get request to server. Server must return .zip file. here is the code:
url = new URL(urlToRead);
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
is = conn.getInputStream();
then I want to write it to file like this:
r = 1;
while(r > 0){
r = is.read(buf);
if(r > 0)
fos.write(buf, 0, r);
}
but to create fileoutputstream i want to use the file name provided by server. I've found that server answer has the file name and all the structure looks like this:
HTTP/1.1 200 OK
Date: Mon, 07 Apr 2003 14:51:19 GMT
Server: Apache/1.3.20 (Win32) PHP/4.3.0
Last-Modified: Mon, 07 Apr 2003 14:51:00 GMT
Accept-Ranges: bytes
Content-Length: 673
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: application/zip
Content-Disposition: attachment; filename=test.zip
Pragma: no-cache
....(zip content)
how to get filename?

You can use the HttpUrlConnection object's getHeaderField() method to read the Content-Disposition header. Also make sure you handle the case when it is not present.

Have you had a look at the methods that HttpURLConnection offers?
Especially the ones that involve headers?

if u have servletRequest object(lets say obj) then
i think this will help
obj.getRequestURL() ;

Incorrect Java HttpClient's response stream

In my application I need to parse a website and save some data from ir to the database. I am using HttpClient to get the page content. My code looks like this:
HttpClient client = new DefaultHttpClient();
System.out.println(doc.getUrl());
HttpGet contentGet= new HttpGet(siteUrl + personUrl);
HttpResponse response = client.execute(contentGet);
String html = convertStreamToString(response.getEntity().getContent());
/*
parse the page
*/
/***********************************************************************/
public static String convertStreamToString(InputStream is) throws Exception {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
return sb.toString();
}
I am doing this in a loop - I try to get content of some pages (their structure is the same). Sometimes it works fine, but unfortunately, my response in many cases is a sequence of similar trash liek this:
�=�v7���9�Hdz$�d7/�$�st��؎I��X^�$A6t_D���!gr�����C^��k#��MQ�2�d�8�]
I
I don't know where is the problem, please help me.
I have displayed headers of all responses that I got. For correct ones, there are:
Server : nginx/1.0.13
Date : Sat, 23 Mar 2013 21:50:31 GMT
Content-Type : text/html; charset=utf-8
Transfer-Encoding : chunked
Connection : close
Vary : Accept-Encoding
Expires : Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control : no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma : no-cache
Set-Cookie : pfSC=1; path=/; domain=.profeo.pl
Set-Cookie : pfSCvp=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.profeo.pl
For incorrect ones:
Server : nginx/1.2.4
Date : Sat, 23 Mar 2013 21:50:33 GMT
Content-Type : text/html
Transfer-Encoding : chunked
Connection : close
Set-Cookie : pfSCvp=3cff2422fd8f9b6e57e858d3883f4eaf; path=/; domain=.profeo.pl
Content-Encoding : gzip
Any other suggestions? My guess is that this gzip encoding is a problem here, but what can I do about it?

This probably has to do with some websites using a different character encoding in their response than your JVM default. To convert from a raw byte stream, like those provided by InputStreams, to a character stream (or a String), you have to choose a character encoding. HTTP responses can use different encodings, but they'll typically tell you what encoding they're using. You could do this manually by finding the "Content-Encoding" header of the HttpResponse, but your library provides a utility for doing this, since it's a common need. It's found in the EntityUtils class, and you can use it like so:
String html = EntityUtils.toString(response.getEntity());
You'll have to add
import org.apache.http.util.EntityUtils;
to the top of your file for that to work.
If that doesn't help, another possibility is that some of the URLs you're retrieving are binary, not textual, in which case the things you're trying to do don't make sense. If that's the case, you can possibly try to distinguish between the textual responses and the binary responses by checking Content-Type header, like so:
boolean isTextual = response.getFirstHeader("Content-Type").getValue().startsWith("text");
NEW MATERIAL:
After looking at the HTTP headers you added to your question, my best guess is that this is being caused by gzip compression of the responses. You can find more info on how to deal with that in this question, but the short version is that you should try using ContentEncodingHttpClient instead of DefaultHttpClient.
Another edit: ContentEncodingHttpClient is now deprecated, and you're supposed to use DecompressingHttpClient instead.

You need a httpclient which don't use compression.
I use this HttpClientBuilder.create().disableContentCompression().build() httpclient

Counting Bytes sent/received over HTTP

In Java,
How could the bytes sent and received over an active HTTP connection be counted?
I want to display some statistics like:
Bytes Sent : xxxx Kb
Bytes Received : xxxx Kb
Duration : hh:mm

It is difficult to see how you could decorate HttpConnection to count the raw byte data. You could reimplement HTTP using sockets, but this would have to be pretty important to go to those lengths.
Sample response from stackoverflow.com:
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Expires: Mon, 21 Sep 2009 11:46:48 GMT
Vary: Accept-Encoding
Server: Microsoft-IIS/7.0
Date: Mon, 21 Sep 2009 11:46:48 GMT
Content-Length: 19452
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>...remaining data....
HTTP requests are reasonably simple, so you might try reconstructing it using the information that is exposed. Something of the form:
// TODO: edge cases, error handling, header delimiter, etc.
int byteCount = 0;
int headerIndex = 0;
while (true) {
String key = httpConnection.getHeaderFieldKey(headerIndex);
if (key == null)
break;
String value = httpConnection.getHeaderField(headerIndex++);
byteCount += key.getBytes("US-ASCII").length
+ value.getBytes("US-ASCII").length + 2;
}
byteCount += httpConnection.getHeaderFieldInt("Content-Length",
Integer.MIN_VALUE);
That implementation is incomplete and untested. You'd need to study up on the details of the HTTP protocol to ensure the accuracy of your results.

Wrap the HTTPURLConnection's getInputStream() and getOutputStream() in your own streams that count the bytes that go through them. Or even better use the Apache Commons IO library - they have counting stream implementations in there.