Develop Reference

Empty response with jersey 2 client

I'm using Jersey 2.16 client for fetching files,
some of the files are coming out empty when I try to parse the response.
For example, while trying to fetch URL:
https://s1.yimg.com/uu/api/res/1.2/3LJG5Qp6cO9WVZ644ybK1A--/YXBwaWQ9eXRhY2h5b247aD0xNjQ7dz0yOTA7/https://ibdp.videovore.com/video/61260788?size=512x288
The response status is 200, I see the content-length header stating there should be 9081 bytes, but the very first call to inputStream.read returns -1.
Following is the code that downloads the data:
private ByteArrayOutputStream downloadFile(Response response) {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream(1024);
try {
InputStream inputStream = response.readEntity(InputStream.class);
byte[] bytes = new byte[1024];
int readBytes = inputStream.read(bytes); // for the given URL this returns -1
while (readBytes > 0) {
outputStream.write(bytes, 0, readBytes);
readBytes = inputStream.read(bytes);
}
} catch (Exception e) {
e.printStackTrace();
}
return outputStream;
}
The response headers I get:
Server=ATS
Public-Key-Pins-Report-Only=max-age=2592000; pin-sha256="2fRAUXyxl4A1/XHrKNBmc8bTkzA7y4FB/GLJuNAzCqY="; pin-sha256="I/Lt/z7ekCWanjD0Cvj5EqXls2lOaThEA0H2Bg4BT/o="; pin-sha256="Wd8xe/qfTwq3ylFNd3IpaqLHZbh2ZNCLluVzmeNkcpw="; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="i7WTqTvh0OioIruIfFR4kMPnBqrS2rdiVPl/s2uC/CY="; pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="; pin-sha256="uUwZgwDOxcBXrQcntwu+kYFpkiVkOaezL0WYEZ3anJc="; pin-sha256="dolnbtzEBnELx/9lOEQ22e6OZO/QNb6VSSX2XHA3E7A="; includeSubdomains; report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-hpkp-report-only"
Last-Modified=Sun, 30 Dec 2018 19:10:17 GMT
P3P=policyref="https://policies.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV"
Referrer-Policy=no-referrer-when-downgrade
Strict-Transport-Security=max-age=15552000
X-Server-Processor=ymagine
X-XSS-Protection=1; mode=block
Content-Length=9081
Age=11549
Content-Type=image/jpeg
X-Content-Type-Options=nosniff
Connection=keep-alive
X-Server-Time-FetchImage=89603
X-Server-Time-Process=3800
Date=Mon, 07 Jan 2019 08:36:25 GMT
Via=http/1.1 e30.ycpi.lob.yahoo.com (ApacheTrafficServer [cRs f ])
Cache-Control=public, max-age=86400
ETag="5c291819-6ec1"
Content-Disposition=inline; filename=61260788?size=512x288.jpg
X-Image-Height=163
X-Image-Width=290
X-Server-Time-Total=93975
Expect-CT=max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only"

JSON Url returns something else?

I am trying to access this json file : http://www.cloudpricingcalculator.appspot.com/static/data/pricelist.json with Java.
But when I read it, sometimes it gives me a JSON string (that's ok) and sometimes it gives me something else and json.simple.parser throw an Unexpected character(<) at position 0.
Based on what I read on stackOverflow, it may be that it returns XML instead of JSON. As my url is "json", how is it possible ?
Here is the code i'm using :
String baseUrl = "http://www.cloudpricingcalculator.appspot.com/static/data/pricelist.json";
...
URL url = new URL(this.baseUrl);
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String l;
String json = "";
System.out.println(url);
while((l=in.readLine()) != null){
System.out.println(l);
json+=l;
}
JSONParser parser = new JSONParser();
JSONObject jsonObject = (JSONObject) parser.parse(json);
and the log is <followed by an lot of squares and unknown characters like ÿÕ[s›È and an error Unexpected character () at position 0.

You are not taking into account the compression and encoding of the resource returned from the server. The response of a HEAD request is the following:
rpax#machine:~$ HEAD http://www.cloudpricingcalculator.appspot.com/static/data/pricelist.json
200 OK
Cache-Control: public, max-age=600
Connection: close
Date: Mon, 21 Aug 2017 12:02:14 GMT
Age: 112
ETag: "n_s_jQ"
Server: Google Frontend
Content-Encoding: gzip <---- *HERE*
Content-Length: 7902
Content-Type: application/json
Expires: Mon, 21 Aug 2017 12:12:14 GMT
...
For avoiding this issue, you can wrap the url stream into a GZIPInputStream :
GZIPInputStream gis = new GZIPInputStream(url.openStream());
BufferedReader in = new BufferedReader(new InputStreamReader(gis));
// ...
And the data returned when executing readline() will be decompressed.

Java downloading a file and retrieving current download completion

In a program I have that auto-updates, I need it to download a specific file (working already) and it does so with the following code:
public static void getFile() {
try {
URL url = new URL("https://dl.dropboxusercontent.com/s/tc301v61zt0v5cd/texture_pack.png?dl=1");
InputStream in = new BufferedInputStream(url.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n = 0;
while (-1 != (n = in.read(buf))) {
out.write(buf, 0, n);
}
out.close();
in.close();
byte[] response = out.toByteArray();
FileOutputStream fos = new FileOutputStream(file4);
fos.write(response);
fos.close();
} catch (Exception e) {
JOptionPane.showMessageDialog(null, "Check your internet connection, then try again.\nOr re-install the program.\nError Message 7", "Could Not Download The Required Resources.", JOptionPane.NO_OPTION);
e.printStackTrace();
System.exit(0);
}
}
How would I implement a way to get the current completion of the download (like the percent it's downloaded) and put it into an integer to be printed out in the console (just for developer testing). Also what sort of equation could i use to get the estimated amount of time left on the download? Anything would help! Thanks.

If you examine the HTTP headers sent in the file download, you'll discover the file size. From this you can calculate percentage complete:
curl --head "https://dl.dropboxusercontent.com/s/tc301v61zt0v5cd/texture_pack.png?dl=1"
Gives you this:
HTTP/1.1 200 OK
accept-ranges: bytes
cache-control: max-age=0
content-disposition: attachment; filename="texture_pack.png"
Content-length: 29187
Content-Type: image/png
Date: Mon, 28 Apr 2014 22:38:34 GMT
etag: 121936d
pragma: public
Server: nginx
x-dropbox-request-id: 1948ddaaa2df2bdf2c4a2ce3fdbeb349
X-RequestId: 4d9ce90907637e06728713be03e6815d
x-server-response-time: 514
Connection: keep-alive
You may have to use something more advanced than standard Java library classes for your download to access the headers however, something like Apache HttpClient.

There is no way to get the size of a streamed file without streaming to the end of the file, so it can't be done within that block of code.
If you are willing to adopt the Dropbox API, you can use that to get the size of the file before starting the download. Then you can work with that size and the number bytes downloaded (n in your code) to achieve what you need.
The class in the Dropbox API that you need is DbxEntry.File.

how to decompress gzipped contents in JAVA [duplicate]

This question already has answers here:
How to decompress a gzipped data in a byte array?
(5 answers)
Closed 9 years ago.
In HTTP request and response, content-encoding is 'gzip' and content is gzipped. Is there a way to decompress the gzipped content so we can see the contents ??
Example for GZipped HTTP request
HTTP/1.1 200 OK
Date: mon, 15 Jul 2014 22:38:34 GMT
Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux)
Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
Accept-Ranges: bytes
Content-Length: 438
Connection: close
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip
//Response body with non type characters

That shouldn't be necessary. Your application server should handle such requests for you and decompress the payload for you automatically.
If that doesn't happen, you need to wrap the InputStream in a GZipInputStream. But this sounds more like a misconfiguration of your server.

Found the answer.
//reply - Response from Http byte[] reply = getResponseBytes();
int i;
for(i=0;i<reply.length;i++){
//Finding Raw Response by two new line bytes
if(reply[i]==13){
if(reply[i+1]==10){
if(reply[i+2]==13){
if(reply[i+3]==10){
break;
}
}
}
}
}
//Creating new Bytes to parse it in GZIPInputStream
byte[] newb = new byte[4096];
int y=0;
for(int st=i+4;st<reply.length;st++){
newb[y++]=reply[st];
}
GZIPInputStream gzip = new GZIPInputStream (new ByteArrayInputStream (newb));
InputStreamReader reader = new InputStreamReader(gzip);
BufferedReader in = new BufferedReader(reader);
String readed;
while ((readed = in.readLine()) != null) {
//Bingo...
System.out.println(readed);
}

Decompressing a gzipped http response

Hello fellow java developers. I receive a response with headers and body as below, but when I try to decompress it using the code below, it fails with this exception:
java.io.IOException: Not in GZIP format
Response:
HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Content-Encoding: gzip
Server: Jetty(6.1.x)
▼ ═UMs¢0►=7┐ép?╙6-C╚$╢gΩ↓╟±╪₧∟zS╨╓╓♦$FÆ╒÷▀G┬╚╞8N≤╤Cf°►╦█╖╗o↨æJÄ+`:↓2
♣»└√S▬L&?∙┬_)U╔|♣%ûíyk_à\,æ] hⁿ?▀xΓ∟o╜4♫ù\#MAHG?┤(Q¶╞⌡▌Ç?▼ô[7Fí¼↔φ☻I%╓╣Z♂?¿↨F;x|♦o/A╬♣╘≡∞─≤╝╘U∙♥0☺æ?|J%à{(éUmHµ %σl┴▼Ç9♣┌Ç?♫╡5╠yë~├╜♦íi♫╥╧
╬û?▓ε?╞┼→RtGqè₧ójWë♫╩∞j05├╞┘|>┘º∙↑j╪2┐|= ÷²
eY\╛P?#5wÑqc╙τ♦▓½Θt£6q∩?┌4┼t♠↕=7æƒ╙?╟|♂;║)∩÷≈═^╛{v⌂┌∞◄>6ä╝|
Code:
byte[] b= IOUtils.toByteArray(sock.getInputStream());
ByteArrayInputStream bais = new ByteArrayInputStream(b);
GZIPInputStream gzis = new GZIPInputStream(bais);
InputStreamReader reader = new InputStreamReader(gzis);
BufferedReader in = new BufferedReader(reader);
String readed;
while ((readed = in.readLine()) != null) {
System.out.println("read: "+readed);
}
Please advise.
Thanks,
Pradeep

The MIME header is NOT in the GZIP format, it's in plain text. You have to read that first before you can decompress the stream.
Also, why not just use this:
InputStream in = sock.getInputStream();
readHeader(in);
InputStream zin = new GZIPInputStream(in);

There are libraries for all of this. You can use, for example, Apache HTTP Components, or you can read its open source to see what it does. At very least, read the relevant specification.

I second bmarguiles' answer.
Only the body (response-body in the RFC) is compressed, so you only need to decompress the part that is after the \r\n\r\n.
Generally speaking, you can cut the response in half by that double CRLF, and only decompress the second half.