Develop Reference

JSON Url returns something else?

I am trying to access this json file : http://www.cloudpricingcalculator.appspot.com/static/data/pricelist.json with Java.
But when I read it, sometimes it gives me a JSON string (that's ok) and sometimes it gives me something else and json.simple.parser throw an Unexpected character(<) at position 0.
Based on what I read on stackOverflow, it may be that it returns XML instead of JSON. As my url is "json", how is it possible ?
Here is the code i'm using :
String baseUrl = "http://www.cloudpricingcalculator.appspot.com/static/data/pricelist.json";
...
URL url = new URL(this.baseUrl);
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String l;
String json = "";
System.out.println(url);
while((l=in.readLine()) != null){
System.out.println(l);
json+=l;
}
JSONParser parser = new JSONParser();
JSONObject jsonObject = (JSONObject) parser.parse(json);
and the log is <followed by an lot of squares and unknown characters like ÿÕ[s›È and an error Unexpected character () at position 0.

You are not taking into account the compression and encoding of the resource returned from the server. The response of a HEAD request is the following:
rpax#machine:~$ HEAD http://www.cloudpricingcalculator.appspot.com/static/data/pricelist.json
200 OK
Cache-Control: public, max-age=600
Connection: close
Date: Mon, 21 Aug 2017 12:02:14 GMT
Age: 112
ETag: "n_s_jQ"
Server: Google Frontend
Content-Encoding: gzip <---- *HERE*
Content-Length: 7902
Content-Type: application/json
Expires: Mon, 21 Aug 2017 12:12:14 GMT
...
For avoiding this issue, you can wrap the url stream into a GZIPInputStream :
GZIPInputStream gis = new GZIPInputStream(url.openStream());
BufferedReader in = new BufferedReader(new InputStreamReader(gis));
// ...
And the data returned when executing readline() will be decompressed.

how to send and accept file over Java Http Post?

I just want to send a text file and a JPEG file over the network. fortunately, i have access to both the server code and the client code. Here's my (google app engine) code.
private void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
GcsService gcsService = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
GcsFilename filename = new GcsFilename("my-bucket", "my-file");
Builder fileOptionsBuilder = new GcsFileOptions.Builder();
fileOptionsBuilder.mimeType("text/html");
GcsFileOptions fileOptions = fileOptionsBuilder.build();
GcsOutputChannel outputChannel = gcsService.createOrReplace(filename, fileOptions);
byte[] buffer = new byte[1024];
InputStream reader = req.getInputStream();
BufferedOutputStream outStream = new BufferedOutputStream(Channels.newOutputStream(outputChannel));
while(true) {
int bytesRead = reader.read(buffer);
if (bytesRead == -1) {
break; // have a break up with the loop.
} else if (bytesRead < 1024) {
byte[] temp = Arrays.copyOf(buffer, bytesRead);
outStream.write(temp);
} else {
outStream.write(buffer);
}
}
outStream.close();
outputChannel.close();
}
As you can see, i use a raw InputStream to get all the data that is sent over the net.
and on the client side, i send a text file over like so: (in Android)
HttpClient httpClient = new DefaultHttpClient();
HttpPost httpost = new HttpPost("http://my-hosted-url/postit");
MultipartEntity entity = new entity.addPart("myImageFile", new FileBody(someLogFile));
httpost.setEntity(entity);
HttpResponse response;
response = httpClient.execute(httpost);
This works just fine... sort of. the problem is that when i try to view the file/data that is sent, it has a header on top of it, as such:
--NNqarc4FsG0G8hUzd82A6TCjgzKH Content-Disposition: form-data; name="myString" Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 8bit STRING_VALUE ---NNqarc4FsG0G8hUzd82A6TCjgzKH Content-Disposition: form-data; name="myImageFile"; filename="something.txt" Content-Type: application/octet-stream Content-Transfer-Encoding: binary
[Thu Aug 14 17:14:26 PDT 2014] then the real log starts here...
How do i get rid of the headers that is somehow stuck to the body?

What you have here is a multipart request. It is a single request where the body consists of the various parts separated by a separator string.
In your case, it's more easily viewed as:
--NNqarc4FsG0G8hUzd82A6TCjgzKH
Content-Disposition: form-data; name="myString"
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 8bit
STRING_VALUE
---NNqarc4FsG0G8hUzd82A6TCjgzKH
Content-Disposition: form-data; name="myImageFile"; filename="something.txt"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
[binary here]
It has two parts where each part has its corresponding headers and body. I'm guessing you're interested in the bodies. You'll need to extract them.
You can either read the HTTP specification and/or the specification about multipart requests and write your own parser, or you can use some built-in (I don't know if GAE is Servlet 3.0 ready or not) or 3rd party methods. See these
How can my Servlet receive parameters from a multipart/form-data form?
Convenient way to parse incoming multipart/form-data parameters in a Servlet
How can I handle multipart/form-data POST requests in my java servlet?

Java downloading a file and retrieving current download completion

In a program I have that auto-updates, I need it to download a specific file (working already) and it does so with the following code:
public static void getFile() {
try {
URL url = new URL("https://dl.dropboxusercontent.com/s/tc301v61zt0v5cd/texture_pack.png?dl=1");
InputStream in = new BufferedInputStream(url.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n = 0;
while (-1 != (n = in.read(buf))) {
out.write(buf, 0, n);
}
out.close();
in.close();
byte[] response = out.toByteArray();
FileOutputStream fos = new FileOutputStream(file4);
fos.write(response);
fos.close();
} catch (Exception e) {
JOptionPane.showMessageDialog(null, "Check your internet connection, then try again.\nOr re-install the program.\nError Message 7", "Could Not Download The Required Resources.", JOptionPane.NO_OPTION);
e.printStackTrace();
System.exit(0);
}
}
How would I implement a way to get the current completion of the download (like the percent it's downloaded) and put it into an integer to be printed out in the console (just for developer testing). Also what sort of equation could i use to get the estimated amount of time left on the download? Anything would help! Thanks.

If you examine the HTTP headers sent in the file download, you'll discover the file size. From this you can calculate percentage complete:
curl --head "https://dl.dropboxusercontent.com/s/tc301v61zt0v5cd/texture_pack.png?dl=1"
Gives you this:
HTTP/1.1 200 OK
accept-ranges: bytes
cache-control: max-age=0
content-disposition: attachment; filename="texture_pack.png"
Content-length: 29187
Content-Type: image/png
Date: Mon, 28 Apr 2014 22:38:34 GMT
etag: 121936d
pragma: public
Server: nginx
x-dropbox-request-id: 1948ddaaa2df2bdf2c4a2ce3fdbeb349
X-RequestId: 4d9ce90907637e06728713be03e6815d
x-server-response-time: 514
Connection: keep-alive
You may have to use something more advanced than standard Java library classes for your download to access the headers however, something like Apache HttpClient.

There is no way to get the size of a streamed file without streaming to the end of the file, so it can't be done within that block of code.
If you are willing to adopt the Dropbox API, you can use that to get the size of the file before starting the download. Then you can work with that size and the number bytes downloaded (n in your code) to achieve what you need.
The class in the Dropbox API that you need is DbxEntry.File.

Decompressing a gzipped http response

Hello fellow java developers. I receive a response with headers and body as below, but when I try to decompress it using the code below, it fails with this exception:
java.io.IOException: Not in GZIP format
Response:
HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Content-Encoding: gzip
Server: Jetty(6.1.x)
▼ ═UMs¢0►=7┐ép?╙6-C╚$╢gΩ↓╟±╪₧∟zS╨╓╓♦$FÆ╒÷▀G┬╚╞8N≤╤Cf°►╦█╖╗o↨æJÄ+`:↓2
♣»└√S▬L&?∙┬_)U╔|♣%ûíyk_à\,æ] hⁿ?▀xΓ∟o╜4♫ù\#MAHG?┤(Q¶╞⌡▌Ç?▼ô[7Fí¼↔φ☻I%╓╣Z♂?¿↨F;x|♦o/A╬♣╘≡∞─≤╝╘U∙♥0☺æ?|J%à{(éUmHµ %σl┴▼Ç9♣┌Ç?♫╡5╠yë~├╜♦íi♫╥╧
╬û?▓ε?╞┼→RtGqè₧ójWë♫╩∞j05├╞┘|>┘º∙↑j╪2┐|= ÷²
eY\╛P?#5wÑqc╙τ♦▓½Θt£6q∩?┌4┼t♠↕=7æƒ╙?╟|♂;║)∩÷≈═^╛{v⌂┌∞◄>6ä╝|
Code:
byte[] b= IOUtils.toByteArray(sock.getInputStream());
ByteArrayInputStream bais = new ByteArrayInputStream(b);
GZIPInputStream gzis = new GZIPInputStream(bais);
InputStreamReader reader = new InputStreamReader(gzis);
BufferedReader in = new BufferedReader(reader);
String readed;
while ((readed = in.readLine()) != null) {
System.out.println("read: "+readed);
}
Please advise.
Thanks,
Pradeep

The MIME header is NOT in the GZIP format, it's in plain text. You have to read that first before you can decompress the stream.
Also, why not just use this:
InputStream in = sock.getInputStream();
readHeader(in);
InputStream zin = new GZIPInputStream(in);

There are libraries for all of this. You can use, for example, Apache HTTP Components, or you can read its open source to see what it does. At very least, read the relevant specification.

I second bmarguiles' answer.
Only the body (response-body in the RFC) is compressed, so you only need to decompress the part that is after the \r\n\r\n.
Generally speaking, you can cut the response in half by that double CRLF, and only decompress the second half.

Read Image File Through Java Socket

This is what I have so far,
Socket clientSocket = new Socket(HOST, PORT);
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
InputStream is = socket.getInputStream();
byte[] byteChunk = new byte[1024];
int c = is.read(byteChunk);
while (c != -1){
buffer.write(byteChunk, 0, c);
c = is.read(byteChunk);
}
BufferedImage bufferedImage = ImageIO.read(new ByteArrayInputStream(buffer.toByteArray()));
My problem with my code is ImageIO.read() returns null.
When I print the content of ByteArrayOutputStream object what i get is header part
HTTP/1.1 200 OK
Date: Fri, 30 Dec 2011 11:34:19 GMT
Server: Apache/2.2.3 (Debian) ...........
Last-Modified: Tue, 20 Dec 2011 19:12:23 GMT
ETag: "502812-490e-4b48ad8d273c0"
Accept-Ranges: bytes
Content-Length: 18702
Connection: close
Content-Type: image/jpeg
followed with a empty line plus many lines with different characters such as Àã$sU,e6‡Í~áŸP;Öã….
Again my problem is ImageIO.read() function returns null.
Thanks in advance.

Why you don't want to use simple http URL to get image from host?
I mean:
URL imageURL = new URL("http://host:port/address");
BufferedImage bufferedImage = ImageIO.read(imageURL);
If you want to use plain socket you have to parse http response and extract data from the http reply manually: read/skip headers, read binary data and pass it to ImageIO.read (or seek stream to correct position and pass stream to ImageIO.read).