I am using HttpClient to send a request a server which is supposed to return xml data. This data is returned as chunked data. I am then trying to write the received xml data to a file. The code I use is shown below:
HttpEntity entity = response.getEntity();
InputStream instream = entity.getContent();
try {
// do something useful
InputStreamReader isr = new InputStreamReader(instream);
FileWriter pw;
pw = new FileWriter(filename, append);
OutputStreamWriter outWriter = new OutputStreamWriter(new FileOutputStream(filename, append), "UTF-8");
BufferedReader rd = new BufferedReader(isr);
String line = "";
while ((line = rd.readLine()) != null) {
// pw.write(line);
outWriter.write(line);
}
isr.close();
pw.close();
} finally {
instream.close();
}
This results in data that looks as follows to be printed to the file:
This code works for non chunked data. How do I properly handle chunked data responses using HttpClient. Any help is greatly appreciated.
I don't think that your problem is the chunking of data.
XML data is plain text data - chunking it means that it is split into several parts that are transfered after another. Therefore each chunk should contain visible plain text xml data which is obviously not the case as shown in the data picture.
May be the content is encoded compressed via gzip or it is not plain text XML but binary encoded XML (e.g. like WBXML).
What concrete type you have you can see from the sent server response headers, especially the used mime type it contains.
Related
I am using the Wargaming API for accessing game data and every time I download one of their JSON files in Java it is never complete no matter the filesize.
I think its related to my way of reading from streams:
//variables:
int data;
InputStreamReader isr;
PrintWriter pw;
//Read from file:
while ((data = isr.read()) != -1)
{
pw.print((char) data);
}
The InputStreamReader comes from a URLConnection.getInputStream() associated with the URL to the API.
So how do I read from the stream without either it breaking or encountering a end of file?
I use org.apache.commons.io.IOUtils with Gson, this set has never disappointed me.
InputStream stream = connection.getInputStream(); // or getErrorStream()
StringWriter writer = new StringWriter();
IOUtils.copy(stream, writer, "UTF-8");
String response = writer.toString();
Entity entity = new Gson().fromJson(response, Entity.class);
I need to send a binary message which is divided to 2 parts:
The first part is 4 bytes and it has some information (say an integer)
The second part has an XMLtext stream.
I have never done something like this before, how can I do this?
My code is something like this:
public String serverCall(String link, String data){
HttpURLConnection connection;
OutputStreamWriter writer = null;
URL url = null;
String parameters = data;
try
{
url = new URL(link);
connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "text/xml");
connection.setRequestMethod("POST");
writer = new OutputStreamWriter(connection.getOutputStream());
writer.write(parameters);
writer.flush();
writer.close();
}
catch(IOException e)
{
e.printStackTrace();
}
}
How do I set the XML to be 4 bytes and how do I have 4 bytes of text before it?
(Info) The HTTP protocol uses method PUT to "transport a file on the server" (instead of POST).
The transport of binary data better not have a content type "text/..." but "application/bin".
You however could send the XML as "text/xml; charset=UTF-8", and use your own header
connection.setHeader("MyCode",
String.format("%02x%02x%02x%02x", bytes[0], bytes[1], bytes[2], bytes[3]));
If you send all as binary, do not use a Writer (converts bytes to some character encoding), but a Stream (BufferedOutputStream). The XML as:
byte[] xmlBytes = xml.getBytes("UTF-8");
UTF-8 if there is no other encoding mentioned in <?xml ...>.
The close() already flushes, so flush() is not needed.
If the binary portion of your message are mere 4 bytes, percent-encode it and send it as an additional url-parameter. alternatively, add it to the existing xml stream.
the java classes Uri and URLEncoder provide the necessary methods.
I'm trying to login to a webpage, but even before that, I'm loading the page using HttpGet, and this is one the lines that's being returned,
ÓA;
That's all I could put, won't let me paste any other characters. But they are all like that, like I'm somehow getting the wrong encoding? Here is the code I am using to GET
HttpGet httpget = new HttpGet(url);
if(headers == null) {
headers = getDefaultHeaders();
}
for(String s : headers.keySet()) {
httpget.addHeader(s, headers.get(s));
}
HttpResponse response = getClient().execute(httpget);
HttpEntity entity = response.getEntity();
System.out.println("Status Line: " + response.getStatusLine());
if (entity != null) {
InputStream input = entity.getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String ln = "";
while((ln = reader.readLine()) != null) {
System.out.println("During Get - " + ln);
}
}
What am I doing wrong?
Thanks for any help.
If you need any more information like headers, just ask.
The following line is possibly the cause of your problems:
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
You are creating a reader using the default characterset of your platform, and completely ignoring any character set that may be specified in the HTTP response headers.
If you are getting the same problem when reading the content the correct way, then it is possible that the server is at fault for not setting the response header correctly.
DO the entity reading like this:
String content = org.apache.http.util.EntityUtils.toString( entity );
System.out.println(content);
This is going to read it all for you so you can check what's being really returned.
Make sure that you didn't accidentally go to port 443 with a simple HTTP connection. Because in that case you will get back the SSL handshake instead of an HTTP response.
In Google App Engine, I tried reading a .txt file from a URL. Because the maximum allowed size is 1MB and the file is slightly larger, I'm using an alternative method described here.
So, what I'm trying to do is this:
FetchOptions fo = FetchOptions.Builder.allowTruncate().doNotFollowRedirects();
HTTPRequest request = new HTTPRequest(url,HTTPMethod.GET,FetchOptions.Builder.allowTruncate());
URLFetchService service = URLFetchServiceFactory.getURLFetchService();
HTTPResponse response = service.fetch(request);
My question is now, how can I read this response line by line? I'm trying to process each line which should be possible somehow as the source file is a simple text file.
I can get a byte[] with
byte[] content = response.getContent();
but I'm struggling with the further processing of it.
Or, can I do something completely different to achieve the same thing ?
I'm trying to read it line by line because I don't need all the lines. Processing would be much easier than to put everything in one large string.
You can try:
ByteArrayInputStream bais = new ByteArrayInputStream(content);
BufferedReader reader = new BufferedReader(new InputStreamReader(bais, "utf-8"));
String line = null;
while ((line = reader.readLine()) != null) {
...
}
Alternatively, you can use IOUtils and call IOUtils.lineIterator(reader) (where reader is the InputStreamReader)
Can any one of you solve this problem !
Problem Description:
i have received content-encoding: gzip header from http web-server.
now i want to decode the content but when i use GZIP classes from jdk 1.6.12, it gives null.
does it means that contents are not in gzip format ? or are there some another classes for decompress http response content?
Sample Code:
System.out.println("Reading InputStream");
InputStream in = httpuc.getInputStream();// httpuc is an object of httpurlconnection<br>
System.out.println("Before reading GZIP inputstream");
System.out.println(in);
GZIPInputStream gin = new GZIPInputStream(in));
System.out.println("After reading GZIP inputstream");
Output:
Reading InputStream
Before reading GZIP inputstream
sun.net.www.protocol.http.HttpURLConnection$HttpInputStream#8acf6e
null
I have found one error in code, but don't able to understand it properly. what does it indicates.
Error ! java.io.EOFException
Thanks
I think you should have a look at HTTPClient, which will handle a lot of the HTTP issues for you. In particular, it allows access to the response body, which may be gzipped, and then you simply feed that through a GZIPInputStream
e.g.
Header hce = postMethod.getResponseHeader("Content-Encoding");
InputStream in = null;
if(null != hce)
{
if(hce.getValue().equals(GZIP)) {
in = new GZIPInputStream(postMethod.getResponseBodyAsStream());
}
// etc...
I second Brian's suggestion. Whenever u need to deal with getting/posting stuff via HTTP don't bother with low-level access use the Apache HTTP client.
InputStream is = con.getInputStream();
InputStream bodyStream = new GZIPInputStream(is);
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
byte[] buffer = new byte[4096];
int length;
while ((length = bodyStream.read(buffer)) > 0) {
outStream.write(buffer, 0, length);
}
String body = new String(outStream.toByteArray(), "UTF-8");