I am using the Wargaming API for accessing game data and every time I download one of their JSON files in Java it is never complete no matter the filesize.
I think its related to my way of reading from streams:
//variables:
int data;
InputStreamReader isr;
PrintWriter pw;
//Read from file:
while ((data = isr.read()) != -1)
{
pw.print((char) data);
}
The InputStreamReader comes from a URLConnection.getInputStream() associated with the URL to the API.
So how do I read from the stream without either it breaking or encountering a end of file?
I use org.apache.commons.io.IOUtils with Gson, this set has never disappointed me.
InputStream stream = connection.getInputStream(); // or getErrorStream()
StringWriter writer = new StringWriter();
IOUtils.copy(stream, writer, "UTF-8");
String response = writer.toString();
Entity entity = new Gson().fromJson(response, Entity.class);
Related
I am currently working with android and i am using a http connection with some headers (i havent included them or the real url for security purposes) to get a JSON response from an API, and feeding that response back into the application. The problem that i am having is that when using the getContentLength method of the http request, the wrong length is being returned (wrong length returned is 1225, the correct length in characters of the JSON array is 3365).
I have a feeling that the JSON is not fully loaded when my reader starts to read it, and as such is only reading the loaded JSON at that point. Is there any way around this, possibly using a delay on the HTTP connection or waiting until it is fully loaded to read the data?
URL url = new URL("https://www.exampleofurl.com");
HttpURLConnection request = (HttpURLConnection) url.openConnection();
request.connect();
int responseCode = request.getResponseCode();
if(responseCode == HttpURLConnection.HTTP_OK) {
InputStream inputStream = request.getInputStream();
InputStreamReader reader = new InputStreamReader(inputStream);
long contentLength2 = Long.parseLong(request.getHeaderField("Content-Length"));
Log.i("contentLength: ", "Content: " + contentLength2);
I generally don't recommend always relying on "Content-Length" as it may not be available (you get -1), or perhaps affected by intermediate proxy.
Why don't you just read your stream until it is exhausted into memory buffer (say, StringBuilder) and then get the actual size, for example :
BufferedReader br = new BufferedReader(inputStream); // inputStream in your code
String line;
StringBuilder sb = new StringBuilder();
while ((line = br.readLine()) != null) {
sb.append(line);
}
// finished reading
System.out.println("data size = " + sb.length());
JSONObject data = new JSONObject(sb.toString());
// and don't forget finally clauses with closing streams/connections
I am calling a wcf web service in android. All appear to work fine until I call a method that is returning quite a bit of data. It appears the result is being cut off and throwing not a valid json object. I check the result, and it return about 9089 characters. The bufferedreader and inputstreamreader have both return the same count with the same result. I try calling a different method that is returning about 2000 records, and it work without problems.
Here is the sample code where I am reading the result:
StringEntity entity = new StringEntity(jsonObject.toString());
httpPost.setEntity(entity);
HttpResponse httpResponse = httpClient.execute(httpPost);
HttpEntity responseEntity = httpResponse.getEntity();
char[] buffer = new char[(int)responseEntity.getContentLength()];
InputStream stream1 =responseEntity.getContent();
InputStreamReader reader = new InputStreamReader(stream1);
reader.read(buffer);
stream1.close();
String sInvokeReturnValue = new String(buffer);
Any kind of help would be greatly appreciated.
All appear to work fine until I call a method that is returning quite a bit of data. It appears the result is being cut off and throwing not a valid json object.
I suspect that your read is hitting a buffer limit and returning not as many bytes as you are expecting. You should put your read in a loop to make sure you get all of the input.
What I would recommend is to use a StringWriter to read in your data. For example:
// not much point on allocating a huge buffer here
char[] buffer = new char[4096];
InputStreamReader reader = new InputStreamReader(responseEntity.getContent());
try {
StringWriter writer = new StringWriter();
while (true) {
int numRead = reader.read(buffer);
if (numRead < 0) {
break;
}
writer.write(buffer, 0, numRead);
}
} finally {
reader.close();
}
stream1.close();
String sInvokeReturnValue = writer.toString();
The following code uses BufferedReader to read from an HTTP response stream:
final StringBuilder responseBuilder = new StringBuilder();
String line = bufferedReader.readLine();
while (line != null) {
responseBuilder.append(line);
responseBuilder.append('\n');
line = bufferedReader.readLine();
}
response = responseBuilder.toString();
But appending '\n' to each line seems a bit flawed. I want to return the HTTP response exactly as-is so what if it doesn't have a return character after the last line? One would get added anyway using the code above - is there a better way?
I want to return the HTTP response exactly as-is
Don't use readLine() then - it's as simple as that. I'd suggest using a StringWriter instead:
StringWriter writer = new StringWriter();
char[] buffer = new char[8192];
int charsRead;
while ((charsRead = bufferedReader.read(buffer)) > 0) {
writer.write(buffer, 0, charsRead);
}
response = writer.toString();
Note that even this won't work if you get the encoding wrong. To preserve the exact HTTP response, you'd need to read (and write) it as a binary stream.
I am using HttpClient to send a request a server which is supposed to return xml data. This data is returned as chunked data. I am then trying to write the received xml data to a file. The code I use is shown below:
HttpEntity entity = response.getEntity();
InputStream instream = entity.getContent();
try {
// do something useful
InputStreamReader isr = new InputStreamReader(instream);
FileWriter pw;
pw = new FileWriter(filename, append);
OutputStreamWriter outWriter = new OutputStreamWriter(new FileOutputStream(filename, append), "UTF-8");
BufferedReader rd = new BufferedReader(isr);
String line = "";
while ((line = rd.readLine()) != null) {
// pw.write(line);
outWriter.write(line);
}
isr.close();
pw.close();
} finally {
instream.close();
}
This results in data that looks as follows to be printed to the file:
This code works for non chunked data. How do I properly handle chunked data responses using HttpClient. Any help is greatly appreciated.
I don't think that your problem is the chunking of data.
XML data is plain text data - chunking it means that it is split into several parts that are transfered after another. Therefore each chunk should contain visible plain text xml data which is obviously not the case as shown in the data picture.
May be the content is encoded compressed via gzip or it is not plain text XML but binary encoded XML (e.g. like WBXML).
What concrete type you have you can see from the sent server response headers, especially the used mime type it contains.
I am working in J2ME.
I am getting json response from server then I try to parse this json data as per my requirement.
this is my code for getting response from server:-
InputStream inputStream = null;
OutputStreamWriter out = null;
byte[] readData = new byte[50000];
String response = "no";
try {
// --- write ---
out = new OutputStreamWriter(connection.openOutputStream(), "UTF-8");
out.write(data.toString());
out.close();
// --- read ---
int responseCode = connection.getResponseCode();
if (responseCode != HttpConnection.HTTP_OK) {
throw new IOException("HTTP response code: " + responseCode);
}
inputStream = connection.openInputStream();
int actual = inputStream.read(readData);
response = new String(readData, 0, actual, "UTF-8");
return response;
}
when response from server is small then it works fine, but if response is large then it get half of response and return to my another method. Please suggest me what should I do to get large amount of data into my readData and response variable.
Thank you in advance.
You will need to read all data before (there can be more data in the stream).
As you have noticed the call to InputStream.read doesn't guarantee you to fill your buffer (readData) it return the number of bytes it could read at the time. It just doesn't matter how big your buffer is on your side.
You will need to read re-call the InputStream.read method to check that you have all the data available in the stream. The read method will return -1 when no more data is available.
This is an example how you can do it:
....
ByteArrayOutputStream bos = new ByteArrayOutputStream();
int read;
byte[] tmp = new byte[1024];
while ((read = inputStream.read(tmp)) != -1)
bos.write(tmp, 0, read);
return new String(bos.toByteArray(), "UTF-8");
Also, when you are done with the connection you should call close on it so that the system knows that you don't need it anymore.
You nead to flush and close your input stream after reading!!!
inputStream.close();