Develop Reference

Download file in java when Content-Length is incorrectly set?

I have an android app that downloads and uses a file at runtime. The file is valid as I can download it via the browser and open it up, etc. However my app kept reporting that the file is corrupted.
After investigation I discovered the server (which I have no control over) is returning an incorrect "Content-Length:" (~180 vs ~120000). The header is the culprit as I confirmed the issue by downloading the file with curl - which also resulted in a truncated file.
After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length. To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Anybody know of a way to download a file if the Content-Length is incorrectly set? A browser can.
Here's my latest attempt:
public static void downloadFileFromRemoteUrl(String urlString, String destination){
try {
URL url = new URL(urlString);
File file = new File(destination);
URLConnection urlConnection = url.openConnection();
InputStream inputStream = urlConnection.getInputStream();
byte[] buffer = new byte[1024];
int curLength = 0;
int newLength = 0;
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
byteArrayOutputStream.write(buffer, 0, newLength);
}
FileOutputStream fos = new FileOutputStream(file);
fos.write(byteArrayOutputStream.toByteArray());
fos.close();
android.util.Log.d("DB UPDATE", "Done downloading database. Size: " + byteArrayOutputStream.toByteArray().length);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}

After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length.
Nonsense. You are crediting those classes with paranormal powers. How could an output stream possibly become aware of the Content-length header? The URLConnection's input stream is being terminated at the content-length. Correctly.
To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Of course not.
Anybody know of a way to download a file if the Content-Length is incorrectly set?
You could use a Socket and engage in HTTP yourself, which is less trivial than it sounds. But the problem is at the server and that's where it should be fixed. Complain. Or else #Zong Yu is correct and the page is HTML containing JavaScript, say.
NB You don't need to read the entire file into memory:
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
fos.write(buffer, 0, newLength);
}

My final "solution" was to realize I was dealing with a 301 redirect response and not the actual resource! I updated the section that handles my url, checking for a 301 and if exists, update the url. The new url contained the Content-Length that corresponded with the file I was downloading.
// start by creating an http url connection object
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
// determine if this is a redirect
boolean redirect = false;
int status = httpURLConnection.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
// if it is, we need a new url
if (redirect) {
String newUrl = httpURLConnection.getHeaderField("Location");
httpURLConnection = (HttpURLConnection) new URL(newUrl).openConnection();
}

Try Fetch. Fetch is an in app download manager for Android. It's very easy to use. Find the GitHub page here. The project comes with several demos that you can try out. Disclaimer: I'm the creator of Fetch, and it is open source.

ConnectTimeout not always blocking reading from inputstream?

In my application, I open UrlConnections, and I set connection timeout.
Sometimes, the timeout is thrown, but the inputstream is still readable (and do contain data)
I successfully reproed this using the following code:
System.setProperty("http.keepAlive", "false");
long length = 0;
while (length == 0) {
HttpURLConnection connection = null;
try {
connection = (HttpURLConnection) (new URL("http://www.worldofwargraphs.com").openConnection());
connection.setConnectTimeout(1); // 1ms
connection.connect();
} catch (IOException ex) {
try {
byte[] buf = new byte[8196];
try (InputStream is0 = connection.getInputStream(); InputStream is = new BufferedInputStream(is0, 8196)) {
if (is0 != null) {
int ret = 0;
while ((ret = is.read(buf)) > 0) {
length += ret;
}
}
}
} catch (IOException ex2) {
Logger.getLogger(JavaApplication6.class.getName()).log(Level.SEVERE, null, ex2);
}
}
}
System.out.println(length);
As you can see in this code, I try to open a connection with a small timeout (1ms) to make sure that the connection timeouts. Then I try to read the inputstream
The thing is that, sometimes when trying to read the stream I get a java.net.SocketTimeoutException: connect timed out (which is what I would expect), but the loop sometimes ends, displaying the number of bytes read (32912)
Could anyone explain me why this second behavior sometimes happens ?
I tried to google it, but found nothing about this.
Thanks

Your ridiculously short connect timeout expired, but the connection subsequently completed, so you were able to read. Note that setting a connection timeout doesn't set a read timeout, so the read blocked until data was available.
EDIT: In any case your code is wrong. You should set a realistic connect timeout and close the socket if you get it. Continuing to the read code after a connect exception doesn't make any sense.

Can't get Java's GZIPInputStream to read "gzip" response from server when using "Range" header

I've got a bit of code I've been using for a while to fetch data from a web server and a few months ago, I added compression support which seems to be working well for "regular" HTTP responses where the whole document is contained in the response. It does not seem to be working when I use a Range header, though.
Here is the code doing the real work:
InputStream in = null;
int bufferSize = 4096;
int responseCode = conn.getResponseCode();
boolean error = 5 == responseCode / 100
|| 4 == responseCode / 100;
int bytesRead = 0;
try
{
if(error)
in = conn.getErrorStream();
else
in = conn.getInputStream();
// Buffer the input
in = new BufferedInputStream(in);
// Handle compressed responses
if("gzip".equalsIgnoreCase(conn.getHeaderField("Content-Encoding")))
in = new GZIPInputStream(in);
else if("deflate".equalsIgnoreCase(conn.getHeaderField("Content-Encoding")))
in = new InflaterInputStream(in, new Inflater(true));
int n;
byte[] buffer = new byte[bufferSize];
// Now, just write out all the bytes
while(-1 != (n = in.read(buffer)))
{
bytesRead += n;
out.write(buffer, 0, n);
}
}
catch (IOException ioe)
{
System.err.println("Got IOException after reading " + bytesRead + " bytes");
throw ioe;
}
finally
{
if(null != in) try { in.close(); }
catch (IOException ioe)
{
System.err.println("Could not close InputStream");
ioe.printStackTrace();
}
}
Hitting a URL with the header Accept-Encoding: gzip,deflate,identity works just great: I can see that the data is returned by the server in compressed format, and the above code decompressed it nicely.
If I then add a Range: bytes=0-50 header, I get the following exception:
Got IOException after reading 0 bytes
Exception in thread "main" java.io.EOFException: Unexpected end of ZLIB input stream
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:116)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at [my code]([my code]:511)
Line 511 in my code is the line containing the in.read() call. The response includes the following headers:
Content-Type: text/html
Content-Encoding: gzip
Content-Range: bytes 0-50/751
Content-Length: 51
I have verified that, if I don't attempt to decompress the response, I actually get 51 bytes in the response... it's not a server failure (at least that I can tell). My server (Apache httpd) does not support "deflate", so I can't test another compression scheme (at least not right now).
I've also tried to request much more data (like 700 bytes of the total 751 bytes in the target resource) and I get the same kind of error.
Is there something I'm missing?
Update
Sorry, I forgot to include that I'm hitting Apache/2.2.22 on Linux. There aren't any server bugs I'm aware of. I'll have a bit of trouble verifying the compressed bytes that I retrieve from the server, as the "gzip" Content-Encoding is quite bare... e.g. I believe I can't just use "gunzip" on the command-line to decompress those bytes. I'll give it a try, though.

You can use 'gunzip' to decompress it, just keep in mind that the first 50 bytes probably aren't enough for gzip to decompress anything (headers, dictionaries etc). Try this: wget -O- -q <URL> | head -c 50 | zcat with your URL to see whether normal gzip works where your code fails.

Sigh switching to another server (happens to be running Apache/2.2.25) shows that my code does in fact work. The original target server appears to be affected by AWS's current outage in the US-EAST availability zone. I'm going to chalk this up to network errors and close this question. Thanks to those who offered suggestions.

uploading bytes to server via outputstream

I am trying to upload some bytes to the server for 15 seconds.I have written the following code to write the bytes to output stream :
long uploadedBytes=0;
ByteArrayInputStream byteArrayInputStream=null;
OutputStream outputStream=null;
try {
byte[] randomData=generateBinData(5*1024);
byte[] bytes = new byte[(int) 1024 * 5];
URL url = new URL(urls[0]);
HttpURLConnection connection =
(HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setRequestMethod("POST");
connection.setRequestProperty("Connection", "Keep-Alive");
outputStream = connection.getOutputStream();
byteArrayInputStream = new ByteArrayInputStream(randomData);
long startTime=System.currentTimeMillis();
while(byteArrayInputStream.read(bytes) > 0
&& timeDiff < 15000) {
outputStream.write(bytes, 0, bytes.length);
uploadedBytes += bytes.length;
byteArrayInputStream = new ByteArrayInputStream(randomData);
timeDiff = System.currentTimeMillis() - startTime;
int progress=(int)(timeDiff *100 / 15000);
publishProgress(progress);
}
But the progress for the above upload is running very fast and it shows large amount of bytes uploaded within seconds.Which is not according to my 2g mobile network connection.
For example it shows :
uploadedBytes =9850880 and with time difference(timeDiff) = 3 sec.
if i run the same code for 15 seconds it terminates the whole application.
Please help me to find where i am goind wrong.
thanks ...waiting for reply

Unless you set chunked or streaming transfer mode, HttpURLConnection buffers all the output before sending any of it, so it can get a Content-Length. So what you're seeing is the progress of the buffering, not of the transfer. Set chunked transfer mode and you will see a difference.
Your copy loop is wrong. It should be like this:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Your code will probably work in this specific case but that's not a reason not to get it right for all cases.

check your random byte length. i think the generateBinData() method is not generating 5Kb of data.
sure the uploadedBytes is huge. say, if a write to outputstream takes 10 milisec to write 5Kb(5*1024) of data,in 3 second you should able to write only 153600 bytes.
Reason for app termination - check if any read operation throws exception.

HttpsURLConnection stalling out when getInputStream() is called

[Java 1.5; Eclipse Galileo]
HttpsURLConnection seems to stall when the getInputStream() method is called. I've tried using different websites to no avail (currently https://www.google.com). I should point out I'm using httpS.
The code below has been modified based on what I've learned from other StackOverflow answers. However, no solutions I've tried thus far have worked.
I'd be very grateful for a nudge in the right direction :)
public static void request( URL url, String query )
{
try{
HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
//connection.setReadTimeout( 5000 ); //<-- uncommenting this line at least allows a timeout error to be thrown
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
System.setProperty("http.keepAlive", "false");
connection.setRequestMethod( "POST" );
// setting headers
connection.setRequestProperty("Content-length",String.valueOf (query.length()));
connection.setRequestProperty("Content-Type","application/x-www-form-urlencoded"); //WAS application/x-www- form-urlencoded
connection.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)");
////////////////////////////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////////////////////////////
System.out.println( "THIS line stalls" + connection.getInputStream() );
////////////////////////////////////////////////////////////////////////////////////
}catch( Exception e ) {
System.out.println( e );
e.printStackTrace();
}
Typical errors look like:
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:782)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:739)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:681)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:626)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:983)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:234)
at https_understanding.HTTPSRequest.request(HTTPSRequest.java:60)
at https_understanding.Main.main(Main.java:17)

connection.setDoOutput(true);
This means that you have to open, write to, and close the connection's output stream before you attempt to read from its input stream. See the docs.

I reproduced the problem in Android 2.2: when downloading from a web-server over wireless and a HTTPS URL, the error is a socket "read time out" at URLConnection.getInputStream()
To fix it, use url.openStream() for the InputStream instead of connection.getInputStream()
Bonus: you can get the length of the file you're downloading so you can show a % complete indicator
code sample:
private final int TIMEOUT_CONNECTION = 5000;//5sec
private final int TIMEOUT_SOCKET = 30000;//30sec
file = new File(strFullPath);
URL url = new URL(strURL);
URLConnection ucon = url.openConnection();
//this timeout affects how long it takes for the app to realize there's a connection problem
ucon.setReadTimeout(TIMEOUT_CONNECTION);
ucon.setConnectTimeout(TIMEOUT_SOCKET);
//IMPORTANT UPDATE:
// ucon.getInputStream() often times-out over wireless
// so, replace it with ucon.connect() and url.openStream()
ucon.connect();
iFileLength = ucon.getContentLength();//returns -1 if not set in response header
if (iFileLength != -1)
{
Log.i(TAG, "Expected Filelength = "+String.valueOf(iFileLength)+" bytes");
}
//Define InputStreams to read from the URLConnection.
// uses 5KB download buffer
InputStream is = url.openStream();//ucon.getInputStream();
BufferedInputStream inStream = new BufferedInputStream(is, 1024 * 5);
outStream = new FileOutputStream(file);
bFileOpen = true;
byte[] buff = new byte[5 * 1024];
//Read bytes (and store them) until there is nothing more to read(-1)
int total=0;
int len;
int percentdone;
int percentdonelast=0;
while ((len = inStream.read(buff)) != -1)
{
//write to file
outStream.write(buff,0,len);
//calculate percent done
if (iFileLength != -1)
{
total+=len;
percentdone=(int)(total*100/iFileLength);
//limit the number of messages to no more than one message every 10%
if ( (percentdone - percentdonelast) > 10)
{
percentdonelast = percentdone;
Log.i(TAG,String.valueOf(percentdone)+"%");
}
}
}
//clean up
outStream.flush();//THIS IS VERY IMPORTANT !
outStream.close();
bFileOpen = false;
inStream.close();

Also don't set the content-length header. Java will do that for you.