Develop Reference

Download file in java when Content-Length is incorrectly set?

I have an android app that downloads and uses a file at runtime. The file is valid as I can download it via the browser and open it up, etc. However my app kept reporting that the file is corrupted.
After investigation I discovered the server (which I have no control over) is returning an incorrect "Content-Length:" (~180 vs ~120000). The header is the culprit as I confirmed the issue by downloading the file with curl - which also resulted in a truncated file.
After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length. To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Anybody know of a way to download a file if the Content-Length is incorrectly set? A browser can.
Here's my latest attempt:
public static void downloadFileFromRemoteUrl(String urlString, String destination){
try {
URL url = new URL(urlString);
File file = new File(destination);
URLConnection urlConnection = url.openConnection();
InputStream inputStream = urlConnection.getInputStream();
byte[] buffer = new byte[1024];
int curLength = 0;
int newLength = 0;
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
byteArrayOutputStream.write(buffer, 0, newLength);
}
FileOutputStream fos = new FileOutputStream(file);
fos.write(byteArrayOutputStream.toByteArray());
fos.close();
android.util.Log.d("DB UPDATE", "Done downloading database. Size: " + byteArrayOutputStream.toByteArray().length);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}

After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length.
Nonsense. You are crediting those classes with paranormal powers. How could an output stream possibly become aware of the Content-length header? The URLConnection's input stream is being terminated at the content-length. Correctly.
To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Of course not.
Anybody know of a way to download a file if the Content-Length is incorrectly set?
You could use a Socket and engage in HTTP yourself, which is less trivial than it sounds. But the problem is at the server and that's where it should be fixed. Complain. Or else #Zong Yu is correct and the page is HTML containing JavaScript, say.
NB You don't need to read the entire file into memory:
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
fos.write(buffer, 0, newLength);
}

My final "solution" was to realize I was dealing with a 301 redirect response and not the actual resource! I updated the section that handles my url, checking for a 301 and if exists, update the url. The new url contained the Content-Length that corresponded with the file I was downloading.
// start by creating an http url connection object
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
// determine if this is a redirect
boolean redirect = false;
int status = httpURLConnection.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
// if it is, we need a new url
if (redirect) {
String newUrl = httpURLConnection.getHeaderField("Location");
httpURLConnection = (HttpURLConnection) new URL(newUrl).openConnection();
}

Try Fetch. Fetch is an in app download manager for Android. It's very easy to use. Find the GitHub page here. The project comes with several demos that you can try out. Disclaimer: I'm the creator of Fetch, and it is open source.

How to detect that whole image was downloaded in Android

I download images into my android application with this code:
private void download(URL url, File file) throws IOException {
Log.d(TAG, "download(): downloading file: " + url);
URLConnection urlConnection = url.openConnection();
InputStream inputStream = urlConnection.getInputStream();
BufferedInputStream bufferStream;
OutputStream outputStream = null;
try {
bufferStream = new BufferedInputStream(inputStream, 512);
outputStream = new FileOutputStream(file);
byte[] buffer = new byte[512];
int current;
while ((current = bufferStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, current);
}
} finally {
if (outputStream != null) {
outputStream.close();
}
if (inputStream != null) {
inputStream.close();
}
}
}
This code works fine, but some users and testers complained about incomplete photos. I suspect small network lags which interrupt connection.
So I would like to detect if whole image was downloaded and saved file is complete image. Is there any way how to detect file size from BufferedInputStream or is there another way how detect download completion?

I suggest using Google Volley which provides a super simple interface for networking in general, and image loading specifically. It takes care of threading and batching for you.
It's what Google use on the Google Play app.
It will solve your problem by providing listeners that notify you when the job is complete.

If you are downloading an ordinary file over HTTP, the method getContentLength() of URLConnection gives you the length that the file should have in the end.
You can compare the returned value of this method to the file length/length of downloaded data. If it's equal, then the file is complete:
int contentLength = urlConnection.getContentLength();
if (contentLength != -1) {
if (contentLength == file.length()) {
System.out.println("file is complete");
} else {
System.out.println("file is incomplete");
}
} else {
System.out.println("unknown if file is complete");
}

Try something like this. I think it could help you.

Download file via HTTP with unknown length with Java

I want to download a HTTP query with java, but the file I download has an undetermined length when downloading.
I thought this would be quite standard, so I searched and found a code snippet for it: http://snipplr.com/view/33805/
But it has a problem with the contentLength variable. As the length is unknown, I get -1 back. This creates an error. When I omit the entire check about contentLength, that means I always have to use the maximum buffer.
But the problem is that the file is not ready yet. So the flush gets only partially filled, and parts of the file get lost.
If you try downloading a link like http://overpass-api.de/api/interpreter?data=area%5Bname%3D%22Hoogstade%22%5D%3B%0A%28%0A++node%28area%29%3B%0A++%3C%3B%0A%29+%3B%0Aout+meta+qt%3B with that snippet, you'll notice the error, and when you always download the maximum buffer to omit the error, you end up with a corrupt XML file.
Is there some way to only download the ready part of the file? I would like if this could download big files (up to a few GB).

This should work, i tested it and it works for me:
void downloadFromUrl(URL url, String localFilename) throws IOException {
InputStream is = null;
FileOutputStream fos = null;
try {
URLConnection urlConn = url.openConnection();//connect
is = urlConn.getInputStream(); //get connection inputstream
fos = new FileOutputStream(localFilename); //open outputstream to local file
byte[] buffer = new byte[4096]; //declare 4KB buffer
int len;
//while we have availble data, continue downloading and storing to local file
while ((len = is.read(buffer)) > 0) {
fos.write(buffer, 0, len);
}
} finally {
try {
if (is != null) {
is.close();
}
} finally {
if (fos != null) {
fos.close();
}
}
}
}
If you want this to run in background, simply call it in a Thread:
Thread download = new Thread(){
public void run(){
URL url= new URL("http://overpass-api.de/api/interpreter?data=area%5Bname%3D%22Hoogstade%22%5D%3B%0A%28%0A++node%28area%29%3B%0A++%3C%3B%0A%29+%3B%0Aout+meta+qt%3B");
String localFilename="mylocalfile"; //needs to be replaced with local file path
downloadFromUrl(url, localFilename);
}
};
download.start();//start the thread

Getting an Image properly from My Java Web Server

I have created a simple HTTP server in Java. When the browser sends a GET request to my web server for a image file, let's say .jpg. Currently my browser does not get the image properly.
Exactly what header fields must be set?
Currently I have Date, Server, Content-type, Content-Length, Connection. I set the length by using:
fin = new FileInputStream(fileName);
contentLength = fin.available();
Content-Type is set to the correct mime-type, so no problem there.
I write the file data using:
public void sendFile (FileInputStream fin, DataOutputStream out)
{
byte[] buffer = new byte[1024];
int bytesRead;
int strCnt = 0;
try
{
int cnt = 0;
while ((bytesRead = fin.read(buffer)) != -1)
{
out.write(buffer, 0, bytesRead);
}
fin.close();
}
catch (IOException ex)
{
}
}
This is what is received by my Chrome Browser
It seems to not download the full content length.
The actual size of the image file is 2.73KB.
If no header fields are missing then what could be causing the problem?

Look likes you do not send all data. Try to add out.flush(); out.close(); before fin.close():
out.flush();
out.close();
fin.close();
I also suggest you to wrap your DataOutputStream into BufferedOutputStream. From my practice it work much more faster comparing to DataOutputStream when writing to hd/network.

How can I send a generic file to a jersey service and receive it correctly?

I'm developing a Jersey service that uses Dropbox's API.
I need to post a generic file to my service (the service would be able to manage every kind of file as well as you can do with the Dropbox API).
Client Side
So, I've implemented a simple client that:
opens the file,
creates a connection to the URL,
sets the correct HTTP method,
creates a FileInputStream and writes the file on the connection's outputstream using byte buffer.
This is the client test code.
public class Client {
public static void main(String args[]) throws IOException, InterruptedException {
String target = "http://localhost:8080/DCService/REST/professor/upload";
URL putUrl = new URL(target);
HttpURLConnection connection = (HttpURLConnection) putUrl.openConnection();
connection.setDoOutput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("POST");
connection.setRequestProperty("content-Type", "application/pdf");
OutputStream os = connection.getOutputStream();
InputStream is = new FileInputStream("welcome.pdf");
byte buf[] = new byte[1024];
int len;
int lung = 0;
while ((len = is.read(buf)) > 0) {
System.out.print(len);
lung += len;
is.read(buf);
os.write(buf, 0, len);
}
}
}
Server Side
I've a method that:
gets an InputStream as an argument,
creates a file with the same name and type of the original file.
The following code implements a test method to receive a specific PDF file.
#PUT
#Path("/upload")
#Consumes("application/pdf")
public Response uploadMaterial(InputStream is) throws IOException {
String name = "info";
String type = "exerc";
String description = "not defined";
Integer c = 10;
Integer p = 131;
File f = null;
try {
f = new File("welcome.pdf");
OutputStream out = new FileOutputStream(f);
byte buf[] = new byte[1024];
int len;
while ((len = is.read(buf)) > 0)
out.write(buf, 0, len);
out.close();
is.close();
System.out.println("\nFile is created........");
} catch (IOException e) {
throw new WebApplicationException(Response.Status.BAD_REQUEST);
}
//I need to pass a java.io.file object to this method
professorManager.uploadMaterial(name, type, description, c,p, f);
return Response.ok("<result>File " + name + " was uploaded</result>").build();
}
This implementation works only with text files. If I try to send a simple PDF the received file is not readable (after I've saved it on disk).
How can I satisfy my requirements? Could anyone suggest me solution?

You're client code is faulty.
while ((len = is.read(buf)) > 0) {
...
is.read(buf);
...
}
You're reading from the InputStream twice in every iteration. Remove the read statement from the loop's body and you'll be fine.
You've also said that the code presented in your question works with text files. I think that doesn't work either. Reading twice from the file you're trying to upload means you're uploading only half of its contents. Half a text file is still a text file, but half a PDF is only rubbish, so you can't open the latter. You should have double checked if the contents of your uploaded and saved text file is the same as the original.