Develop Reference

Download file in java when Content-Length is incorrectly set?

I have an android app that downloads and uses a file at runtime. The file is valid as I can download it via the browser and open it up, etc. However my app kept reporting that the file is corrupted.
After investigation I discovered the server (which I have no control over) is returning an incorrect "Content-Length:" (~180 vs ~120000). The header is the culprit as I confirmed the issue by downloading the file with curl - which also resulted in a truncated file.
After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length. To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Anybody know of a way to download a file if the Content-Length is incorrectly set? A browser can.
Here's my latest attempt:
public static void downloadFileFromRemoteUrl(String urlString, String destination){
try {
URL url = new URL(urlString);
File file = new File(destination);
URLConnection urlConnection = url.openConnection();
InputStream inputStream = urlConnection.getInputStream();
byte[] buffer = new byte[1024];
int curLength = 0;
int newLength = 0;
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
byteArrayOutputStream.write(buffer, 0, newLength);
}
FileOutputStream fos = new FileOutputStream(file);
fos.write(byteArrayOutputStream.toByteArray());
fos.close();
android.util.Log.d("DB UPDATE", "Done downloading database. Size: " + byteArrayOutputStream.toByteArray().length);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}

After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length.
Nonsense. You are crediting those classes with paranormal powers. How could an output stream possibly become aware of the Content-length header? The URLConnection's input stream is being terminated at the content-length. Correctly.
To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Of course not.
Anybody know of a way to download a file if the Content-Length is incorrectly set?
You could use a Socket and engage in HTTP yourself, which is less trivial than it sounds. But the problem is at the server and that's where it should be fixed. Complain. Or else #Zong Yu is correct and the page is HTML containing JavaScript, say.
NB You don't need to read the entire file into memory:
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
fos.write(buffer, 0, newLength);
}

My final "solution" was to realize I was dealing with a 301 redirect response and not the actual resource! I updated the section that handles my url, checking for a 301 and if exists, update the url. The new url contained the Content-Length that corresponded with the file I was downloading.
// start by creating an http url connection object
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
// determine if this is a redirect
boolean redirect = false;
int status = httpURLConnection.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
// if it is, we need a new url
if (redirect) {
String newUrl = httpURLConnection.getHeaderField("Location");
httpURLConnection = (HttpURLConnection) new URL(newUrl).openConnection();
}

Try Fetch. Fetch is an in app download manager for Android. It's very easy to use. Find the GitHub page here. The project comes with several demos that you can try out. Disclaimer: I'm the creator of Fetch, and it is open source.

How to save a PDF file from the PDF URL of Orbeon form with Java

I tried to save a PDF document from the PDF URL generated by Orbeon with Java, and always my InputStream is null. When I open this URL in the browser it work well but when I inspect the code I don't find the body of the PDF.
try {
URL urlPDF = new URL(urlPdfOrbeon);
URLConnection connection = urlPDF.openConnection();
InputStream in = connection.getInputStream();
System.out.println(IOUtils.toString(in)) ;
FileOutputStream fos = new FileOutputStream(newFile("yourFile.pdf"));
int length = -1;
byte[] buffer = new byte[1024];
while ((length = in.read(buffer)) > -1) {
fos.write(buffer, 0, length);
}
fos.close();
in.close();
}

The input stream is not null. But it doesn't have any byte to read anymore when you try writing its content to the file. And that's expected, since immediately before, you read everything in the stream by calling
System.out.println(IOUtils.toString(in));
EDIT:
If the input stream actually contains an HTML page asking you to log in, it's probably that you need to be logged in to access this PDF.

Download servlet is very slow

I have written a servlet which will download the file from a server location. In our own INTRAnet the download seems to be very very slow and also when I have the Adobe addon installed in my browser and if I am downloading a PDF file, the Adobe addon will display the progress bar while downloading the PDF but this is not happening in my case! Below is my code! Should I not response it as attachment?
PrintWriter out = response.getWriter();
response.setContentType("APPLICATION/OCTET-STREAM");
response.setIntHeader("Refresh", 1);
response.setHeader("Content-Disposition",
"inline; filename=\"" + fileNameWithExtension
+ "\"");
FileInputStream fileInStream = new FileInputStream(
filePathWithExtension);
BufferedInputStream bufferInStream = new BufferedInputStream(
fileInStream);
int cnt;
while ((cnt = bufferInStream.read()) != -1) {
out.write(cnt);
}
fileInStream.close();
out.close();
Not sure if there is a better way to do. Basically I tried converting one of my dot net code into this Java Servlet. THe current .NET code is very fast compared to this!
This is hosted on Apache Tomcat and the .NET code is hosted on IIS.

Reading and writing a byte at a time is horrifically inefficient. The canonical way to copy a stream in Java is as follows:
byte[] buffer = new byte[8192]; // or more if you like
int count;
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
And you should not use a Writer here, use an OutputStream.

Downloading a file created in server using Java

I am trying to download a file from a given URL. The URL is not a direct file URL. When this URL is provided in browser manually, we get a prompt for download/save.
For example, http://www.my-domain.com/download/type/salary/format/excel
I have no issues in the URL which has the file name directly in the URL. In the above URL, based on the format and type, server generates the file.
In Java I am trying to download the file using the below code. The file is created, but the content is just the domain content and not the actual excel data.
URL url = new URL("http://www.my-domain.com/download/type/salary/format/excel");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
float totalDataRead = 0;
BufferedInputStream in = new BufferedInputStream(connection.getInputStream());
FileOutputStream fos = new FileOutputStream("c:\\test.xls");
BufferedOutputStream bout = new BufferedOutputStream(fos, 1024);
byte[] data = new byte[1024];
int i = 0;
while ((i = in.read(data, 0, 1024)) >= 0) {
totalDataRead = totalDataRead + i;
bout.write(data, 0, i);
}
bout.close();
in.close();

The content is whatever the server sent for that URL. You can't do anything about that from the client end. If it contained Javascript for example it won't get executed.

When you want to solve a problem you have to use the adequate tools to get the thing done. The adequate tools can be found at poi.apache.org. Have a look at Apache POI.

URL Connection (FTP) in Java - Simple Question

I have a simple question. I'm trying to upload a file to my ftp server in Java.
I have a file on my computer, and I want to make a copy of that file and upload it. I tried manually writing each byte of the file to the output stream, but that doesn't work for complicated files, like zip files or pdf files.
File file = some file on my computer;
String name = file.getName();
URL url = new URL("ftp://user:password#domain.com/" + name +";type=i");
URLConnection urlc = url.openConnection();
OutputStream os = urlc.getOutputStream();
//then what do I do?
Just for kicks, here is what I tried to do:
OutputStream os = urlc.getOutputStream();
BufferedReader br = new BufferedReader(new FileReader(file));
String line = br.readLine();
while(line != null && (!line.equals(""))) {
os.write(line.getBytes());
os.write("\n".getBytes());
line = br.readLine();
}
os.close();
For example, when I do this with a pdf and then try and open the pdf that I run with this program, it says an error occurred when trying to open the pdf. I'm guessing because I am writing a "\n" to the file? How do I copy the file without doing this?

Do not use any of the Reader or Writer classes when you're trying to copy the byte-for-byte exact contents of a binary file. Use these only for plain text! Instead, use the InputStream and OutputStream classes; they do not interpret the data at all, while the Reader and Writer classes interpret the data as characters. For example
OutputStream os = urlc.getOutputStream();
FileInputStreamReader fis = new FileInputStream(file);
byte[] buffer = new byte[1000];
int count = 0;
while((count = fis.read(buffer)) > 0) {
os.write(buffer, 0, count);
}
Whether your URLConnection usage is correct here, I don't know; using Apache Commons FTP (as suggested elsewhere) would be an excellent idea. Regardless, this would be the way to read the file.

Use a BufferedInputStream to read and BufferedOutputStream to write. Take a look at this post: http://www.ajaxapp.com/2009/02/21/a-simple-java-ftp-connection-file-download-and-upload/
InputStream is = new FileInputStream(localfilename);
BufferedInputStream bis = new BufferedInputStream(is);
OutputStream os =m_client.getOutputStream();
BufferedOutputStream bos = new BufferedOutputStream(os);
byte[] buffer = new byte[1024];
int readCount;
while( (readCount = bis.read(buffer)) > 0) {
bos.write(buffer, 0, readCount);
}
bos.close();

FTP usually opens another connection for data transfer.
So I am not convinced that this approach with URLConnection is going
to work.
I highly recommend that you use specialized ftp client. Apache commons
may have one.
Check this out
http://commons.apache.org/net/api/org/apache/commons/net/ftp/FTPClient.html