Develop Reference

saving file as .pdf as recieved in http response error

For my project i need to download a pdf file from google drive using java
I get my httpresponse code 200 and by using following method i store it in abc.pdf file
String url = "https://docs.google.com/uc?id="+fileid+"&export=download";
URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
// optional default is GET
conn.setRequestMethod("GET");
//add request header
conn.setRequestProperty("User-Agent", USER_AGENT);
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(conn.getInputStream()));
String inputLine;
OutputStream f0 = new FileOutputStream("C:\\Users\\Darshil\\Desktop\\abc.pdf",true);
while ((inputLine = in.readLine()) != null) {
//System.out.println(inputLine);
byte b[]=inputLine.getBytes();
//System.out.println(b);
f0.write(b);
}
in.close();
f0.close();
But when i try to open abc.pdf in my adobe reader x i get following error:
There was an error opening this document.The file is damaged and could not be repaired

You seem to be directly accessing the Google drive using Raw HTTP requests.
You may be better of using the Google Drive SDK. This link contains good examples to address the use cases you state in your question.
However if you do want to stick to your technique then you should not be using a BufferedReader.readLine(). This is because the PDF file is a binary finally that would depend upon the correct byte sequences to be preserved in order to be read correctly by the PDF reader software. Hopefully the below technique should help you:
//read in chunks of 2KB
byte[] buffer = new byte[2048];
int bytesRead = 0;
try(InputStream is = conn.getInputStream())
{
try(DataOutputStream os = new DataOutputStream(new FileOutputStream("file.pdf"))
{
while((bytesRead = is.read(buffer)) != -1)
{
os.write(buffer, 0, bytesRead);
}
}
}
catch(Exception ex)
{
//handle exception
}
Note that I am using the try-with-resources statement in Java 7
Hope this helps.

How to download a part of a file from URL in android?

I am trying to download a part of file given the download URL using setRequestProperty("Range","bytes=" + startbytes + "-" + endbytes); The following code snippet shows what I am trying to do.
protected String doInBackground(String... aurl) {
int count;
Log.d(TAG,"Entered");
try {
URL url = new URL(aurl[0]);
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
int lengthOfFile = connection.getContentLength();
Log.d(TAG,"Length of file: "+ lengthOfFile);
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
The problem is that, an exception is being raised, which says "Cannot set request property after connection is made". Please help me resolve this issue.

Option 1
If you do not need to know the content length:
[Beware, do not call the connection.getContentLength(). If you call that, you will get the exception. If you need to call it, then check the second option]
URL url = new URL(aurl[0]);
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
//Note that, response code will be 206 (Partial Content) instead of usual 200 (OK)
if(connection.getResponseCode() == HttpURLConnection.HTTP_PARTIAL){
//Your code here to read response data
}
Option 2
If you need to know the content length:
URL url = new URL(aurl[0]);
//First make a HEAD call to get the content length
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
connection.setRequestMethod("HEAD");
if(connection.getResponseCode() == HttpURLConnection.HTTP_OK){
int lengthOfFile = connection.getContentLength();
Log.d("ERF","Length of file: "+ lengthOfFile);
connection.disconnect();
//Now that we know the content lenght, make the GET call
connection =(HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
//Note that, response code will be 206 (Partial Content) instead of usual 200 (OK)
if(connection.getResponseCode() == HttpURLConnection.HTTP_PARTIAL){
//Your code here to read response data
}
}

Assuming you're using HTTP for the download, you'll want to use the HEAD http verb and RANGE http header.
HEAD will give you the filesize (if available), and then RANGE lets you download a byte range.
Once you have the filesize, divide it into roughly equal sized chunks and spawn download thread for each chunk. Once all are done, write the file chunks in the correct order.
If you don't know how to use the RANGE header, here's another SO answer that explains how: https://stackoverflow.com/a/6323043/1355166
[EDIT]
To make file into chunks use this, and start the downloading process,
private void getBytesFromFile(File file) throws IOException {
FileInputStream is = new FileInputStream(file); //videorecorder stores video to file
java.nio.channels.FileChannel fc = is.getChannel();
java.nio.ByteBuffer bb = java.nio.ByteBuffer.allocate(10000);
int chunkCount = 0;
byte[] bytes;
while(fc.read(bb) >= 0){
bb.flip();
//save the part of the file into a chunk
bytes = bb.array();
storeByteArrayToFile(bytes, mRecordingFile + "." + chunkCount);//mRecordingFile is the (String)path to file
chunkCount++;
bb.clear();
}
}
private void storeByteArrayToFile(byte[] bytesToSave, String path) throws IOException {
FileOutputStream fOut = new FileOutputStream(path);
try {
fOut.write(bytesToSave);
}
catch (Exception ex) {
Log.e("ERROR", ex.getMessage());
}
finally {
fOut.close();
}
}

Implement pause/resume in file downloading

I'm trying to implement pause/resume in my download manager, I search the web and read several articles and change my code according them but resume seems not working correctly, Any ideas?
if (!downloadPath.exists())
downloadPath.mkdirs();
if (outputFileCache.exists())
{
downloadedSize = outputFileCache.length();
connection.setAllowUserInteraction(true);
connection.setRequestProperty("Range", "bytes=" + downloadedSize + "-");
connection.setConnectTimeout(14000);
connection.connect();
input = new BufferedInputStream(connection.getInputStream());
output = new FileOutputStream(outputFileCache, true);
input.skip(downloadedSize); //Skip downloaded size
}
else
{
connection.setConnectTimeout(14000);
connection.connect();
input = new BufferedInputStream(url.openStream());
output = new FileOutputStream(outputFileCache);
}
fileLength = connection.getContentLength();
byte data[] = new byte[1024];
int count = 0;
int __progress = 0;
long total = downloadedSize;
while ((count = input.read(data)) != -1 && !this.isInterrupted())
{
total += count;
output.write(data, 0, count);
__progress = (int) (total * 100 / fileLength);
}
output.flush();
output.close();
input.close();

Okay problem fixed, here is my code for other users who wants to implement pause/resume:
if (outputFileCache.exists())
{
connection.setAllowUserInteraction(true);
connection.setRequestProperty("Range", "bytes=" + outputFileCache.length() + "-");
}
connection.setConnectTimeout(14000);
connection.setReadTimeout(20000);
connection.connect();
if (connection.getResponseCode() / 100 != 2)
throw new Exception("Invalid response code!");
else
{
String connectionField = connection.getHeaderField("content-range");
if (connectionField != null)
{
String[] connectionRanges = connectionField.substring("bytes=".length()).split("-");
downloadedSize = Long.valueOf(connectionRanges[0]);
}
if (connectionField == null && outputFileCache.exists())
outputFileCache.delete();
fileLength = connection.getContentLength() + downloadedSize;
input = new BufferedInputStream(connection.getInputStream());
output = new RandomAccessFile(outputFileCache, "rw");
output.seek(downloadedSize);
byte data[] = new byte[1024];
int count = 0;
int __progress = 0;
while ((count = input.read(data, 0, 1024)) != -1
&& __progress != 100)
{
downloadedSize += count;
output.write(data, 0, count);
__progress = (int) ((downloadedSize * 100) / fileLength);
}
output.close();
input.close();
}

It is impossible to tell what is wrong without some more information, however things to note:
You must make a HTTP/1.1 request (it's hard to tell from your sample code)
The server must support HTTP/1.1
The server will tell you what it supports with an Accept-Ranges header in the response
If-Range should be the etag the server gave you for the resource, not the last modified time
You should check your range request with something simple to test the origin actually supports the Range request first (like curl or wget )

It nay be that your server is taking to long to respond (more then the timeout limit) or this is also a fact that not all servers support pause - resume.
It is also a point to ponder that weather the file is downloaded through Http, https, ftp or udp.
Pausing" could just mean reading some of the stream and writing it to disk. When resuming you would have to use the headers to specify what is left to download.
you may try something like :
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
if(ISSUE_DOWNLOAD_STATUS.intValue()==ECMConstant.ECM_DOWNLOADING){
File file=new File(DESTINATION_PATH);
if(file.exists()){
downloaded = (int) file.length();
connection.setRequestProperty("Range", "bytes="+(file.length())+"-");
}
}else{
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
}
connection.setDoInput(true);
connection.setDoOutput(true);
progressBar.setMax(connection.getContentLength());
in = new BufferedInputStream(connection.getInputStream());
fos=(downloaded==0)? new FileOutputStream(DESTINATION_PATH): new FileOutputStream(DESTINATION_PATH,true);
bout = new BufferedOutputStream(fos, 1024);
byte[] data = new byte[1024];
int x = 0;
while ((x = in.read(data, 0, 1024)) >= 0) {
bout.write(data, 0, x);
downloaded += x;
progressBar.setProgress(downloaded);
}
and please try to sync things.

I would start debugging from this line:
connection.setRequestProperty("Range", "bytes=" + downloadedSize + "-");
As from the source code it is not possible to determine what downloadedSize is, it's hard to elaborate further, but the format should be bytes=from-to.
Anyway, I would suggest you to use Apache HttpClient to avoid common pitfalls. Here is a question from someone who uses Apache HttpClient on a similar topic and some sample code is provided.

I think you just need to delete the input.skip(downloadedSize) line. Setting the HTTP header for byte range means the server will skip sending those bytes.
Say you have a file that's 20 bytes long consisting of "aaaaabbbbbcccccddddd", and suppose the transfer is paused after downloading 5 bytes. Then the Range header will cause the server to send "bbbbbcccccddddd", you should read all of this content and append it to the file -- no skip(). But the skip() call in your code will skip "bbbbb" leaving "cccccddddd" to be downloaded. If you've already downloaded at least 50% of the file, then skip() will exhaust all of the input and nothing will happen.
Also, all of the things in stringy05's post apply. You should make sure the server supports HTTP/1.1, make sure the Range header is supported for the resource (dynamically generated content may not support it), and make sure the resource isn't modified using etag and modification date.

How to disable buffering in Java HttpURLConnection for multi-part form post?

(This is for a signed applet and I have decided against HTTPClient to keep my jar very small)
I am using HttpURLConnection to successfully upload a file from the user to a server using multi-part form post.
The problem is that HttpURLConnection is caching the data -- before sending it. So when I am reading from the file and writing to Outputstream, it is merely buffering the data -- and therefore my progress bar, that shows the upload status , is completely wrong. Howevere please note that the form post code works and the file does get uploaded correctly with return code of 200.
So how do I ensure that HttpURLConnection does not cache the data that I am sending to the server ?
Here is my source:
public UploadResponse send(String formPostUrlStr,String fileFieldName,File targetFile, Map<String, String> valuesMap, UploadStatusListener uploadStatusListener) throws Exception{
String sendStr=getBoundaryMessage(Boundary, valuesMap, fileFieldName, targetFile.getName(), valuesMap.get("content-type") );//"image/png") ;
System.out.println(" multi-part start \n "+ sendStr+ " multi-part end \n");
String lenstr=Long.toString((long)(sendStr.length()*2)+ targetFile.length());
System.out.println("Content-Length"+ lenstr);
//Content-Length
URL url= new URL(formPostUrlStr);
long startTime= System.currentTimeMillis();
HttpURLConnection s3Connection = (HttpURLConnection) url.openConnection();
System.out.println("opened url to "+ formPostUrlStr +", connection ok ..");
s3Connection.setRequestProperty("Content-Type", "multipart/form-data; boundary="
+ Boundary);
s3Connection.setRequestProperty("content-length", lenstr);
s3Connection.setDoOutput(true);
s3Connection.setDoInput(true);
s3Connection.setUseCaches(false);
s3Connection.setInstanceFollowRedirects(true);
s3Connection.setAllowUserInteraction(true);
s3Connection.setRequestProperty("User-Agent", "Mozilla/4.5");
if (uploadStatusListener != null) {
uploadStatusListener.statusUpdate(targetFile.length(), 0);
}
String debugStr= s3Connection.toString();
System.out.println("conmnection "+ debugStr);
DataOutputStream httpOut = new DataOutputStream(s3Connection.getOutputStream());
System.out.println("opened DataOutputStream ok ..");
httpOut.write(sendStr.getBytes());
//httpOut.flush();
System.out.println("httpOut.flush 1 ok ..");
FileInputStream uploadFileReader = new FileInputStream(targetFile);
long totalBytes = uploadFileReader.available();
if (uploadStatusListener != null) {
uploadStatusListener.statusUpdate(totalBytes, 0);
}
System.out.println(" uploading file with size "+ uploadFileReader.available());
int bufSize = 102400;
long availableBytesToRead;
long totalSent = 0;
while ((availableBytesToRead = uploadFileReader.available()) > 0) {
byte[] bufferBytesRead;
bufferBytesRead = availableBytesToRead >= bufSize ? new byte[bufSize]
: new byte[(int)availableBytesToRead];
int count = uploadFileReader.read(bufferBytesRead);
try{
httpOut.write(bufferBytesRead);
totalSent += ((long) count);
System.out.println(" wrote bytes = "+count+ ", total sent = "+ totalSent +", pendingSize"+ (availableBytesToRead-count) );
}
catch(IOException ioe){
System.out.println(" io exceotion e"+ ioe.getMessage());
throw ioe;
}
//httpOut.flush();
if (uploadStatusListener != null) {
uploadStatusListener.statusUpdate(totalBytes, totalSent);
}
}
// FILE DATA END
httpOut.write(("--" + Boundary + "--\r\n").getBytes());
// form end
httpOut.write(("--" + Boundary + "--\r\n").getBytes());
httpOut.flush();
httpOut.close();
long endTime= System.currentTimeMillis();
System.out.println("Completed Writing Data to S3 Connection in "+ (endTime-startTime)+"ms.,now waiting for rsponse code ");
int code=s3Connection.getResponseCode();
long endTime2= System.currentTimeMillis();
System.out.println("Completed Sendind Data to S3 in "+ (endTime2-startTime)+ "ms., rsponse code time "+ (endTime2-endTime)+"ms. ");
UploadResponse uploadResponse = new UploadResponse();
uploadResponse.setCode(code);
System.out.println(" response code : " + code);
StringBuilder response = new StringBuilder();
byte[] respBuffer = new byte[4096];
if (code > 300) {
if (code == 404) {
throw new Exception("Error 404");
}
BufferedReader err = new BufferedReader(
new InputStreamReader(s3Connection.getErrorStream()));
String ret;
StringBuffer buff = new StringBuffer();
while ((ret = err.readLine()) != null) {
buff.append(ret);
}
uploadResponse.setMessage(buff.toString());
System.out.println(" error :"+ buff.toString());
err.close();
} else {
BufferedReader inp = new BufferedReader(
new InputStreamReader(s3Connection.getInputStream()));
StringBuffer buff = new StringBuffer();
String ret;
while ((ret = inp.readLine()) != null) {
buff.append(ret);
}
inp.close();
uploadResponse.setMessage(buff.toString());
if(buff.toString().contains("fail"))
throw new Exception("Upload failed");
}
System.out.println(response.toString());
return uploadResponse;
}
}

I have the same problem.
I didn't find any other solution than writing my HTTP request on a raw Socket.
Did you find a better workaround ?
EDIT : I just did : we just have to use obj.setFixedLengthStreamingMode(12345) on the HttpURLConnection object obtained from url.openConnection(), where 12345 is the length of POST request body.

As a complementation for the answer that #Antares gave, there is another method setChunkedStreamingMode that is used when you don't know the content size in advance. So when you do a POST request, call that method on the connection:
HttpURLConnection connection = ...
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setChunkedStreamingMode(0);
connection.connect();
... connection.getOutputStream();
This will avoid the OutputStream to buffer the entire content before start to send.

HttpsURLConnection stalling out when getInputStream() is called

[Java 1.5; Eclipse Galileo]
HttpsURLConnection seems to stall when the getInputStream() method is called. I've tried using different websites to no avail (currently https://www.google.com). I should point out I'm using httpS.
The code below has been modified based on what I've learned from other StackOverflow answers. However, no solutions I've tried thus far have worked.
I'd be very grateful for a nudge in the right direction :)
public static void request( URL url, String query )
{
try{
HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
//connection.setReadTimeout( 5000 ); //<-- uncommenting this line at least allows a timeout error to be thrown
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
System.setProperty("http.keepAlive", "false");
connection.setRequestMethod( "POST" );
// setting headers
connection.setRequestProperty("Content-length",String.valueOf (query.length()));
connection.setRequestProperty("Content-Type","application/x-www-form-urlencoded"); //WAS application/x-www- form-urlencoded
connection.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)");
////////////////////////////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////////////////////////////
System.out.println( "THIS line stalls" + connection.getInputStream() );
////////////////////////////////////////////////////////////////////////////////////
}catch( Exception e ) {
System.out.println( e );
e.printStackTrace();
}
Typical errors look like:
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:782)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:739)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:681)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:626)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:983)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:234)
at https_understanding.HTTPSRequest.request(HTTPSRequest.java:60)
at https_understanding.Main.main(Main.java:17)

connection.setDoOutput(true);
This means that you have to open, write to, and close the connection's output stream before you attempt to read from its input stream. See the docs.

I reproduced the problem in Android 2.2: when downloading from a web-server over wireless and a HTTPS URL, the error is a socket "read time out" at URLConnection.getInputStream()
To fix it, use url.openStream() for the InputStream instead of connection.getInputStream()
Bonus: you can get the length of the file you're downloading so you can show a % complete indicator
code sample:
private final int TIMEOUT_CONNECTION = 5000;//5sec
private final int TIMEOUT_SOCKET = 30000;//30sec
file = new File(strFullPath);
URL url = new URL(strURL);
URLConnection ucon = url.openConnection();
//this timeout affects how long it takes for the app to realize there's a connection problem
ucon.setReadTimeout(TIMEOUT_CONNECTION);
ucon.setConnectTimeout(TIMEOUT_SOCKET);
//IMPORTANT UPDATE:
// ucon.getInputStream() often times-out over wireless
// so, replace it with ucon.connect() and url.openStream()
ucon.connect();
iFileLength = ucon.getContentLength();//returns -1 if not set in response header
if (iFileLength != -1)
{
Log.i(TAG, "Expected Filelength = "+String.valueOf(iFileLength)+" bytes");
}
//Define InputStreams to read from the URLConnection.
// uses 5KB download buffer
InputStream is = url.openStream();//ucon.getInputStream();
BufferedInputStream inStream = new BufferedInputStream(is, 1024 * 5);
outStream = new FileOutputStream(file);
bFileOpen = true;
byte[] buff = new byte[5 * 1024];
//Read bytes (and store them) until there is nothing more to read(-1)
int total=0;
int len;
int percentdone;
int percentdonelast=0;
while ((len = inStream.read(buff)) != -1)
{
//write to file
outStream.write(buff,0,len);
//calculate percent done
if (iFileLength != -1)
{
total+=len;
percentdone=(int)(total*100/iFileLength);
//limit the number of messages to no more than one message every 10%
if ( (percentdone - percentdonelast) > 10)
{
percentdonelast = percentdone;
Log.i(TAG,String.valueOf(percentdone)+"%");
}
}
}
//clean up
outStream.flush();//THIS IS VERY IMPORTANT !
outStream.close();
bFileOpen = false;
inStream.close();

Also don't set the content-length header. Java will do that for you.