I'm trying to download a large file from my Yahoo! web site server which apparently is setup (not by me) to disconnect downloads if they are not completed within 100 seconds. The file is small enough to usually successfully transfer. On the occasions when the data rate is slow and the download gets disconnected, is there a way to resume the URLConnection at the file offset where the disconnection occurred? Here's the code:
// Setup connection.
URL url = new URL(strUrl[0]);
URLConnection cx = url.openConnection();
cx.connect();
// Setup streams and buffers.
int lengthFile = cx.getContentLength();
InputStream input = new BufferedInputStream(url.openStream());
OutputStream output = new FileOutputStream(strUrl[1]);
byte data[] = new byte[1024];
// Download file.
for (total=0; (count=input.read(data, 0, 1024)) != -1; total+=count) {
publishProgress((int)(total*100/lengthFile));
output.write(data, 0, count);
Log.d("AsyncDownloadFile", "bytes: " + total);
}
// Close streams.
output.flush();
output.close();
input.close();
Try using a "Range" request header:
// Open connection to URL.
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
// Specify what portion of file to download.
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
// here "downloaded" is the data length already previously downloaded.
// Connect to server.
connection.connect();
Having done that, you can seek at a given point (just before the length of your download data, say X) and start writing the newly downloaded data there. Be sure to use the same value X for the range header.
Details about 14.35.2 Range Retrieval Requests
More details and source code can be found here
Here's an example code that you can use:
import java.io.*;
import java.net.*;
public class HttpUrlDownload {
public static void main(String[] args) {
String strUrl = "http://VRSDLSCEN001:80//DLS//lib//clics.jar";
String DESTINATION_PATH = "clics.jar";
int count = 0;
while (true) {
count++;
if (download(strUrl, DESTINATION_PATH) == true || count > 20) {
break;
}
}
}
public static boolean download(String strUrl, String DESTINATION_PATH) {
BufferedInputStream in = null;
FileOutputStream fos = null;
BufferedOutputStream bout = null;
URLConnection connection = null;
int downloaded = 0;
try {
System.out.println("mark ... download start");
URL url = new URL(strUrl);
connection = url.openConnection();
File file=new File(DESTINATION_PATH);
if(file.exists()){
downloaded = (int) file.length();
}
if (downloaded == 0) {
connection.connect();
}
else {
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
connection.connect();
}
try {
in = new BufferedInputStream(connection.getInputStream());
} catch (IOException e) {
int responseCode = 0;
try {
responseCode = ((HttpURLConnection)connection).getResponseCode();
} catch (IOException e1) {
e1.printStackTrace();
}
if (responseCode == 416) {
return true;
} else {
e.printStackTrace();
return false;
}
}
fos=(downloaded==0)? new FileOutputStream(DESTINATION_PATH): new FileOutputStream(DESTINATION_PATH,true);
bout = new BufferedOutputStream(fos, 1024);
byte[] data = new byte[1024];
int x = 0;
while ((x = in.read(data, 0, 1024)) >= 0) {
bout.write(data, 0, x);
}
in.close();
bout.flush();
bout.close();
return false;
} catch (IOException e) {
e.printStackTrace();
return false;
} finally {
if (in != null) {
try {
in.close();
} catch (IOException e) {
}
}
if (fos != null) {
try {
fos.close();
} catch (IOException e) {
}
}
if (bout != null) {
try {
bout.close();
} catch (IOException e) {
}
}
if (connection != null) {
((HttpURLConnection)connection).disconnect();
}
}
}
}
Related
I am downloading a zip file from dropbox. When it keeps downloading I measure the file size and get it is increasing its size with the Below code. It downloads whole 84M and after finishing download it turns into 0 bytes. What wrong am I actually doing?
public static void downloadDropBox(File file) {
String url = "https://www.dropbox.com/sh/jx4b2wvqg8d4ze1/AAA0J3LztkRc6FJ5tKy4dUKha?dl=1";
int bytesRead;
byte[] bytesArray = new byte[1024];
InputStream is = null;
FileOutputStream outputStream = null;
long progres = 0;
try {
URL fileUrl = new URL(url);
HttpURLConnection connection = (HttpURLConnection)fileUrl.openConnection();
connection.connect();
is = connection.getInputStream();
outputStream = new FileOutputStream(file);
while ((bytesRead = is.read(bytesArray, 0, 1024)) != -1) {
outputStream.write(bytesArray, 0, bytesRead);
}
}
} catch (Exception e) {
e.printStackTrace();
}
finally {
if (is != null) {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (outputStream != null) {
try {
outputStream.flush();
outputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
During download file:
After Finishing download file:
I am trying to download a video from a link, but it only downloads a small part of it, so it can't be watched at all. How would you download an entire video no matter how large the file is from a link?
try {
URL url;
byte[] buf;
int byteRead, byteWritten = 0;
url = new URL(fAddress);
outStream = new BufferedOutputStream(new FileOutputStream(destinationDir + "\\" + localFileName));
conn = url.openConnection();
is = conn.getInputStream();
buf = new byte[size];
while ((byteRead = is.read(buf)) != -1) {
outStream.write(buf, 0, byteRead);
byteWritten += byteRead;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
is.close();
outStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
It looks like server may redirect you to other location which your code doesn't handle. To get final location you can try method like (based on: http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/):
public static String getFinalLocation(String address) throws IOException{
URL url = new URL(address);
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
int status = conn.getResponseCode();
if (status != HttpURLConnection.HTTP_OK)
{
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
{
String newLocation = conn.getHeaderField("Location");
return getFinalLocation(newLocation);
}
}
return address;
}
Now you simply need to change
url = new URL(fAddress);
to
url = new URL(getFinalLocation(fAddress));
I got this code for downloading my files:
public static void download()
throws MalformedURLException, IOException
{
BufferedInputStream in = null;
FileOutputStream out = null;
try
{
in = new BufferedInputStream(new URL("https://www.dropbox.com/s/1uff8eeujplz4sf/files.zip?dl=1").openStream());
out = new FileOutputStream(System.getProperty("user.home") + "/Adasti/files.zip");
byte data[] = new byte[1024];
int count;
while ((count = in.read(data, 0, 1024)) != -1)
{
out.write(data, 0, count);
}
} catch (MalformedURLException e1)
{
e1.printStackTrace();
} catch (IOException e2)
{
e2.printStackTrace();
} finally
{
if (in != null)
in.close();
if (out != null)
out.close();
}
}
I want to print out the percentage of the download while its downloading. Is this possible with my method and if yes, how will I do it?
You have to get the file size before start the download. Be carefull, not always the server can give you the file size (for sample streaming or some file server). Try this:
/**
*
* #param remotePath
* #param localPath
*/
public static void download(String remotePath, String localPath) {
BufferedInputStream in = null;
FileOutputStream out = null;
try {
URL url = new URL(remotePath);
URLConnection conn = url.openConnection();
int size = conn.getContentLength();
if (size < 0) {
System.out.println("Could not get the file size");
} else {
System.out.println("File size: " + size);
}
in = new BufferedInputStream(url.openStream());
out = new FileOutputStream(localPath);
byte data[] = new byte[1024];
int count;
double sumCount = 0.0;
while ((count = in.read(data, 0, 1024)) != -1) {
out.write(data, 0, count);
sumCount += count;
if (size > 0) {
System.out.println("Percentace: " + (sumCount / size * 100.0) + "%");
}
}
} catch (MalformedURLException e1) {
e1.printStackTrace();
} catch (IOException e2) {
e2.printStackTrace();
} finally {
if (in != null)
try {
in.close();
} catch (IOException e3) {
e3.printStackTrace();
}
if (out != null)
try {
out.close();
} catch (IOException e4) {
e4.printStackTrace();
}
}
}
The call to this method is something like this:
download("https://www.dropbox.com/s/1uff8eeujplz4sf/files.zip?dl=1", System.getProperty("user.home") + "/files.zip");
We are getting some very weird behavior in Android. Our network stack (that talks to a REST server) works fine in almost all situations, except when we do a GET shortly after doing a larger POST. What appears to be happening is that the Output stream is not flushing, and ends up sending the last line that was in there when the new socket is opened. Please note, each connection is a new object created, so this is unexpected behavior. First, the error code that seems to point me to the output stream, these are from the server logs.
10.1.8.195 - - [07/Nov/2012:13:36:28 -0700] "POST /iou/lender HTTP/1.1" 200 28 "-" "Android"
10.1.8.195 - - [07/Nov/2012:13:36:36 -0700] "------------V2ymHFg03ehbqgZCaKO6jy" 400 173 "-" "-"
That attempt after should be a GET that then pulls the data from the server that includes the new entry added via the POST. However, all we get is again what appears to be the last line from the output stream from the POST. Here is our core code for the network stack, if more of the surrounding code is needed, let me know.
public Object serverConnect(String url, String method,
Hashtable<String, Object> params) {
HttpConnection c = null;
InputStream is = null;
OutputStream out = null;
ByteArrayOutputStream postDataByteArrayImage = new ByteArrayOutputStream();
byte[] data;
String boundry = "----------V2ymHFg03ehbqgZCaKO6jy";
try {
if (!url.startsWith("/")) {
url = "/" + url;
}
String uri = Control.URL_Secure + Control.dtserver + ":"
+ Control.port + url;
ByteArrayOutputStream postDataByteArray = new ByteArrayOutputStream();
params.put("sessionId", Control.sessionId);
if (method.equals("GET")) {
uri = uri + "?";
Enumeration enumParams = params.keys();
while (enumParams.hasMoreElements()) {
if (!uri.endsWith("?")) {
uri = uri + "&";
}
String key = (String) enumParams.nextElement();
uri = uri
+ key
+ "="
+ java.net.URLEncoder.encode((String) params
.get(key));
}
} else if (method.equals("POST")) {
Enumeration enumParams = params.keys();
postDataByteArray.write(("--").getBytes());
postDataByteArray.write((boundry).getBytes());
postDataByteArray.write(("\r\n").getBytes());
while (enumParams.hasMoreElements()) {
String key = (String) enumParams.nextElement();
if (!key.equals("image")){
postDataByteArray
.write(("Content-Disposition: form-data; name=\"")
.getBytes());
postDataByteArray.write((key).getBytes());
postDataByteArray.write(("\"").getBytes());
postDataByteArray.write(("\r\n\r\n").getBytes());
postDataByteArray.write(((String) params.get(key))
.getBytes());
postDataByteArray.write(("\r\n").getBytes());
postDataByteArray.write(("--").getBytes());
postDataByteArray.write(boundry.getBytes());
postDataByteArray.write(("\r\n").getBytes());
}
}
postDataByteArray.close();
}
Log.i("URL", uri);
URL urltoConenct = new URL(uri);
URLConnection connection = urltoConenct.openConnection();
HttpURLConnection urlConnection = (HttpURLConnection) connection;
URLConnection.setDefaultRequestProperty("Method", method); // default
urlConnection.setRequestProperty("User-Agent", "Android");
if (method.equals("POST")) {
urlConnection.setDoOutput(true);
urlConnection.setFixedLengthStreamingMode(postDataByteArray.toByteArray().length + postDataByteArrayImage.toByteArray().length);
urlConnection.setRequestProperty("Content-Type",
"multipart/form-data; boundary=" + boundry);
out = urlConnection.getOutputStream();
out.write(postDataByteArray.toByteArray());
out.write(postDataByteArrayImage.toByteArray());
out.close();
}
int response = 0;
try {
response = urlConnection.getResponseCode();
} catch (IOException e) {
if (e.toString()
.equals("java.io.IOException: Received authentication challenge is null"))
throw new RESTException(401, "Invalid Phone or Pin");
else
throw e;
}
if (response == HttpURLConnection.HTTP_OK) {
is = urlConnection.getInputStream();
if (is == null) {
return new IOException(
"Cannot open HTTP InputStream, aborting");
}
ByteArrayOutputStream bo = new ByteArrayOutputStream();
int ch;
int count = 0;
while ((ch = is.read()) != -1) {
bo.write(ch);
count++;
}
data = bo.toByteArray();
return new String(data);
} else if (response == 500) {
return new RESTException(500, "Internal server error");
} else {
RESTException x = new RESTException();
x.setCode(response);
try {
is = urlConnection.getInputStream();
if (is == null) {
x.setMessage("Unable to retrieve message");
return x;
}
ByteArrayOutputStream bo = new ByteArrayOutputStream();
int ch;
int count = 0;
while ((ch = is.read()) != -1) {
bo.write(ch);
count++;
}
data = bo.toByteArray();
String output = new String(data);
JSONObject obj;
try {
obj = new JSONObject(output);
JSONObject err = obj.getJSONArray("errors")
.getJSONObject(0);
x.setMessage(err.getString("message"));
} catch (JSONException e) {
Log.e("stuff", output);
// TODO Auto-generated catch block
e.printStackTrace();
}
} catch (FileNotFoundException e) {
// Damn you android! I'm using a REST service here, stop
// trying to interpret my errors!
x.setMessage("Unable to retrieve message");
}
return x;
}
} catch (Exception x) {
x.printStackTrace();
/*
* if (!retried && x.toString().equals(
* "java.io.IOException: Persistent connection dropped after first chunk sent, cannot retry"
* )) { retry = true; } if (!retry) { return x; }
*/
return x;
} finally {
try {
out.close();
} catch (Exception x) {
}
try {
is.close();
} catch (Exception x) {
}
try {
c.close();
} catch (Exception x) {
}
params.clear();
}
// return null;
}
After a very long time of frustration, we discovered that Android tries to keep a connection alive even if you manually call .close() on the connection. This worked fine for our GET methods, but POST methods left the socket in a state that it couldn't then process a GET. Adding the following fixed all our problems:
urlConnection.setRequestProperty("connection", "close");
I have this method that downloads .csv files from yahoo finance and saves them locally. It is accessed during a loop so it is downloading many files from a list. However sometimes a symbol is entered incorrectly, no longer exists, or the connection times out. How can I amend this method so that connection time outs are retried and incorrect symbols (meaning the url does not work) are just skipped over without ending the program?
public static void get_file(String symbol){
OutputStream outStream = null;
URLConnection uCon = null;
InputStream is = null;
String finance_url = "http://ichart.finance.yahoo.com/table.csv?s="+symbol;
String destination = "C:/"+symbol+"_table.csv";
try {
URL Url;
byte[] buf;
int ByteRead,ByteWritten=0;
Url= new URL(finance_url);
outStream = new BufferedOutputStream(new FileOutputStream(destination));
uCon = Url.openConnection();
is = uCon.getInputStream();
buf = new byte[size];
while ((ByteRead = is.read(buf)) != -1) {
outStream.write(buf, 0, ByteRead);
ByteWritten += ByteRead;
}
}catch (Exception e) {
System.out.println("Error while downloading "+symbol);
e.printStackTrace();
}finally {
try {
is.close();
outStream.close();
}catch (IOException e) {
e.printStackTrace();
}
}
}
Why not call the method again when an exception is thrown. You can narrow down the exception type to indicate when a retry should be initiated.
public static void get_file(String symbol){
OutputStream outStream = null;
URLConnection uCon = null;
InputStream is = null;
String finance_url = "http://ichart.finance.yahoo.com/table.csv?s="+symbol;
String destination = "C:/"+symbol+"_table.csv";
try {
URL Url;
byte[] buf;
int ByteRead,ByteWritten=0;
Url= new URL(finance_url);
outStream = new BufferedOutputStream(new FileOutputStream(destination));
uCon = Url.openConnection();
is = uCon.getInputStream();
buf = new byte[size];
while ((ByteRead = is.read(buf)) != -1) {
outStream.write(buf, 0, ByteRead);
ByteWritten += ByteRead;
}
}catch (Exception e) {
getFile(symbol);
}finally {
try {
is.close();
outStream.close();
}catch (IOException e) {
e.printStackTrace();
}
}
}