I'm trying to implement pause/resume in my download manager, I search the web and read several articles and change my code according them but resume seems not working correctly, Any ideas?
if (!downloadPath.exists())
downloadPath.mkdirs();
if (outputFileCache.exists())
{
downloadedSize = outputFileCache.length();
connection.setAllowUserInteraction(true);
connection.setRequestProperty("Range", "bytes=" + downloadedSize + "-");
connection.setConnectTimeout(14000);
connection.connect();
input = new BufferedInputStream(connection.getInputStream());
output = new FileOutputStream(outputFileCache, true);
input.skip(downloadedSize); //Skip downloaded size
}
else
{
connection.setConnectTimeout(14000);
connection.connect();
input = new BufferedInputStream(url.openStream());
output = new FileOutputStream(outputFileCache);
}
fileLength = connection.getContentLength();
byte data[] = new byte[1024];
int count = 0;
int __progress = 0;
long total = downloadedSize;
while ((count = input.read(data)) != -1 && !this.isInterrupted())
{
total += count;
output.write(data, 0, count);
__progress = (int) (total * 100 / fileLength);
}
output.flush();
output.close();
input.close();
Okay problem fixed, here is my code for other users who wants to implement pause/resume:
if (outputFileCache.exists())
{
connection.setAllowUserInteraction(true);
connection.setRequestProperty("Range", "bytes=" + outputFileCache.length() + "-");
}
connection.setConnectTimeout(14000);
connection.setReadTimeout(20000);
connection.connect();
if (connection.getResponseCode() / 100 != 2)
throw new Exception("Invalid response code!");
else
{
String connectionField = connection.getHeaderField("content-range");
if (connectionField != null)
{
String[] connectionRanges = connectionField.substring("bytes=".length()).split("-");
downloadedSize = Long.valueOf(connectionRanges[0]);
}
if (connectionField == null && outputFileCache.exists())
outputFileCache.delete();
fileLength = connection.getContentLength() + downloadedSize;
input = new BufferedInputStream(connection.getInputStream());
output = new RandomAccessFile(outputFileCache, "rw");
output.seek(downloadedSize);
byte data[] = new byte[1024];
int count = 0;
int __progress = 0;
while ((count = input.read(data, 0, 1024)) != -1
&& __progress != 100)
{
downloadedSize += count;
output.write(data, 0, count);
__progress = (int) ((downloadedSize * 100) / fileLength);
}
output.close();
input.close();
}
It is impossible to tell what is wrong without some more information, however things to note:
You must make a HTTP/1.1 request (it's hard to tell from your sample code)
The server must support HTTP/1.1
The server will tell you what it supports with an Accept-Ranges header in the response
If-Range should be the etag the server gave you for the resource, not the last modified time
You should check your range request with something simple to test the origin actually supports the Range request first (like curl or wget )
It nay be that your server is taking to long to respond (more then the timeout limit) or this is also a fact that not all servers support pause - resume.
It is also a point to ponder that weather the file is downloaded through Http, https, ftp or udp.
Pausing" could just mean reading some of the stream and writing it to disk. When resuming you would have to use the headers to specify what is left to download.
you may try something like :
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
if(ISSUE_DOWNLOAD_STATUS.intValue()==ECMConstant.ECM_DOWNLOADING){
File file=new File(DESTINATION_PATH);
if(file.exists()){
downloaded = (int) file.length();
connection.setRequestProperty("Range", "bytes="+(file.length())+"-");
}
}else{
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
}
connection.setDoInput(true);
connection.setDoOutput(true);
progressBar.setMax(connection.getContentLength());
in = new BufferedInputStream(connection.getInputStream());
fos=(downloaded==0)? new FileOutputStream(DESTINATION_PATH): new FileOutputStream(DESTINATION_PATH,true);
bout = new BufferedOutputStream(fos, 1024);
byte[] data = new byte[1024];
int x = 0;
while ((x = in.read(data, 0, 1024)) >= 0) {
bout.write(data, 0, x);
downloaded += x;
progressBar.setProgress(downloaded);
}
and please try to sync things.
I would start debugging from this line:
connection.setRequestProperty("Range", "bytes=" + downloadedSize + "-");
As from the source code it is not possible to determine what downloadedSize is, it's hard to elaborate further, but the format should be bytes=from-to.
Anyway, I would suggest you to use Apache HttpClient to avoid common pitfalls. Here is a question from someone who uses Apache HttpClient on a similar topic and some sample code is provided.
I think you just need to delete the input.skip(downloadedSize) line. Setting the HTTP header for byte range means the server will skip sending those bytes.
Say you have a file that's 20 bytes long consisting of "aaaaabbbbbcccccddddd", and suppose the transfer is paused after downloading 5 bytes. Then the Range header will cause the server to send "bbbbbcccccddddd", you should read all of this content and append it to the file -- no skip(). But the skip() call in your code will skip "bbbbb" leaving "cccccddddd" to be downloaded. If you've already downloaded at least 50% of the file, then skip() will exhaust all of the input and nothing will happen.
Also, all of the things in stringy05's post apply. You should make sure the server supports HTTP/1.1, make sure the Range header is supported for the resource (dynamically generated content may not support it), and make sure the resource isn't modified using etag and modification date.
Related
I am trying to upload a file on a server and display how many bytes are uploaded per second by this way:
public void RunUploadTest () {
try {
URL url = new URL(serverLink);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("content-type", "video/mp4");
for(int i =0; i<10; i++) {
FileInputStream fis = new FileInputStream(myFileVideo);
DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
int bytesWrite = 0;
byte[] buffer = new byte[512];
Stopwatch timer = new Stopwatch();
int read;
while ((read = fis.read(buffer)) != -1&& timer.elapsedTime()<1000) {
dos.write(buffer, 0, read);
dos.flush();
bytesWrite++;
}
Log.d("Upload", "Bytes written: " + bytesWrite*512);
}
fis.close();
} catch (IOException e){
e.printStackTrace();
}
}
The problem is that is not calculating how many bytes are uploaded as I expect.
Do you have any idea why is not working?
If you have 10 times the same bytes written, there are 2 options:
your file is written in less than 1 second;
your elapsedTime method, returns in seconds(for example) and not in milliseconds
You probably don't need that for loop too, it makes you read the file 10 times.
I would rewrite your code like this:
public void RunUploadTest () {
try {
URL url = new URL(serverLink);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("content-type", "video/mp4");
FileInputStream fis = new FileInputStream(myFileVideo);
DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
int bytesWrite = 0;
byte[] buffer = new byte[512];
int read;
while((read = fis.read(buffer))) {
Stopwatch timer = new Stopwatch();
while (timer.elapsedTime()<1) { //my guess here is that this method returns in seconds
dos.write(buffer, 0, read);
dos.flush();
bytesWrite += read; //taken from Yoni Gross answer
}
Log.d("Upload", "Bytes written: " + bytesWrite);
}
fis.close();
} catch (IOException e){
e.printStackTrace();
}
}
There are few problems with your code:
1. You are not counting right
This is a relatively minor issue, but the last buffer you read/write will probably not going to be 512 bytes long, so your calculation will not be precise.
you should change
bytesWrite++;
}
Log.d("Upload", "Bytes written: " + bytesWrite*512);
to
bytesWrite += read;
}
Log.d("Upload", "Bytes written: " + bytesWrite);
2. You are stopping the upload after 1 second anyway
The following condition:
while ((read = fis.read(buffer)) != -1&& timer.elapsedTime()<1000)
Will stop either when the file has finished reading, or 1 second elapsed. Unless a very small file, most likely file upload will stop after 1 second.
I'm not completely sure if you want an update every second, or just the average.
In case of average you should just change the while condition to:
while (read = fis.read(buffer))
And change log to:
float uploadRate = (float)(bytesWrite / timer.elapsedTime()) / 1000.0f;
String logString = String.format("Upload rate: %.2f bytes per second", uploadRate)
Log.d("Upload", logString);
In case you do want to print progress every second, you'll need to do something more fancy and expensive, like:
while (read = fis.read(buffer)) {
dos.write(buffer, 0, read);
dos.flush();
bytesWrite += read;
if (timer.shouldPrintSecondsProgress(){
Log.d("Upload", "Bytes written: " + bytesWrite);
}
}
StopWatch.shouldPrintSecondsProgress should be something like:
public boolean shouldPrintSecondsProgress(){
int currentSecond = elapsedTime() / 1000;
if (currentSecond == nextUpdate){ //I've ignored the > case, which is probably an error in our case
nextUpdate++; //this is an int field, marking the next second cause update
return true;
}
return false;
}
3. It's possible you measure only 'send' and not 'upload'
While you are sending all your data and flushing the request stream, some layer of the networking stack might have buffer/async behavior. Actually the roundtrip alone might effect calculation correctness.
It best if you close the output stream prior to measure time elapsed, and/or read the response from server (which will ensure upload is finished).
I'm trying to download a song file. The following code (well, the original code, this is just an example of what I'm doing) is working perfectly on an Asha 310 device. However, on the newer Asha 501 devices, the resulting downloaded file is much larger than the actual file size.
A 2.455.870 byte file ends up downloading 2.505.215 bytes if I use a 512 buffer, and it doesn't load either. Using a 4096 buffer, the file ends up being 3.342.335 bytes in size!!
What could be the reason for this happening? It's working perfectly on the other phone, and I'm using very reasonable buffers.
downloadedFile = (FileConnection) Connector.open(saveLocation+"testing.m4a", Connector.READ_WRITE);
if (!downloadedFile.exists()) {
downloadedFile.create();
}
ops = downloadedFile.openOutputStream();
hc = (HttpConnection) Connector.open(url);
hc.setRequestMethod(HttpsConnection.POST);
hc.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
String postData = "sid=" + session.sid + "&fileid=" + file.getId();
byte[] request_body = postData.getBytes();
DataOutputStream dos = null;
dos = hc.openDataOutputStream();
for (int i = 0; i < request_body.length; i++) {
dos.writeByte(request_body[i]);
}
byte[] buf = new byte[512];
dis = hc.openInputStream();
int downloadSize = 0;
while (dis.read(buf) != -1) {
ops.write(buf, 0, buf.length);
downloadedSize += buf.length;
}
Turns out the buffer isn't being fully filled out, so the rest of each buffer that isn't filled out is junk. Which explains why when a bigger buffer is set, the file is bigger, as it has more junk.
http://developer.nokia.com/Community/Discussion/showthread.php/244179-Download-size-larger-than-available-stream-file-size-(Asha-501)
int len;
while((len=dis.read(buf))!=-1)
{
ops.write(buf,0,len);
downloadedSize += len;
}
Edit: It was working on the older phones because they filled out the entire buffer all the time with actual data. The newer devices don't.
I'm stuck trying to stop a download initiated with HtmlUnit after a certain size was reached. The InputStream
InputStream input = button.click().getWebResponse().getContentAsStream();
downloads the complete file correctly. However, seems like using
OutputStream output = new FileOutputStream(fileName);
int bytesRead;
int total = 0;
while ((bytesRead = input.read(buffer)) != -1 && total < MAX_SIZE) {
output.write(buffer, 0, bytesRead);
total += bytesRead;
System.out.print(total + "\n");
}
output.flush();
output.close();
input.close();
somehow downloads the file to a different location (unknown to me) and once finished copies the max size into the file "fileName". No System.out is printed during this process. Interestingly, while running the debugger in Netbeans and going slowly step-by-step, the total is printed and I get the MAX_SIZE file.
Varying the buffer size in a range between 1024 to 102400 didn't make any difference.
I also tried Commons'
BoundedInputStream b = new BoundedInputStream(button.click().getWebResponse().getContentAsStream(), MAX_SIZE);
without success.
There's this 2,5 years old post, but I couldn't figure out how to implement the proposed solution.
Is there something I'm missing in order to stop the download at MAX_SIZE?
(Exceptions handling and other etcetera omitted for brevity)
There is no need to use HTMLUnit for this. Actually, using it to such a simple task is a very overkill solution and will make things slow. The best approach I can think of is the following:
final String url = "http://yoururl.com";
final String file = "/path/to/your/outputfile.zip";
final int MAX_BYTES = 1024 * 1024 * 5; // 5 MB
URLConnection connection = new URL(url).openConnection();
InputStream input = connection.getInputStream();
byte[] buffer = new byte[4096];
int pendingRead = MAX_BYTES;
int n;
OutputStream output = new FileOutputStream(new File(file));
while ((n = input.read(buffer)) >= 0 && (pendingRead > 0)) {
output.write(buffer, 0, Math.min(pendingRead, n));
pendingRead -= n;
}
input.close();
output.close();
In this case I've set a maximum download size of 5 MB and a buffer of 4 KB. The file will be written to disk in every iteration of the while loop, which seems to be what you're looking for.
Of course, make sure you handle all the needed exceptions (eg: FileNotFoundException).
I am trying to upload some bytes to the server for 15 seconds.I have written the following code to write the bytes to output stream :
long uploadedBytes=0;
ByteArrayInputStream byteArrayInputStream=null;
OutputStream outputStream=null;
try {
byte[] randomData=generateBinData(5*1024);
byte[] bytes = new byte[(int) 1024 * 5];
URL url = new URL(urls[0]);
HttpURLConnection connection =
(HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setRequestMethod("POST");
connection.setRequestProperty("Connection", "Keep-Alive");
outputStream = connection.getOutputStream();
byteArrayInputStream = new ByteArrayInputStream(randomData);
long startTime=System.currentTimeMillis();
while(byteArrayInputStream.read(bytes) > 0
&& timeDiff < 15000) {
outputStream.write(bytes, 0, bytes.length);
uploadedBytes += bytes.length;
byteArrayInputStream = new ByteArrayInputStream(randomData);
timeDiff = System.currentTimeMillis() - startTime;
int progress=(int)(timeDiff *100 / 15000);
publishProgress(progress);
}
But the progress for the above upload is running very fast and it shows large amount of bytes uploaded within seconds.Which is not according to my 2g mobile network connection.
For example it shows :
uploadedBytes =9850880 and with time difference(timeDiff) = 3 sec.
if i run the same code for 15 seconds it terminates the whole application.
Please help me to find where i am goind wrong.
thanks ...waiting for reply
Unless you set chunked or streaming transfer mode, HttpURLConnection buffers all the output before sending any of it, so it can get a Content-Length. So what you're seeing is the progress of the buffering, not of the transfer. Set chunked transfer mode and you will see a difference.
Your copy loop is wrong. It should be like this:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Your code will probably work in this specific case but that's not a reason not to get it right for all cases.
check your random byte length. i think the generateBinData() method is not generating 5Kb of data.
sure the uploadedBytes is huge. say, if a write to outputstream takes 10 milisec to write 5Kb(5*1024) of data,in 3 second you should able to write only 153600 bytes.
Reason for app termination - check if any read operation throws exception.
I'm trying to make a partial GET request and I expect a 206 response, but I'm still getting 200 OK. How can I make it responded with a 206?
Here is the code I wrote:
HttpURLConnection urlConn = (HttpURLConnection) url.openConnection();
int start = Integer.parseInt(args[1]);
int end = Integer.parseInt(args[2]);
urlConn.setRequestMethod("GET");
urlConn.setRequestProperty("Range", "bytes=" + start + "-" + end);
if (urlConn.getResponseCode() != HttpURLConnection.HTTP_PARTIAL)
System.out.println ("File cannot be downloaded.");
else
{
String filepath = url.getPath();
String filename = filepath.substring (filepath.lastIndexOf('/') + 1);
BufferedInputStream in = new BufferedInputStream (urlConn.getInputStream());
BufferedOutputStream out = new BufferedOutputStream (new FileOutputStream (filename));
int temp;
while ((temp = in.read()) != -1)
{
out.write ((char)temp);
}
out.flush();
out.close();
System.out.println ("File downloaded successfully.");
}
How can I make it responded with a 206?
You can't force a server to respect your Range headers. The HTTP 1.1 spec says:
"A server MAY ignore the Range header."
You comment:
I know the server does not ignore it. At least shouldn't
Well apparently, the server >>IS<< ignoring the header. Or alternatively, the range that you are requesting encompasses the entire document.