I want to achieve that when downloading, it can be paused, stopped by user. How can I achieve this with DownloadManager?
You can remove a download from the queue. Then you can start it up again, passing the Range HTTP Header when re-adding it to the queue (like TryTryAgain mentioned). I don't believe this currently can be done more easily.
It may be that "pausing" isn't exactly necessary for you? As DownloadManager actually doesn't need to pause to be resumed.
Instead you may be getting tripped up with resuming? See if this helps:
how to resume an interrupted download Where the solution is implementing a Range request header...
// Open connection to URL.
HttpURLConnection connection =
(HttpURLConnection) url.openConnection();
// Specify what portion of file to download.
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
// here "downloaded" is the data length already previously downloaded.
// Connect to server.
connection.connect();
Unless you need to actually allow "pausing" (because of switching file download priority, maintaining a list or something else more complicated) and not just provide resuming capability, that should help.
Do you mean to do this in Web Application ?
I am also thinking about the same thing. This might have been possible with flash or sliverlight etc., but in pure HTML?
On the server side we can use RandomAccessFileReader(Apache) or similar FileReader (in bytes) to access file and write to output stream. We can remember how much bytes was downloaded in previous session by the user too. But the problem is there is no standard way to save file to disk in the client's local file system. You can use ActiveX Object in IE to write some file provided it is a sequential (plain text) file and not binary file(like mp3, flv etc). But this ActiveX is not a solution. So you may need to design something like 'Downloader' for you website. Just like Youtube Downloader.
Related
I am developing an API in Java. It is basically a java servlet that returns content in json (application/json). Using a Tomcat server. One of the field in the response is supposed to be a link to a downloadable .txt file.
I wonder what is the best way to deliver this file:
Generating this file on every request seems to me killer, even having some cron to clean directories with files
Any way to give a temporary link only while that request for a period without saving to the file system?
Thank you.
If you say writing to the file system would kill your application, then I deduce from that that your IO performance is too weak for that, right? I mean, if you even would not have the storage capacity for that, then your infrastructure is not suitable for your application at all. I can only see four other ways for solving that problem (but maybe there are more, my list is not exclusive):
Store the text file in a database. The database should also store timeout information. Good if there are more than 1 application servers and there is a load balancer in front of them (but all application servers share the same database).
Store the text file in RAM, maybe using a cache library which does the cleanup tasks automatically for you - but be aware that a cache library will usually not guarantee a minimum storage time for each file.
Do not store the text file at all, but create it just when it is requested (no idea if that is possible in your application).
Do not provide a link to the text file, but directly include its content in the json answer (of course it would then be escaped as a JSon String), which means your server can directly forget about it when the answer has been sent, but the client _must_ download it without checking if it needs the file or not.
I'm currently using commons-net library for FTP client in my app. I have to download from remote server some files, by some criteria based on the file name. This is a very simplified and reduced version of my actual code (because I do some checks and catch all possible exceptions), but the essence is there:
//ftp is FTPClient object
//...
files = ftp.listFiles();
for (FTPFile ftpFile : files) {
String name = ftpFile.getName();
if(conformsCriteria(name)) {
String path = outDirectory + File.separatorChar + name;
os = new FileOutputStream(path);
ftp.retrieveFile(name, os);
}
}
Now, what I noticed is that when I run this code, wait a few seconds, and then plug out network cable, output directory contains some "empty" files plus the files actually downloaded, which leads me to believe that this method is working somewhat asynchronously... But then again, some files are downloaded (size > 0KB), and there are these empty files (size = 0KB), which leads me to believe that it is still serialized download... Also, function retrieveFile() returns, I quote documentation:
True if successfully completetd, false if not
What I need is serialized download, because I need to log every unsuccessful download.
What I saw browsing through the commons-net source is that, if I'm not wrong, new Socket is created for each retrieveFile() call.
I'm pretty confused about this, so If someone could explain what is actually happening, and offer solution with this library, or recommend some other FTP java library that supports blocking download per file, that would be nice.
Thanks.
You could just use the java.net.URLConnection class that has been present forever. It should know how to handle FTP URLs just fine. Here is a simple example that should give the blocking behavior that you are looking for.
The caveat is that you have to manage the input/output streams yourself, but this should be pretty simple.
Ok, to briefly answer this in order not to confuse people who might see this question.
Yes, commons-net for FTP is working as I thought it would, that is, retrieveFile() method blocks until it's finished with the download.
It was (of course) my own "mistake" in the code that let me think otherwise.
I'm building a Java application that will allow our users to load a list of files and have those files transferred to our server for video encoding. I've already built an API for managing the files before and after they've been transferred, but I need to decide on a good transfer protocol for actually moving the files.
Right now I'm leaning towards using the Apache Commons Net ( see: http://commons.apache.org/net/ ) package along with FTP to move the files from the client computer to the server. Once there I'll use secure API calls to move the files to wherever they need to go.
Is this the best route? Is there a better way to reliably transfer large (1 GB+) files? Is there a way to resume a broken download using this methodology? I'd like to avoid traditional HTTP POST requests as they're unreliable and cannot resume broken uploads.
Thanks!
You didn't mention if using Amazon S3 is an option for your solution, but they do offer native partial upload support. The basic workflow is:
Create an upload-placeholder and hold on to the response key
Upload chunks -- can be concurrent and retried as necessary
Use the response key to combine the chunks into a single file
Their SDK offers built-in file slicing and chunk upload.
Even if S3 is not the final location, you could use S3 as an upload holding pen and download the file at your convenience for permanent storage.
Your understanding of HTTP post is not exactly correct. HTTP standard doesn't restrict range requests to only GET method - one can use them with POST or PUT as well.
Also if you have control over both client and server-side script, you can post both data chunk and the StartAt position as a separate parameter. On the server you check the parameter and append the received data at specified position.
I have an assignment about uploading and downloading a file to a server. I managed to do the uploading part using Java Sockets however I am having a hard time doing the downloading part. I should use Range: for downloading parellel. In my request, I should have the Range: header. But I don't understand how I will receive the file with that HTTP GET request. All the examples I have seen was about uploading a file. I already did it. I can upload .exe, image, .pdf, anything and when I download them back (by my browser), there are no errors. Can you help me with the downloading part? Can you give me an example beacuse I really didn't get it.
You need to read the HTTP response from the same socket on which you put the request. As a starting point, just print it out. When you are familiar with it, start parsing it. The file will be in there.
This doesn't directly answer your question, but it is (IMO) worth saying anyway ...
If your homework assignment doesn't specifically tell you to use a socket directly, there are simpler, and better ways of doing HTTP file upload and download in Java:
Using java.net.URL.openConnection() on an "http:" url will give you an HttpURLConnection that you can use to make GET, PUT, POST and so on requests to the remote server. This takes care of the basic HTTP protocol stuff for you.
The Apache HttpClient libraries do the same thing, but in a more sophisticated way, with more options and more hooks for things like handling content (including forms and MIME multiparts), connection and credential management, proxying and route finding and so on.
If the aim of your homework exercise is to teach you practical ways to talk to remote servers, then using these classes is far more practical than trying to implement a subset of the HTTP protocol from the socket level up.
(Of course, the aim could be to give you a deeper understanding of the HTTP protocol at the "wire" level ... which would make your current approach the right one.)
Thank you I did it. I used a byte array and read the input stream and wrote it to a file using fileinputstream and fileoutputstream
I have a servlet that allows users to download (potentially large) zip files from a web page. If the user clicks on a link to download a zip file, code similar to the following is executed in the servlet:
response.setContentType("application/zip");
response.setHeader("Content-disposition", "attachment; filename=foo.zip");
response.setHeader("Pragma", "");
response.setHeader("Cache-Control", "no-store");
ZipOutputStream out = new ZipOutputStream(response.getOutputStream());
// write entries to the zip file...
...
out.close()
However, if the user refreshes or navigates away from the page after the download begins and before it completes (in Firefox 3.5.7), the download will fail. The following error pops up:
C:\blah\foo.zip.part could not be
saved, because the source file could
not be read.
Try again later, or contact the server
administrator.
Any ideas on how I can make sure the download continues in this case?
UPDATE: The link that initiates the download is a plain vanilla link. Interestingly, the behavior is different on IE. Clicking on links elsewhere on the site (from the currently loaded screen) seem not to load (the browser status bar says "Waiting for https://mysite/clicked_linky.do..."), blocking until the download completes. Typing a different URL into the address bar or using a shortcut/favorite link navigates away from the page, but the download continues as expected. Only Firefox seems to display the exact behavior I described above, although the IE blocking is not optimal.
This should in fact not happen. The download counts as a separate request which is supposed to be run in the background independently from the parent page once invoked. How exactly are you firing the download request? By a plain vanilla link or a link which (incorrectly) fires an ajaxical request to run the download?
At any way, you at least clearly want to be able to resume downloads. In this case you need to send at least the Accept-Ranges, ETag and Last-Modified response headers along the download accordingly. The client can then ask to resume the download by sending the If-Range and Range request headers with respectively the file identifier and a specified byte range which you could use in combination with RandomAccessFile to send the remaining bytes. You can find more information and a servlet sample in this article.
That's the theory. In your particular case, it's a bit tricker as you're zipping the files on the fly. You'll need to write the zip into a temporary folder of the server's local disk file system first and then stream from it and finally delete the file only when the download is successfully completed (i.e. the out.close() didn't throw IOException). You can identify the associated zip file with help of request parameter or pathinfo or maybe a key in session.
Update: as per your update: I honestly don't know and I've never experienced it, but at least I can tell that you're not the only one who suffered from this problem. At least, implementing the resume capabilities as described before may be a solution to this particular problem as Firefox would then automatically resume the download without jerking about an incomplete part.
Update 2: after having a little thought after reading your update and the browser behaviours, it look like that there's a fairly huge time gap between firing the actual request and the arrival of the response headers. I don't know the exact details how you load the files, but it look like that there is a time cost in gathering the ZIP files (maybe you're loading them from a networked filesystem or database beforehand?) and that you set/send the response headers only after you have gathered all the ZIP files. Try setting the headers and doing the output.flush() before doing the expensive task. This way the browser will get the headers as soon as possible and it will know what it may expect.
I suspect it's an artefact of using servlets - probably as a result of reassigning the thread. Certainly I have no such problem with a similar setup written in PHP (where each request is handled by a (effectively) new process.
HTH
C.