I'm working on a web application. There is one place where the user can upload files with the HTTP protocol. There is a choice between the classic HTML file upload control and a Java applet to upload the files.
The classic HTML file upload isn't great because you can only select one file at a time, and it's quite hard to get any progress indication during the actual upload (I finally got it using a timer refreshing a progress indicator with data fetched from the server via an AJAX call). The advantage: it's always working.
With the Java applet I can do more things: select multiple files at once (even a folder), compress the files, get a real progress bar, drag'n'drop files on the applet, etc...
BUT there are a few drawbacks:
it's a nightmare to get it to work properly on Mac Safari and Mac Firefox (Thanks Liveconnect)
the UI isn't exactly the native UI and some people notice that
the applet isn't as responsive as it should (could be my fault, but everything looks ok to me)
there are bugs in the Java UrlConnection class with HTTPS, so I use the Apache common HTTP client to do the actual HTTP upload. It's quite big a package and slows down the download of the .jar file
the Apache common HTTP client has sometimes trouble going through proxies
the Java runtime is quite big
I've been maintaining this Java applet for a while but now I'm fed up with all the drawbacks, and considering writing/buying a completely new component to upload theses files.
Question
If you had the following requirements:
upload multiple files easily from a browser, through HTTP or HTTPS
compress the files to reduce the upload time
upload should work on any platform, with native UI
must be able to upload huge files, up to 2gb at least
you have carte blanche on the technology
What technology/compontent would you use?
Edit :
Drag'n'Drop of files on the component would be a great plus.
It looks like there are a lot of issues related to bugs with the Flash Player (swfupload known issues). Proper Mac support and upload through proxies with authentication are options I can not do without. This would probably rule out all Flash-based options :-( .
I rule out all HTML/Javascript-only options because you can't select more than one file at a time with the classic HTML control. It's a pain to click n-times the "browse" button when you want to select multiple files in a folder.
I implemented something very recently in Silverlight.
Basically uses HttpWebRequest to send a chunk of data to a GenericHandler.
On the first post, 4KB of data is sent. On the 2nd chunk, I send another 4K chunk.
When the 2nd chunk is received, I calculate the round trip it took between first and 2nd chunk and so now
the 3rd chunk when sent will know to increase speed.
Using this method I can upload files of ANY size and I can resume.
Each post I send along this info:
[PARAMETERS]
[FILEDATA]
Here, parameters contain the following:
[Chunk #]
[Filename]
[Session ID]
After each chunk is received, I send a response back to my Silverlight saying how fast it took so that it can now send a larger
chunk.
Hard to put my explaination without code but that's basically how I did it.
At some point I will put together a quick writeup on how I did this.
I've never used it with files of 2GB in size, but the YUI File Uploader worked pretty well on a previous project. You may also be interested in this jQuery Plugin.
That said, I still think the Java Applet is the way to go. I think you'll end up with less portability and UI issues than you expect and Drag/Drop works great. For the record, Box.net uses a Java Applet for their multi-file quick uploads.
OK this is my take on this
I did some testing with swfupload, and I have my previous experience with Java, and my conclusion is that whatever technology is used there is no perfect solution to do uploads on the browser : you'll always end up with bugs when uploading huge files, going through proxies, with ssl, etc...
BUT :
a flash uploader (a la swfupload) is really lightweight, doesn't need authorization from the user and has a native interface which is REALLY cool, me thinks
a java uploader needs authorization but you can do whatever you want with the files selected by the user (aka compression if needed), and drag and drop works well. Be prepared for some epic bugs debuggin' though.
I didn't get a change to play with Silverlight as long as I'd like maybe that's the real answer, though the technology is still quite young so ... I'll edit this post if I get a chance to fiddle a bit with Silverlight
Thanks for all the answers !!
There are a number of free flash components that exist with nice multiple file upload capability. They make use of ActionScripts FileReference class with a PHP (or whatever) receiver on the server side. Some have recently broken with the launch of FP10 but I know for certain that swfupload will work :)
Hope this helps!
What about these two
Jupload
http://jupload.sourceforge.net/
and
jumploader
http://jumploader.com/
Both are java applets but they are also both really easy to use and implement.
what about google gears?
There are HTTP/HTTPS upload controls that allow multi-file upload. Here is one from Telerik, which I have found to be solid and reliable. The latest version looks to have most if not all of your feature requirements.
You can upload multiple files with HTTP forms as well, as Dave already pointed out, but if you're set on using something beyond what HTTP and Javascript offers I would heavily consider Flash. There are even some pre-existing solutions for it such as MultiPowUpload and it offers many of the features you're looking for. It's also easier to obtain progress information using a Flash client than with AJAX calls from Javascript since you have a little more flexibility.
You may check the Apache Commons FileUpload package. It allows you to upload multiple files, monitor the progress of the upload, and more. You can find more information here:
http://commons.apache.org/fileupload/
http://commons.apache.org/fileupload/using.html
Good luck
Related
I need to allow for csv-file downloads on my page and I was going to try ngCsv (from Angular) but for browser support this seems fairly limited. I've seen quite a few examples of this being done with vanilla Javascript. And after a discussion with a colleague of "backend vs. frontend" I'm feeling more and more unsure of what to do.
Are there any true optimization/efficiency reasons why I should avoid doing this on the client side (assuming the files are no more than 100MB each download)?
Are there any true optimization/efficiency reasons why I should avoid
doing this on the client side (assuming the files are no more than
100MB each download)?
If the data on the .csv would be the same for each user, and only updated every now and then, I would suggest you have your server create / update a static .csv. It wouldn't be resource-intensive, and you wouldn't have to worry about browser compatibility / user resources.
If, however, the data you need to create a .csv for is different on a per-user basis, then you should consider creating the file client-side. If you can help it, you don't want your server having to dynamically generate 100MB .csv files each time a user clicks the link.
You could write a script that only generates the .csv client-side if the browser is not mobile and there is web-worker support. If either of those conditions are not met, you could fall back to having your server do it.
Ultimately, your answer is going to really depend on the requirements / context of this project. Try to cache the results where possible, and use common sense. Good luck :)
I am currently in the process of developing an application that will request some information from Websites. What I'm looking to do is parse the HTML files through a connection online. I was just wondering, by parsing the Website will it put any strain on the server, will it have to download any excess information or will it simply connect to the site as I would do through my browser and then scan the source?
If this is putting extra strain on the Website then I'm going to have to make a special request to some of the companies I'm scanning. However if not then I have the permission to do this.
I hope this made some sort of sense.
Kind regards,
Jamie.
No extra strain on other people servers. The server will get your simple HTML GET request, it won't even be aware that you're then parsing the page/html.
Have you checked this: JSoup?
Consider doing the parsing and the crawling/scraping in separate steps. If you do that, you can probably use an existing open-source crawler such as crawler4j that already has support for politeness delays, robots.txt, etc. If you just blindly go grabbing content from somebody's site with a bot, the odds are good that you're going to get banned (or worse, if the admin is feeling particularly vindictive or creative that day).
Depends on the website. If you do this to Google then most likely you will be on a hold for a day. If you parse Wikipedia, (which I have done myself) it won't be a problem because its already a huge, huge website.
If you want to do it the right way, first respect robots.txt, then try to scatter your requests. Also try to do it when the traffic is low. Like around midnight and not at 8AM or 6PM when people get to computers.
Besides Hank Gay's recommendation, I can only suggest that you can also re-use some open-source HTML parser, such as Jsoup, for parsing/processing the downloaded HTML files.
You could use htmlunit. It gives you virtual gui less browser.
Your Java program hitting other people's server to download the content of a URL won't put any more strain on the server than a web browser doing so-- essentially they're precisely the same operation. In fact, you probably put less strain on them, because your program probably won't be bothered about downloading images, scripts etc that a web browser would.
BUT:
if you start bombarding a server of a company with moderate resources with downloads or start exhibiting obvious "robot" patterns (e.g. downloading precisely every second), they'll probably block you; so put some sensible constraints on what you do (e.g. every consecutive download to the same server happens at random intervals of between 10 and 20 seconds);
when you make your request, you probably want to set the "referer" request header either to mimic an actual browser, or to be open about what it is (invent a name for your "robot", create a page explaining what it does and include a URL to that page in the referer header)-- many server owners will let through legitimate, well-behaved robots, but block "suspicious" ones where it's not clear what they're doing;
on a similar note, if you're doing things "legally", don't fetch pages that the site's "robot.txt" files prohibits you from fetching.
Of course, within some bounds of "non-malicious activity", in general it's perfectly legal for you to make whatever request you want whenever you want to whatever server. But equally, that server has a right to serve or deny you that page. So to prevent yourself from being blocked, one way or another, you need to either get approval from the server owners, or "keep a low profile" in your requests.
I am trying to upload a lot of files from a browser to a webserver that I have total control of. I've tried to use SWFUpload, but the problem is that my browser hangs when I am uploading a file. The files are supposed to be of any length (up to 2GB due to other restrictions, of course).
While the upload is in progress the browser hangs and the GUI is not updated. When the file is finally uploaded the GUI responds again. In firefox the entire browser stalls and in Chrome just the tab.
I thought it might have something to do with flash trying to inject the entire file in to the memory first, but that does not seem to be the case (although different test cases yielded different results, but I'm not sure that it's because of that reason).
An applet is also an option, although flash is preferred. The reason I don't want to use a form is because you can select multiple in flash (also possible with Java, I assume) which is a nice feature that I'm going for.
Any ideas? Thought?
(math|tob)ias
If you're using linux, this is a known problem. For Windows users this doesn't happen. Otherwise, you'll have to stick to ajax (the best way is with the jQuery forms plugin).
http://www.plupload.com/, this one looks interesting, but I haven't looked into it. You may want to give it a try.
Try an AJAX one:
http://sourceforge.net/projects/uber-uploader/
Lighter on the resources and more platform-independent.
client-side I like this jquery plugin:
http://valums.com/ajax-upload/
if you need examples of server-side handling:
http://php.net/manual/en/features.file-upload.php
I've use this for my projects... http://jupload.sourceforge.net/ I found I had better results though when I used it in FTP mode. I had issues with http uploads. It does support multiple files which is nice. You can upload the files into an "incoming directory" and then process them later when the transfer is finished.
Check SWFUpload swfupload.org
and there is a descussion about this point:
http://swfupload.org/forum/generaldiscussion/173
Looking for a descent looking Java File Upload Applet (or even Flash) that fits the following requirements:
Free/cheap (yes looking to use in a commercial web application)
Preferably allow multiple file selection
Ability to create/edit folders on destination server? (within a secure root folder)
Ability to overwrite files in the destination directory?
IE + Firefox support required... Safari, Chrome, etc. a bonus
Ability to "hook" into the flow to apply additional logic
Notes:
In particular I'm looking for the ability to upload to a directory on another server (e.g. not where my web server or app server is running)
The destination server has a shared drive where all users of the app have permission to store files.
By "decent looking" I'd really, really like to avoid a UI that is "skinned" like this:
(source: swtguibuilder.com)
Before I'm shut down for "not programming related" I would need to be able to hook into such an applet to apply some additional permission logic etc. I don't want to re-invent the wheel if someone else has made a component that already does what I need.
This is relatively easy with Java. Look into Apache Commons Fileupload for the server-side code. For the front-end, there are many examples of flash-based file uploads.
The simple answer to your question used to be Flash. Since the release of Flash Player 10 it isn't anymore. Adobe has added a restriction on File upload which only allows a file to be uploaded in the scope of UIA - User Initiated Action. This means that a file upload must be started with a mouse click.
There are some of free file uploaders built in flash available. The only problem is that the queuing is broken by FP10. Adding the files to the pending uploads list is no problem. Starting the upload is no problem either. But when one file upload is done and the next should begin, Flash Player will not allow it if you don't click again which is pretty much useless.
There at least two requirements:
Applet file uploader These are available, e.g.: http://sourceforge.net/projects/jupload/
Widget to configure destination This is unusual in my experience, but I wrote one once for a Swing app which would be similar for an applet. It was necessary to provide the user a view of the remote file system, enable manipulation of the view, and then upload the changes. Note that there are transactions issues here, so it's not trivial.
I've been tasked with implementing large (2gb+) file uploads via a web browser. After evaluating various technologies, java applets seem to be the way forward (only one's which provide proper access to the local disk). I was wondering if anyone can recommend a 3rd party file upload app we can use as a base? requirements are
Decent UI, ideally we want something similar to facebooks photo uploader
Can handle large (2gb+) files
Resumable uploads
We beed the source to extend it to our needs (dont mind paying extra)
You're probably looking for JUpload.
Update: not sure if it has as nice of UI as you're hoping, but unless you want to build a custom solution like I have it's your best option.
Just a tip, maybe it is obvious, i don't know :P
It is nice to send the big file in chunks like 2mb, and on the server side you just append the bytes to the target file. The server knows what bytes it needs, and if a upload is aborted and continued later, the server can just send a message about from what bytes to start uploading the file again. Then we get resumability (is it a word? :P) and safety of large HTTP-uploads (since, in fact, we are sending many smuller uploads, and each upload is checked to be of the correct size on the server).
We wrote an implementation like this once with a Java-applet as the client and PHP on the server, I'll see if I can dig it out as a reference for you :p
Not really a solution : from experience you may bump into the following issues:
problems when uploading over HTTPs
problems uploading through proxies
Just wanted to make you aware of these two cases, for you to test when evaluating a solution.
Hope, you will get solutions for your prob over here.. http://jupload.sourceforge.net/