I am trying to upload a lot of files from a browser to a webserver that I have total control of. I've tried to use SWFUpload, but the problem is that my browser hangs when I am uploading a file. The files are supposed to be of any length (up to 2GB due to other restrictions, of course).
While the upload is in progress the browser hangs and the GUI is not updated. When the file is finally uploaded the GUI responds again. In firefox the entire browser stalls and in Chrome just the tab.
I thought it might have something to do with flash trying to inject the entire file in to the memory first, but that does not seem to be the case (although different test cases yielded different results, but I'm not sure that it's because of that reason).
An applet is also an option, although flash is preferred. The reason I don't want to use a form is because you can select multiple in flash (also possible with Java, I assume) which is a nice feature that I'm going for.
Any ideas? Thought?
(math|tob)ias
If you're using linux, this is a known problem. For Windows users this doesn't happen. Otherwise, you'll have to stick to ajax (the best way is with the jQuery forms plugin).
http://www.plupload.com/, this one looks interesting, but I haven't looked into it. You may want to give it a try.
Try an AJAX one:
http://sourceforge.net/projects/uber-uploader/
Lighter on the resources and more platform-independent.
client-side I like this jquery plugin:
http://valums.com/ajax-upload/
if you need examples of server-side handling:
http://php.net/manual/en/features.file-upload.php
I've use this for my projects... http://jupload.sourceforge.net/ I found I had better results though when I used it in FTP mode. I had issues with http uploads. It does support multiple files which is nice. You can upload the files into an "incoming directory" and then process them later when the transfer is finished.
Check SWFUpload swfupload.org
and there is a descussion about this point:
http://swfupload.org/forum/generaldiscussion/173
Related
I am currently in the process of developing an application that will request some information from Websites. What I'm looking to do is parse the HTML files through a connection online. I was just wondering, by parsing the Website will it put any strain on the server, will it have to download any excess information or will it simply connect to the site as I would do through my browser and then scan the source?
If this is putting extra strain on the Website then I'm going to have to make a special request to some of the companies I'm scanning. However if not then I have the permission to do this.
I hope this made some sort of sense.
Kind regards,
Jamie.
No extra strain on other people servers. The server will get your simple HTML GET request, it won't even be aware that you're then parsing the page/html.
Have you checked this: JSoup?
Consider doing the parsing and the crawling/scraping in separate steps. If you do that, you can probably use an existing open-source crawler such as crawler4j that already has support for politeness delays, robots.txt, etc. If you just blindly go grabbing content from somebody's site with a bot, the odds are good that you're going to get banned (or worse, if the admin is feeling particularly vindictive or creative that day).
Depends on the website. If you do this to Google then most likely you will be on a hold for a day. If you parse Wikipedia, (which I have done myself) it won't be a problem because its already a huge, huge website.
If you want to do it the right way, first respect robots.txt, then try to scatter your requests. Also try to do it when the traffic is low. Like around midnight and not at 8AM or 6PM when people get to computers.
Besides Hank Gay's recommendation, I can only suggest that you can also re-use some open-source HTML parser, such as Jsoup, for parsing/processing the downloaded HTML files.
You could use htmlunit. It gives you virtual gui less browser.
Your Java program hitting other people's server to download the content of a URL won't put any more strain on the server than a web browser doing so-- essentially they're precisely the same operation. In fact, you probably put less strain on them, because your program probably won't be bothered about downloading images, scripts etc that a web browser would.
BUT:
if you start bombarding a server of a company with moderate resources with downloads or start exhibiting obvious "robot" patterns (e.g. downloading precisely every second), they'll probably block you; so put some sensible constraints on what you do (e.g. every consecutive download to the same server happens at random intervals of between 10 and 20 seconds);
when you make your request, you probably want to set the "referer" request header either to mimic an actual browser, or to be open about what it is (invent a name for your "robot", create a page explaining what it does and include a URL to that page in the referer header)-- many server owners will let through legitimate, well-behaved robots, but block "suspicious" ones where it's not clear what they're doing;
on a similar note, if you're doing things "legally", don't fetch pages that the site's "robot.txt" files prohibits you from fetching.
Of course, within some bounds of "non-malicious activity", in general it's perfectly legal for you to make whatever request you want whenever you want to whatever server. But equally, that server has a right to serve or deny you that page. So to prevent yourself from being blocked, one way or another, you need to either get approval from the server owners, or "keep a low profile" in your requests.
Is there any methods for downloading multiple files at the same time? That is, after select multiple files, click the download button, and choose the destination directory, the files selected will be downloaded at the same time.
i googled it and not find any solutions, can anybody know? im using struts1 and spring2.
This is not part of the normal file download model supported by web browsers. You might be able to do this with some client-side Javascript. However, I suspect it is not possible, since Javascript driven upload of multiple files would probably entail breaking Javascript's security sandbox.
Don't think so. You can download a zip though.
I've been tasked with implementing large (2gb+) file uploads via a web browser. After evaluating various technologies, java applets seem to be the way forward (only one's which provide proper access to the local disk). I was wondering if anyone can recommend a 3rd party file upload app we can use as a base? requirements are
Decent UI, ideally we want something similar to facebooks photo uploader
Can handle large (2gb+) files
Resumable uploads
We beed the source to extend it to our needs (dont mind paying extra)
You're probably looking for JUpload.
Update: not sure if it has as nice of UI as you're hoping, but unless you want to build a custom solution like I have it's your best option.
Just a tip, maybe it is obvious, i don't know :P
It is nice to send the big file in chunks like 2mb, and on the server side you just append the bytes to the target file. The server knows what bytes it needs, and if a upload is aborted and continued later, the server can just send a message about from what bytes to start uploading the file again. Then we get resumability (is it a word? :P) and safety of large HTTP-uploads (since, in fact, we are sending many smuller uploads, and each upload is checked to be of the correct size on the server).
We wrote an implementation like this once with a Java-applet as the client and PHP on the server, I'll see if I can dig it out as a reference for you :p
Not really a solution : from experience you may bump into the following issues:
problems when uploading over HTTPs
problems uploading through proxies
Just wanted to make you aware of these two cases, for you to test when evaluating a solution.
Hope, you will get solutions for your prob over here.. http://jupload.sourceforge.net/
I'm working on a web application. There is one place where the user can upload files with the HTTP protocol. There is a choice between the classic HTML file upload control and a Java applet to upload the files.
The classic HTML file upload isn't great because you can only select one file at a time, and it's quite hard to get any progress indication during the actual upload (I finally got it using a timer refreshing a progress indicator with data fetched from the server via an AJAX call). The advantage: it's always working.
With the Java applet I can do more things: select multiple files at once (even a folder), compress the files, get a real progress bar, drag'n'drop files on the applet, etc...
BUT there are a few drawbacks:
it's a nightmare to get it to work properly on Mac Safari and Mac Firefox (Thanks Liveconnect)
the UI isn't exactly the native UI and some people notice that
the applet isn't as responsive as it should (could be my fault, but everything looks ok to me)
there are bugs in the Java UrlConnection class with HTTPS, so I use the Apache common HTTP client to do the actual HTTP upload. It's quite big a package and slows down the download of the .jar file
the Apache common HTTP client has sometimes trouble going through proxies
the Java runtime is quite big
I've been maintaining this Java applet for a while but now I'm fed up with all the drawbacks, and considering writing/buying a completely new component to upload theses files.
Question
If you had the following requirements:
upload multiple files easily from a browser, through HTTP or HTTPS
compress the files to reduce the upload time
upload should work on any platform, with native UI
must be able to upload huge files, up to 2gb at least
you have carte blanche on the technology
What technology/compontent would you use?
Edit :
Drag'n'Drop of files on the component would be a great plus.
It looks like there are a lot of issues related to bugs with the Flash Player (swfupload known issues). Proper Mac support and upload through proxies with authentication are options I can not do without. This would probably rule out all Flash-based options :-( .
I rule out all HTML/Javascript-only options because you can't select more than one file at a time with the classic HTML control. It's a pain to click n-times the "browse" button when you want to select multiple files in a folder.
I implemented something very recently in Silverlight.
Basically uses HttpWebRequest to send a chunk of data to a GenericHandler.
On the first post, 4KB of data is sent. On the 2nd chunk, I send another 4K chunk.
When the 2nd chunk is received, I calculate the round trip it took between first and 2nd chunk and so now
the 3rd chunk when sent will know to increase speed.
Using this method I can upload files of ANY size and I can resume.
Each post I send along this info:
[PARAMETERS]
[FILEDATA]
Here, parameters contain the following:
[Chunk #]
[Filename]
[Session ID]
After each chunk is received, I send a response back to my Silverlight saying how fast it took so that it can now send a larger
chunk.
Hard to put my explaination without code but that's basically how I did it.
At some point I will put together a quick writeup on how I did this.
I've never used it with files of 2GB in size, but the YUI File Uploader worked pretty well on a previous project. You may also be interested in this jQuery Plugin.
That said, I still think the Java Applet is the way to go. I think you'll end up with less portability and UI issues than you expect and Drag/Drop works great. For the record, Box.net uses a Java Applet for their multi-file quick uploads.
OK this is my take on this
I did some testing with swfupload, and I have my previous experience with Java, and my conclusion is that whatever technology is used there is no perfect solution to do uploads on the browser : you'll always end up with bugs when uploading huge files, going through proxies, with ssl, etc...
BUT :
a flash uploader (a la swfupload) is really lightweight, doesn't need authorization from the user and has a native interface which is REALLY cool, me thinks
a java uploader needs authorization but you can do whatever you want with the files selected by the user (aka compression if needed), and drag and drop works well. Be prepared for some epic bugs debuggin' though.
I didn't get a change to play with Silverlight as long as I'd like maybe that's the real answer, though the technology is still quite young so ... I'll edit this post if I get a chance to fiddle a bit with Silverlight
Thanks for all the answers !!
There are a number of free flash components that exist with nice multiple file upload capability. They make use of ActionScripts FileReference class with a PHP (or whatever) receiver on the server side. Some have recently broken with the launch of FP10 but I know for certain that swfupload will work :)
Hope this helps!
What about these two
Jupload
http://jupload.sourceforge.net/
and
jumploader
http://jumploader.com/
Both are java applets but they are also both really easy to use and implement.
what about google gears?
There are HTTP/HTTPS upload controls that allow multi-file upload. Here is one from Telerik, which I have found to be solid and reliable. The latest version looks to have most if not all of your feature requirements.
You can upload multiple files with HTTP forms as well, as Dave already pointed out, but if you're set on using something beyond what HTTP and Javascript offers I would heavily consider Flash. There are even some pre-existing solutions for it such as MultiPowUpload and it offers many of the features you're looking for. It's also easier to obtain progress information using a Flash client than with AJAX calls from Javascript since you have a little more flexibility.
You may check the Apache Commons FileUpload package. It allows you to upload multiple files, monitor the progress of the upload, and more. You can find more information here:
http://commons.apache.org/fileupload/
http://commons.apache.org/fileupload/using.html
Good luck
I need assistance finding a delivery method that best fulfills the following requirements:
We wish to deliver a single file to my clients.
Clients should be able to launch this file from the operating system shell - much like running an '.exe' on Windows.
After being launched, the program/script should be able to display a window with HTML content. this may be done using a stand alone program, a runtime or by running within a browser.
We need the ability to embed a resource within the delivered file, such as an mp3 file, which i can later extract programmatically.
Optimally, the solution should run on Windows, Mac and Linux machines. Less than perfect cross-platform interoperability is acceptable, but we want as broad a penetration as possible.
Clients should not need to pre-install anything (unless it is done transparently), pre-configure anything, or approve any thing for this to happen.
For example:
We could use a regular executable file, written in C++ to do this, but it will not be cross-platform.
We could use a sliverlight XAP file, an adobe Flex file or a Java JAR, but internet explorer warns users when local content is launched. In addition these approaches mean that we have less than perfect penetration, even though it is acceptable in these cases.
We could use a python (or equivalent) script, but the installed-base (penetration) of the python interpreter is not good enough.
Using a standard HTML is not enough because of the difficulty of embedding resources in it. Embedding Silverlight XAML or uuencoded content in HTML causes IE to display a warning.
Using something along the lines of a jpeg as a delivery method is not rich enough since we need to display HTML.
..but internet explorer warns users when local content is launched..
I don't get it, what's the problem with IE saying "Hey this app is trying to run your files!"
I don't mean you don't have a good reason for this, it is just, I don't get it.
IE will only warn the user if the app has not been downloaded and try to access local resources, for instance if running from an applet or a JNLP like this one:(click on the first orange button you see )
But if the users download the jar and run it from the computer ( double click on it ) the app is local and can run without problems.
The jar file is a zip file after all, so you can attach your mp3 file with it. Double click is supported in the desired platform, and the HTML content could be either a local file ( un-packed along with the mp3 file ) or an internet web page.
Java is preinstalled on those OS already.
"internet explorer warns users when local content is launched"
There's a reason for this. How can they distinguish your excellent, well-behaved, polite application from a virus?
Since the line between your app and a virus is very, very blurry, go with any of Silverlight XAP file, an adobe Flex file or a Java JAR.
The IE business is a good thing, not a bad thing.
You could try using the 'Jetty' application server.
This supposes there is a working java environment on the target machine.
Jetty is java servlet container but it is possible to configure
everything (web server, html templates, applications, etc.) in a
single executable jar, which launches the web server and opens a default page.
Exactly how the jar file is launched will vary from platform to platform
but otherwise the user interface will be identicle, and, as its a java application
you can do pretty much anything one it has started.
Sounds like MIME HTML does exactly what you want - unfortunately, it is not supported by many browsers other than IE.
I'd investigate Adobe AIR. It can display both HTML and Flex content in a desktop application without using a web browser. However this will require installation of the AIR runtime, also I'm not sure if the Linux version is out of the beta stage.
http://www.adobe.com/products/air/
You can also use a binary for each platform.
As per your description the app is very simple, and porting from one platf to another sounds like just matter of re-compile and offer binary based on the dist.
Is this an option?