Develop Reference

How do I download a file that is embedded in a web site?

At this moment I can only download files that has this type of format:
https://jdbc.postgresql.org/download/postgresql-8.1-415.jdbc2.jar
But how do I download files that aren't visible in the url file?
For e.g Skype's url path:
http://www.skype.com/sv/download-skype/skype-for-mac/downloading/
As you guys can see, there is no way I can download the file using
filePath.subString(filePath.lastIndexOf("/") + 1);
So are there other ways to do this? I did find the file embedded in the page using FireBug which is
http://www.skype.com/go/getskype-macosx.dmg
My question is, can I programmatically go through the page and get access to this file?
Here is the code which works fine for downloading
public static void fileDownload(String urlFile) throws IOException {
URL url = new URL(urlFile);
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
int responseCode = httpURLConnection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
String fileName = "";
String disposition = httpURLConnection.getHeaderField("Content-Disposition");
String contentType = httpURLConnection.getContentType();
int contentLength = httpURLConnection.getContentLength();
if (disposition != null) {
int index = disposition.indexOf("filename=");
if (index > 0) {
fileName = disposition.substring(index + 10, disposition.length() - 1);
}
} else {
fileName = urlFile.substring(urlFile.lastIndexOf("/") + 1, urlFile.length());
}
System.out.println("Content-type= " + contentType);
System.out.println("Disposition= " + disposition);
System.out.println("Content-length= " + contentLength);
System.out.println("File name= " + fileName);
InputStream inputStream = httpURLConnection.getInputStream();
String saveFilePath = getDesiredPath() + File.separator + fileName;
FileOutputStream fileOutputStream = new FileOutputStream(saveFilePath);
int byteRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((byteRead = inputStream.read(buffer)) != -1) {
fileOutputStream.write(buffer, 0, byteRead);
}
fileOutputStream.close();
inputStream.close();
System.out.println("File downloaded");
} else {
System.out.println("No file to download. Server replied httpCode=" + responseCode);
}
httpURLConnection.disconnect();
}
It's my first time working with file management and this code is actually taken from here.

You can download the file if the file download link is embedded in the page.
Something like this in the web page html:
. . .
Download Skype
. . .
For downloading the page and scanning it for links you may use JSoup
Code may look something like this:
Document doc = Jsoup.connect("http://example.com/").get();
Elements anchors = doc.select("a");
// Untested code
for (var anchor of anchors) // ECMA 6 (i think)
{
if (anchor.href.endsWith(".exe")
{
// if href is not full url i.e. not starting with http://
var downloadLink = url + anchor.href;
// Download the file with the about url
}
}

How to download a part of a file from URL in android?

I am trying to download a part of file given the download URL using setRequestProperty("Range","bytes=" + startbytes + "-" + endbytes); The following code snippet shows what I am trying to do.
protected String doInBackground(String... aurl) {
int count;
Log.d(TAG,"Entered");
try {
URL url = new URL(aurl[0]);
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
int lengthOfFile = connection.getContentLength();
Log.d(TAG,"Length of file: "+ lengthOfFile);
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
The problem is that, an exception is being raised, which says "Cannot set request property after connection is made". Please help me resolve this issue.

Option 1
If you do not need to know the content length:
[Beware, do not call the connection.getContentLength(). If you call that, you will get the exception. If you need to call it, then check the second option]
URL url = new URL(aurl[0]);
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
//Note that, response code will be 206 (Partial Content) instead of usual 200 (OK)
if(connection.getResponseCode() == HttpURLConnection.HTTP_PARTIAL){
//Your code here to read response data
}
Option 2
If you need to know the content length:
URL url = new URL(aurl[0]);
//First make a HEAD call to get the content length
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
connection.setRequestMethod("HEAD");
if(connection.getResponseCode() == HttpURLConnection.HTTP_OK){
int lengthOfFile = connection.getContentLength();
Log.d("ERF","Length of file: "+ lengthOfFile);
connection.disconnect();
//Now that we know the content lenght, make the GET call
connection =(HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
//Note that, response code will be 206 (Partial Content) instead of usual 200 (OK)
if(connection.getResponseCode() == HttpURLConnection.HTTP_PARTIAL){
//Your code here to read response data
}
}

Assuming you're using HTTP for the download, you'll want to use the HEAD http verb and RANGE http header.
HEAD will give you the filesize (if available), and then RANGE lets you download a byte range.
Once you have the filesize, divide it into roughly equal sized chunks and spawn download thread for each chunk. Once all are done, write the file chunks in the correct order.
If you don't know how to use the RANGE header, here's another SO answer that explains how: https://stackoverflow.com/a/6323043/1355166
[EDIT]
To make file into chunks use this, and start the downloading process,
private void getBytesFromFile(File file) throws IOException {
FileInputStream is = new FileInputStream(file); //videorecorder stores video to file
java.nio.channels.FileChannel fc = is.getChannel();
java.nio.ByteBuffer bb = java.nio.ByteBuffer.allocate(10000);
int chunkCount = 0;
byte[] bytes;
while(fc.read(bb) >= 0){
bb.flip();
//save the part of the file into a chunk
bytes = bb.array();
storeByteArrayToFile(bytes, mRecordingFile + "." + chunkCount);//mRecordingFile is the (String)path to file
chunkCount++;
bb.clear();
}
}
private void storeByteArrayToFile(byte[] bytesToSave, String path) throws IOException {
FileOutputStream fOut = new FileOutputStream(path);
try {
fOut.write(bytesToSave);
}
catch (Exception ex) {
Log.e("ERROR", ex.getMessage());
}
finally {
fOut.close();
}
}

Java ProgressBar with HTTP Upload don't update

I want my jProgressBar to update its value during HTTP File Upload.
I'm new to Java and I'm not sure I'm doing the right things, here's my code:
private static final String Boundary = "--7d021a37605f0";
public void upload(URL url, File f) throws Exception
{
HttpURLConnection theUrlConnection = (HttpURLConnection) url.openConnection();
theUrlConnection.setDoOutput(true);
theUrlConnection.setDoInput(true);
theUrlConnection.setUseCaches(false);
theUrlConnection.setChunkedStreamingMode(1024);
theUrlConnection.setRequestProperty("Content-Type", "multipart/form-data; boundary="
+ Boundary);
DataOutputStream httpOut = new DataOutputStream(theUrlConnection.getOutputStream());
String str = "--" + Boundary + "\r\n"
+ "Content-Disposition: form-data;name=\"file1\"; filename=\"" + f.getName() + "\"\r\n"
+ "Content-Type: image/png\r\n"
+ "\r\n";
httpOut.write(str.getBytes());
FileInputStream uploadFileReader = new FileInputStream(f);
int numBytesToRead = 1024;
int availableBytesToRead;
jProgressBar1.setMaximum(uploadFileReader.available());
while ((availableBytesToRead = uploadFileReader.available()) > 0)
{
jProgressBar1.setValue(jProgressBar1.getMaximum() - availableBytesToRead);
byte[] bufferBytesRead;
bufferBytesRead = availableBytesToRead >= numBytesToRead ? new byte[numBytesToRead]
: new byte[availableBytesToRead];
uploadFileReader.read(bufferBytesRead);
httpOut.write(bufferBytesRead);
httpOut.flush();
}
httpOut.write(("--" + Boundary + "--\r\n").getBytes());
httpOut.flush();
httpOut.close();
// read & parse the response
InputStream is = theUrlConnection.getInputStream();
StringBuilder response = new StringBuilder();
byte[] respBuffer = new byte[4096];
while (is.read(respBuffer) >= 0)
{
response.append(new String(respBuffer).trim());
}
is.close();
System.out.println(response.toString());
}
Is this line jProgressBar1.setValue(jProgressBar1.getMaximum() - availableBytesToRead); correct ?

Something like one in every 30 questions tagged java here has the same solution as yours. You are doing all your work inside an event handler, which means that it's happening on the Event Dispatch Thread -- and blocks all further GUI updates until it's over. You must use a SwingWorker and delegate your work to it.

I second #Marko Topolnic's suggestion of using a SwingWorker , Have a look at these useful links to get you further on Howto ,
How to Use Progress Bars
Concurrency in Swing
Worker Threads and SwingWorker
and an example by #trashgod.

How to disable buffering in Java HttpURLConnection for multi-part form post?

(This is for a signed applet and I have decided against HTTPClient to keep my jar very small)
I am using HttpURLConnection to successfully upload a file from the user to a server using multi-part form post.
The problem is that HttpURLConnection is caching the data -- before sending it. So when I am reading from the file and writing to Outputstream, it is merely buffering the data -- and therefore my progress bar, that shows the upload status , is completely wrong. Howevere please note that the form post code works and the file does get uploaded correctly with return code of 200.
So how do I ensure that HttpURLConnection does not cache the data that I am sending to the server ?
Here is my source:
public UploadResponse send(String formPostUrlStr,String fileFieldName,File targetFile, Map<String, String> valuesMap, UploadStatusListener uploadStatusListener) throws Exception{
String sendStr=getBoundaryMessage(Boundary, valuesMap, fileFieldName, targetFile.getName(), valuesMap.get("content-type") );//"image/png") ;
System.out.println(" multi-part start \n "+ sendStr+ " multi-part end \n");
String lenstr=Long.toString((long)(sendStr.length()*2)+ targetFile.length());
System.out.println("Content-Length"+ lenstr);
//Content-Length
URL url= new URL(formPostUrlStr);
long startTime= System.currentTimeMillis();
HttpURLConnection s3Connection = (HttpURLConnection) url.openConnection();
System.out.println("opened url to "+ formPostUrlStr +", connection ok ..");
s3Connection.setRequestProperty("Content-Type", "multipart/form-data; boundary="
+ Boundary);
s3Connection.setRequestProperty("content-length", lenstr);
s3Connection.setDoOutput(true);
s3Connection.setDoInput(true);
s3Connection.setUseCaches(false);
s3Connection.setInstanceFollowRedirects(true);
s3Connection.setAllowUserInteraction(true);
s3Connection.setRequestProperty("User-Agent", "Mozilla/4.5");
if (uploadStatusListener != null) {
uploadStatusListener.statusUpdate(targetFile.length(), 0);
}
String debugStr= s3Connection.toString();
System.out.println("conmnection "+ debugStr);
DataOutputStream httpOut = new DataOutputStream(s3Connection.getOutputStream());
System.out.println("opened DataOutputStream ok ..");
httpOut.write(sendStr.getBytes());
//httpOut.flush();
System.out.println("httpOut.flush 1 ok ..");
FileInputStream uploadFileReader = new FileInputStream(targetFile);
long totalBytes = uploadFileReader.available();
if (uploadStatusListener != null) {
uploadStatusListener.statusUpdate(totalBytes, 0);
}
System.out.println(" uploading file with size "+ uploadFileReader.available());
int bufSize = 102400;
long availableBytesToRead;
long totalSent = 0;
while ((availableBytesToRead = uploadFileReader.available()) > 0) {
byte[] bufferBytesRead;
bufferBytesRead = availableBytesToRead >= bufSize ? new byte[bufSize]
: new byte[(int)availableBytesToRead];
int count = uploadFileReader.read(bufferBytesRead);
try{
httpOut.write(bufferBytesRead);
totalSent += ((long) count);
System.out.println(" wrote bytes = "+count+ ", total sent = "+ totalSent +", pendingSize"+ (availableBytesToRead-count) );
}
catch(IOException ioe){
System.out.println(" io exceotion e"+ ioe.getMessage());
throw ioe;
}
//httpOut.flush();
if (uploadStatusListener != null) {
uploadStatusListener.statusUpdate(totalBytes, totalSent);
}
}
// FILE DATA END
httpOut.write(("--" + Boundary + "--\r\n").getBytes());
// form end
httpOut.write(("--" + Boundary + "--\r\n").getBytes());
httpOut.flush();
httpOut.close();
long endTime= System.currentTimeMillis();
System.out.println("Completed Writing Data to S3 Connection in "+ (endTime-startTime)+"ms.,now waiting for rsponse code ");
int code=s3Connection.getResponseCode();
long endTime2= System.currentTimeMillis();
System.out.println("Completed Sendind Data to S3 in "+ (endTime2-startTime)+ "ms., rsponse code time "+ (endTime2-endTime)+"ms. ");
UploadResponse uploadResponse = new UploadResponse();
uploadResponse.setCode(code);
System.out.println(" response code : " + code);
StringBuilder response = new StringBuilder();
byte[] respBuffer = new byte[4096];
if (code > 300) {
if (code == 404) {
throw new Exception("Error 404");
}
BufferedReader err = new BufferedReader(
new InputStreamReader(s3Connection.getErrorStream()));
String ret;
StringBuffer buff = new StringBuffer();
while ((ret = err.readLine()) != null) {
buff.append(ret);
}
uploadResponse.setMessage(buff.toString());
System.out.println(" error :"+ buff.toString());
err.close();
} else {
BufferedReader inp = new BufferedReader(
new InputStreamReader(s3Connection.getInputStream()));
StringBuffer buff = new StringBuffer();
String ret;
while ((ret = inp.readLine()) != null) {
buff.append(ret);
}
inp.close();
uploadResponse.setMessage(buff.toString());
if(buff.toString().contains("fail"))
throw new Exception("Upload failed");
}
System.out.println(response.toString());
return uploadResponse;
}
}

I have the same problem.
I didn't find any other solution than writing my HTTP request on a raw Socket.
Did you find a better workaround ?
EDIT : I just did : we just have to use obj.setFixedLengthStreamingMode(12345) on the HttpURLConnection object obtained from url.openConnection(), where 12345 is the length of POST request body.

As a complementation for the answer that #Antares gave, there is another method setChunkedStreamingMode that is used when you don't know the content size in advance. So when you do a POST request, call that method on the connection:
HttpURLConnection connection = ...
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setChunkedStreamingMode(0);
connection.connect();
... connection.getOutputStream();
This will avoid the OutputStream to buffer the entire content before start to send.

file upload using java servlet as a service without a web browser

I am very new to java and servlet programming.
I am not sure whether it is possible to write a servlet which when passed a URL from the local client machine, uploads the file to the server.
basically on the client machine we have a C# program and on the server side we have Apache-tomcat installed. I need to upload file(s) to the server using C# program on client machine.
Should I provide any more information (?)
Thanks in Advance

Note this code illustrates the general idea and not guaranteed to work without modification.
The C# file upload part
// this code shows you how the browsers wrap the file upload request, you still can fine a way simpler code to do the same thing.
public void PostMultipleFiles(string url, string[] files)
{
string boundary = "----------------------------" + DateTime.Now.Ticks.ToString("x");
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.ContentType = "multipart/form-data; boundary=" + boundary;
httpWebRequest.Method = "POST";
httpWebRequest.KeepAlive = true;
httpWebRequest.Credentials = System.Net.CredentialCache.DefaultCredentials;
Stream memStream = new System.IO.MemoryStream();
byte[] boundarybytes =System.Text.Encoding.ASCII.GetBytes("\r\n--" + boundary +"\r\n");
string formdataTemplate = "\r\n--" + boundary + "\r\nContent-Disposition: form-data; name=\"{0}\";\r\n\r\n{1}";
string headerTemplate = "Content-Disposition: form-data; name=\"{0}\"; filename=\"{1}\"\r\n Content-Type: application/octet-stream\r\n\r\n";
memStream.Write(boundarybytes, 0, boundarybytes.Length);
for (int i = 0; i < files.Length; i++)
{
string header = string.Format(headerTemplate, "file" + i, files[i]);
//string header = string.Format(headerTemplate, "uplTheFile", files[i]);
byte[] headerbytes = System.Text.Encoding.UTF8.GetBytes(header);
memStream.Write(headerbytes, 0, headerbytes.Length);
FileStream fileStream = new FileStream(files[i], FileMode.Open,
FileAccess.Read);
byte[] buffer = new byte[1024];
int bytesRead = 0;
while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
{
memStream.Write(buffer, 0, bytesRead);
}
memStream.Write(boundarybytes, 0, boundarybytes.Length);
fileStream.Close();
}
httpWebRequest.ContentLength = memStream.Length;
Stream requestStream = httpWebRequest.GetRequestStream();
memStream.Position = 0;
byte[] tempBuffer = new byte[memStream.Length];
memStream.Read(tempBuffer, 0, tempBuffer.Length);
memStream.Close();
requestStream.Write(tempBuffer, 0, tempBuffer.Length);
requestStream.Close();
try
{
WebResponse webResponse = httpWebRequest.GetResponse();
Stream stream = webResponse.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string var = reader.ReadToEnd();
}
catch (Exception ex)
{
response.InnerHtml = ex.Message;
}
httpWebRequest = null;
}
and to understand how the above code was written you might wanna take a look at How does HTTP file upload work?
POST /upload?upload_progress_id=12344 HTTP/1.1
Host: localhost:3000
Content-Length: 1325
Origin: http://localhost:3000
... other headers ...
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryePkpFF7tjBAqx29L
------WebKitFormBoundaryePkpFF7tjBAqx29L
Content-Disposition: form-data; name="MAX_FILE_SIZE"
100000
------WebKitFormBoundaryePkpFF7tjBAqx29L
Content-Disposition: form-data; name="uploadedfile"; filename="hello.o"
Content-Type: application/x-object
... contents of file goes here ...
------WebKitFormBoundaryePkpFF7tjBAqx29L--
and finally all you have to do is to implement a servlet that can handle the file upload request, then you do whatever that you want to do with the file, take a look at this file upload tutorial
protected void processRequest(HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
// Create path components to save the file
final String path = request.getParameter("destination");
final Part filePart = request.getPart("file");
final String fileName = getFileName(filePart);
OutputStream out = null;
InputStream filecontent = null;
final PrintWriter writer = response.getWriter();
try {
out = new FileOutputStream(new File(path + File.separator
+ fileName));
filecontent = filePart.getInputStream();
int read = 0;
final byte[] bytes = new byte[1024];
while ((read = filecontent.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
writer.println("New file " + fileName + " created at " + path);
LOGGER.log(Level.INFO, "File{0}being uploaded to {1}",
new Object[]{fileName, path});
} catch (FileNotFoundException fne) {
writer.println("You either did not specify a file to upload or are "
+ "trying to upload a file to a protected or nonexistent "
+ "location.");
writer.println("<br/> ERROR: " + fne.getMessage());
LOGGER.log(Level.SEVERE, "Problems during file upload. Error: {0}",
new Object[]{fne.getMessage()});
} finally {
if (out != null) {
out.close();
}
if (filecontent != null) {
filecontent.close();
}
if (writer != null) {
writer.close();
}
}
}
private String getFileName(final Part part) {
final String partHeader = part.getHeader("content-disposition");
LOGGER.log(Level.INFO, "Part Header = {0}", partHeader);
for (String content : part.getHeader("content-disposition").split(";")) {
if (content.trim().startsWith("filename")) {
return content.substring(
content.indexOf('=') + 1).trim().replace("\"", "");
}
}
return null;
}