Develop Reference

Upload progress of files using HttpUrlConnection with multipart is super fast in android

I am trying to upload files using multipart with HttpUrlConnection POST method. The files are getting uploaded correctly and I am getting the response for that. But when I am tracking the progress, it just gives all the progress in 1 second even for files >100 MBs. It seems like the progress is for writing the file to buffer and not the network OutputStream. Calling flush() on the stream after writing each chunk of data doesn't help. Seems like flush just clears the stream to network and doesn't wait for the response before writing the next chunk.
Here's my code for uploading the file:
//Initialised in Constructor
boundary = twoHyphens + System.currentTimeMillis() + twoHyphens;
URL url = new URL(requestURL);
private HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
httpConn.setUseCaches(false);
httpConn.setDoOutput(true);
httpConn.setDoInput(true);
httpConn.setRequestMethod("POST");
httpConn.setRequestProperty("Connection", "Keep-Alive");
httpConn.setRequestProperty("Content-Type",
"multipart/form-data; boundary=" + boundary);
//This method is called to upload a file
public String uploadFile(String fieldName, File uploadFile) throws IOException, InterruptedException {
String fileName = uploadFile.getName();
FileInputStream fileInputStream = new FileInputStream(uploadFile);
DataOutputStream dataOutputStream = new DataOutputStream(new BufferedOutputStream(httpConn.getOutputStream()));
dataOutputStream.writeBytes(twoHyphens + boundary + lineEnd);
dataOutputStream.writeBytes("Content-Disposition: form-data; name=\"" + fieldName
+ "\"; filename=\"" + fileName + "\"" + lineEnd);
dataOutputStream.writeBytes(lineEnd);
int bytesRead, bytesAvailable, bufferSize;
byte[] buffer;
int maxBufferSize = 1 * 1024 * 1024;
//returns no. of bytes present in fileInputStream
bytesAvailable = fileInputStream.available();
bufferSize = 4096;
buffer = new byte[4096];
long size = uploadFile.length();
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
while (bytesRead > 0) {
int percentage = (int) ((bytesRead / (float) size) * 100);
dataOutputStream.write(buffer, 0, bufferSize);
dataOutputStream.flush(); //doesn't help
bytesAvailable = fileInputStream.available();
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
}//This finishes in 1 second
dataOutputStream.writeBytes(lineEnd);
dataOutputStream.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);
int status = httpConn.getResponseCode();
StringBuilder sb = new StringBuilder();
BufferedReader reader = null;
try {
if (status == HttpURLConnection.HTTP_OK) {
reader = new BufferedReader(new InputStreamReader(httpConn.getInputStream(), "UTF-8"));
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line);
}
} else {
throw new IOException("Server exception with status code: " + status);
}
} catch (Exception e) {
} finally {
if (reader != null) {
reader.close();
httpConn.disconnect();
}
}
return sb.toString();
}
Any help or explanation on this is really appreciated.

Writing to the output stream does not actually trigger the network connection. The data upload to the server is done when you call getResponseCode() , as I have detailed in my other ansswer
Only if you have some way to ask the getResponseCode() to report progress, otherwise you have no way to monitor the progress.

How do I put HttpURLConnection into BroadcastReceiver OnReceive()?

I have an HttpURLConnection uploading a file, and my server then emails it. It works great. However, I don't want it in an Activity, I want it in the SMS receiver OnRecieve(), yet I can't seem to get it to work.
Please help? When I put the code into OnReceive(), it simply fails:(
Here's my code, it's quite simple:
StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
StrictMode.setThreadPolicy(policy);
try{
int serverResponseCode = 0;
final String upLoadServerUri = "http://MY_URL_EXAMPLE/upload_file_functions.php";
String fileName = "/mnt/sdcard/MyFile.dat";
HttpURLConnection conn = null;
DataOutputStream dos = null;
String lineEnd = "\r\n";
String twoHyphens = "--";
String boundary = "*****";
int bytesRead, bytesAvailable, bufferSize;
byte[] buffer;
int maxBufferSize = 1 * 1024 * 1024;
File sourceFile = new File("/mnt/sdcard/MyFile.dat");
// open a URL connection to the Servlet
FileInputStream fileInputStream = new FileInputStream(sourceFile);
URL url = new URL(upLoadServerUri);
// Open a HTTP connection to the URL
conn = (HttpURLConnection) url.openConnection();
conn.setDoInput(true); // Allow Inputs
conn.setDoOutput(true); // Allow Outputs
conn.setUseCaches(false); // Don't use a Cached Copy
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary);
dos = new DataOutputStream(conn.getOutputStream());
dos.writeBytes(twoHyphens + boundary + lineEnd);
dos.writeBytes("Content-Disposition: form-data; name=\"file\";filename=\""+ fileName + "\"" + lineEnd);
dos.writeBytes(lineEnd);
// create a buffer of maximum size
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
buffer = new byte[bufferSize];
// read file and write it into form...
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
while (bytesRead > 0) {
dos.write(buffer, 0, bufferSize);
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
}
// send multipart form data necesssary after file data...
dos.writeBytes(lineEnd);
dos.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);
// Responses from the server (code and message)
serverResponseCode = conn.getResponseCode();
String serverResponseMessage = conn.getResponseMessage();
Log.i("uploadFile", "HTTP Response is : "
+ serverResponseMessage + ": " + serverResponseCode);
if(serverResponseCode == 200){
// it worked !
}
//close the streams //
fileInputStream.close();
dos.flush();
dos.close();
}catch (Exception e){
}

Bunch of F%$kn losers on here sometimes, I must say. I'll go ahead and answer my OWN question again:
You cannot do an httpURLrequest in BroadcastReceiver(). It must be done in a service:
public class UploadService extends IntentService
{
public UploadService()
{
super("UploadService");
}
#Override
protected void onHandleIntent(Intent intent)
{
public void postForm(String phn, String mssg)
{
try{
int serverResponseCode = 0;
final String upLoadServerUri = "http://MY_URL_EXAMPLE/upload_file_functions.php";
String fileName = "/mnt/sdcard/MyFile.dat";
HttpURLConnection conn = null;
DataOutputStream dos = null;
String lineEnd = "\r\n";
String twoHyphens = "--";
String boundary = "*****";
int bytesRead, bytesAvailable, bufferSize;
byte[] buffer;
int maxBufferSize = 1 * 1024 * 1024;
File sourceFile = new File("/mnt/sdcard/MyFile.dat");
// open a URL connection to the Servlet
FileInputStream fileInputStream = new FileInputStream(sourceFile);
URL url = new URL(upLoadServerUri);
// Open a HTTP connection to the URL
conn = (HttpURLConnection) url.openConnection();
conn.setDoInput(true); // Allow Inputs
conn.setDoOutput(true); // Allow Outputs
conn.setUseCaches(false); // Don't use a Cached Copy
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary);
dos = new DataOutputStream(conn.getOutputStream());
dos.writeBytes(twoHyphens + boundary + lineEnd);
dos.writeBytes("Content-Disposition: form-data; name=\"file\";filename=\""+ fileName + "\"" + lineEnd);
dos.writeBytes(lineEnd);
// create a buffer of maximum size
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
buffer = new byte[bufferSize];
// read file and write it into form...
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
while (bytesRead > 0) {
dos.write(buffer, 0, bufferSize);
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
}
// send multipart form data necesssary after file data...
dos.writeBytes(lineEnd);
dos.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);
// Responses from the server (code and message)
serverResponseCode = conn.getResponseCode();
String serverResponseMessage = conn.getResponseMessage();
Log.i("uploadFile", "HTTP Response is : "
+ serverResponseMessage + ": " + serverResponseCode);
if(serverResponseCode == 200){
// it worked !
}
//close the streams //
fileInputStream.close();
dos.flush();
dos.close();
}catch (Exception e){
}
}
}
Called with:
context.startService(new Intent(context, UploadService.class));

How to use properly setFixedLengthStreamingMode(int)

I'm having issues using this function. I get the length(); of the file I need to upload and i get the next error ->
Always more Bytes than expected
Exception : expected 589715 bytes but received 589840
java.io.IOException: expected 589715 bytes but received 589840
at libcore.net.http.FixedLengthOutputStream.write(FixedLengthOutputStream.java:39)
at java.io.DataOutputStream.write(DataOutputStream.java:98)
at com.androidexample.uploadtoserver.UploadToServer.uploadFile(UploadToServer.java:152)
at com.androidexample.uploadtoserver.UploadToServer$1.run(UploadToServer.java:62)
at java.lang.Thread.run(Thread.java:856)
I used this for get the size of the file
int fixedLength = (int) fileInputStream.getChannel().size();
int total = (int) sourceFile.length();
Is possible post an example of a working setFixedLengthStreamingMode(int) correctly? I can only see problems with this method
Complete code here, yes it's for tranfer a file or some files, can be little or more than 15 Mb each, and some times I get an out of memory in some old devices.
public int uploadFile(String sourceFileUri) {
String fileName = sourceFileUri;
HttpURLConnection conn = null;
DataOutputStream dos = null;
String lineEnd = "\r\n";
String twoHyphens = "--";
String boundary = "*****";
int bytesRead, bytesAvailable, bufferSize;
byte[] buffer;
int maxBufferSize = 1 * 1024 * 1024;
File sourceFile = new File(sourceFileUri);
if (!sourceFile.isFile()) {
runOnUiThread(new Runnable() {
public void run() {
/* messageText.setText("Source File not exist :"
+uploadFilePath + "" + uploadFileName);*/
}
});
return 0;
} else {
try {
FileInputStream fileInputStream = new FileInputStream(sourceFile);
int fixedLength = (int) fileInputStream.getChannel().size();
URL url = new URL(upLoadServerUri);
//int total = (int) sourceFile.length();
// Open a HTTP connection to the URL
conn = (HttpURLConnection) url.openConnection();
conn.setFixedLengthStreamingMode(fixedLength);
conn.setDoInput(true); // Allow Inputs
conn.setDoOutput(true); // Allow Outputs
conn.setUseCaches(false); // Don't use a Cached Copy
conn.setRequestMethod("POST");
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setRequestProperty("ENCTYPE", "multipart/form-data");
conn.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary);
conn.setRequestProperty("uploaded_file", fileName);
conn.setRequestProperty("mail", MAIL);
conn.setRequestProperty("OS", "1");
conn.setRequestProperty("LANG", "ES");
//conn.setChunkedStreamingMode(maxBufferSize);
dos = new DataOutputStream(conn.getOutputStream());
dos.writeBytes(twoHyphens + boundary + lineEnd);
dos.writeBytes("Content-Disposition: form-data; name=\"uploaded_file\";filename=\""
+ fileName + "\"" + lineEnd);
dos.writeBytes(lineEnd);
// create a buffer of maximum size
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
buffer = new byte[bufferSize];
// read file and write it into form...
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
while (bytesRead > 0) {
dos.write(buffer, 0, bufferSize);
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
}
// send multipart form data necesssary after file data...
dos.writeBytes(lineEnd);
dos.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);
// Responses from the server (code and message)
serverResponseCode = conn.getResponseCode();
// String serverResponseMessage = conn.getResponseMessage();
// Log.i("uploadFile", "HTTP Response is : "
// + serverResponseMessage + ": " + serverResponseCode);
if(serverResponseCode == 200) {
runOnUiThread(new Runnable() {
public void run() {
/*messageText.setText(msg);
Toast.makeText(UploadToServer.this, "File Upload Complete.",
Toast.LENGTH_SHORT).show();*/
}
});
}
//close the streams //
fileInputStream.close();
dos.flush();
dos.close();
}
}
}

You have to set it to the total number of bytes you're going to transmit. Clearly you are sending more than that. You haven't shown the relevant code so it is impossible to comment further.
Why the DataOutputStream? You don't need that just for sending a file.
EDIT: There are numerous problems with your code.
dos.writeBytes(twoHyphens + boundary + lineEnd);
dos.writeBytes("Content-Disposition: form-data; name=\"uploaded_file\";filename=\""
+ fileName + "\"" + lineEnd);
dos.writeBytes(lineEnd);
This is what is fouling up the fixed-length transfer mode. You haven't counted this stuff as part of the fixed-length.
// create a buffer of maximum size
You don't need a buffer of maximum size. A buffer of 8192 bytes is perfectly adequate.
bytesAvailable = fileInputStream.available();
InputStream.available() is specifically described in the Javadoc as follows: "It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream." Fortunately you can just delete this line.
bufferSize = Math.min(bytesAvailable, maxBufferSize);
buffer = new byte[bufferSize];
Unnecessary, see above. Just use new byte[8192].
// read file and write it into form...
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
while (bytesRead > 0) {
dos.write(buffer, 0, bufferSize);
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
}
Again this is all wrong. It's a misuse of available() for a start. The standard way to copy streams in Java is as follows:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
It works for any buffer size greater than zero; it doesn't care how long the input is; and it works correctly for the final read whatever its size.
// send multipart form data necesssary after file data...
dos.writeBytes(lineEnd);
dos.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);
Again you haven't counted this as part of the fixed-length transfer size.
But I must say you would be better off using chunked transfer mode and letting HttpURLConnection do all the heavy lifting: all you need then is the copy loop above.

EPIPE (Broken pipe) while uploading?

i have a problem in my code but i don't know where is it the E log report
04-08 05:47:46.745: E/Upload Server(20080): Starting : /storage/sdcard1/Music/Piano (my favourites)/11 Tchaikovsky - The Music Lovers.mp3
04-08 05:47:47.136: E/Upload Server(20080): Connection Error : sendto failed: EPIPE (Broken pipe)
what is (EPIPE) ? , when i attempt to upload image its upload successfully but any other file E Cat report (Broken pipe) why !
this is my uploading code
#Override
protected String doInBackground(String... urls) {
String upLoadServerUri = "http://test.com/test.php";
String fileName = this.file_path;
HttpURLConnection connection = null;
DataOutputStream outputStream = null;
String lineEnd = "\r\n";
String twoHyphens = "--";
String boundary = "*****";
int bytesRead, bytesAvailable, bufferSize;
byte[] buffer;
int maxBufferSize = 1*1024*1024;
File sourceFile = new File(fileName);
int sentBytes = 0;
long fileSize = sourceFile.length();
connection = null;
try
{
FileInputStream fileInputStream = new FileInputStream(sourceFile);
Log.e("Upload Server ", "Starting : "+ fileName );
URL url = new URL(upLoadServerUri);
connection = (HttpURLConnection) url.openConnection();
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setChunkedStreamingMode(1024);
connection.setRequestMethod("POST");
connection.setRequestProperty("Connection", "Keep-Alive");
connection.setRequestProperty("Content-Type","multipart/form-data;boundary="+boundary);
outputStream = new DataOutputStream(connection.getOutputStream() );
outputStream.writeBytes(twoHyphens + boundary + lineEnd);
outputStream.writeBytes("Content-Disposition: form-data; name=\"file[]\";filename=\""+ fileName + "\"" + lineEnd);
outputStream.writeBytes(lineEnd);
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
buffer = new byte[bufferSize];
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
while (bytesRead > 0)
{
if(isCancelled()){
break;
}
sentBytes += bytesRead;
double percentDone = (sentBytes * 1.0) / fileSize * 100;
publishProgress((int)percentDone);
outputStream.write(buffer, 0, bufferSize);
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
}
if(isCancelled()){
fileInputStream.close();
outputStream.flush();
outputStream.close();
Log.e("Upload Server ", "upload Canceled " );
return "canceled";
}
outputStream.writeBytes(lineEnd);
outputStream.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);
int serverResponseCode = connection.getResponseCode();
fileInputStream.close();
outputStream.flush();
outputStream.close();
if(serverResponseCode == 200)
{
Scanner s;
s = new Scanner(connection.getInputStream());
s.useDelimiter("\\Z");
final String response = s.next();
Log.e("Upload Server ", "Message : " + response);
return response;
}else
{
Log.e("Upload Server ", "Server Code Error : " + serverResponseCode );
return "faild";
}
} catch (final Exception e) {
Log.e("Upload Server ", "Error : " + e.getMessage() );
}
return "falid";
}
please note aim still newer in android apps :)
i googled my problem i couldn't found a solution please help !

'Broken pipe' means you have written to a connection that has already been closed by the peer.
Probably you have exceeded an upload size limit.
You should also note that your use of available() is invalid. There is a specific warning in the Javadoc about not using it the way you are using it. You don't need it anyway:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
where buffer is any reasonable size, e.g. 8192 bytes.

I had a similar problem using HttpURLConnection. Just add:
conn.setRequestProperty("connection", "close"); // disables Keep Alive
to your connection or disable it for all connections:
System.setProperty("http.keepAlive", "false");
From the API about disconnect():
Releases this connection so that its resources may be either reused or closed.
Unlike other Java implementations, this will not necessarily close socket connections that can be reused. You can disable all connection reuse by setting the http.keepAlive system property to false before issuing any HTTP requests.
So, Android will reuse the old socket connection. Disabling it with the code above you can fix it.

Progress bar resets itself?

When I upload files over 5MB, the progress bar resets itself when it reaches 4% and start from 0 every 10 sec. But, for files under 5MB, the progress bar works fine and reaches 100%
Is this because of maxBufferSize? The server max post size is 512MB and every thing else is unlimited, so the problem must be from my code but I'm not sure where.
Here is my code
#Override
protected void onProgressUpdate(Integer... progress) {
super.onProgressUpdate(progress);
//update Dialog box progress
mProgressDialog.setProgress(progress[0]);
if ( nofti_appended )
{
//update Notification progress
mBuilder.setProgress(100, progress[0], false);
}
}
protected String doInBackground(String... urls) {
String upLoadServerUri = upApi.uploadUrl;
String fileName = this.file_path;
HttpURLConnection connection = null;
DataOutputStream outputStream = null;
String lineEnd = "\r\n";
String twoHyphens = "--";
String boundary = "*****";
int bytesRead, bytesAvailable, bufferSize;
byte[] buffer;
int maxBufferSize = 1*1024*1024;
File sourceFile = new File(fileName);
int sentBytes = 0;
long fileSize = sourceFile.length();
try
{
FileInputStream fileInputStream = new FileInputStream(new File(fileName));
URL url = new URL(upLoadServerUri);
connection = (HttpURLConnection) url.openConnection();
// Allow Inputs & Outputs
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setChunkedStreamingMode(1024);
// Enable POST method
connection.setRequestMethod("POST");
connection.setRequestProperty("Connection", "Keep-Alive");
connection.setRequestProperty("Content-Type", "multipart/form-data;boundary="+boundary);
outputStream = new DataOutputStream( connection.getOutputStream() );
outputStream.writeBytes(twoHyphens + boundary + lineEnd);
outputStream.writeBytes("Content-Disposition: form-data; name=\"file[]\";filename=\""+ fileName + "\"" + lineEnd);
outputStream.writeBytes(lineEnd);
bytesAvailable = fileInputStream.available();
Log.v("Size",bytesAvailable+"");
bufferSize = Math.min(bytesAvailable, maxBufferSize);
buffer = new byte[bufferSize];
// Read file
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
while (bytesRead > 0)
{
if(isCancelled()){
break;
}
sentBytes += bytesRead;
publishProgress((int)(sentBytes * 100 / fileSize));
outputStream.write(buffer, 0, bufferSize);
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
bytesRead = fileInputStream.read(buffer, 0, bufferSize);
}
if(isCancelled()){
return "faild";
}
outputStream.writeBytes(lineEnd);
outputStream.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);
Scanner s;
s = new Scanner(connection.getInputStream());
s.useDelimiter("\\Z");
final String response = s.next();
// Responses from the server (code and message)
int serverResponseCode = connection.getResponseCode();
fileInputStream.close();
outputStream.flush();
outputStream.close();
if(serverResponseCode == 200)
{
return response;
}
} catch (MalformedURLException ex) {
} catch (final Exception e) {
}
return "faild";
}
Any ideas?

I think you actually have two problems with your progress calculation:
First, your logic is not quite right. It should be (see the next paragraph for the reason why):
(sentBytes / fileSize) * 100
The second problem, though, is that you're using int values, but trying to do a floating point calculation. (Percentage is 0.01 - 1.00, and then it's multiplied by 100 to turn it into 1% - 100%.) You need to do the calculation as floating point, and then cast the final value to integer.
Try something like
publishProgress((int)(((sentBytes * 1.0) / fileSize) * 100)));
That's getting to be a lot of () pairs, though. It would be better to declare a separate float or double variable, and use it instead:
percentDone = (sentBytes * 1.0) / fileSize * 100;
publishProgress((int)percentDone);