Develop Reference

HttpUrlConnection Expect-100 Broken Pipe

I use a Expect 100-continue in the http header like you see in the code below. The HttpUrlConnection when i get the outputstream dont wait the answer of the 100 Expectation or any other 400 error. (In the RFC they tell it's not a problem: https://www.rfc-editor.org/rfc/rfc7231#section-5.1.1).
I've looked at this post: How to wait for Expect 100-continue response in Java using HttpURLConnection in case they have more info on the problem but i have tested the setChuncked method just to see if it's make a difference but no.
The problem is when the server answer with an error and close the connection without let me finish the upload i have a Broken Pipe Exception. It's right i write bytes on a close connection but i can't handle that correctly. I can't read the server answer. I can't know that the connection is close before the write. If i try to get the ErrorStream i get a null reference same for the InputStream, ResponseCode and other methods to know what happen.
Any idea how can make it right or what i miss to make it work?
javax.net.ssl.SSLException: Write error: ssl=0x73de8c02c0: I/O error during system call, Broken pipe
at com.android.org.conscrypt.NativeCrypto.SSL_write(Native Method)
at com.android.org.conscrypt.OpenSSLSocketImpl$SSLOutputStream.write(OpenSSLSocketImpl.java:824)
at com.android.okhttp.okio.Okio$1.write(Okio.java:76)
at com.android.okhttp.okio.AsyncTimeout$1.write(AsyncTimeout.java:155)
at com.android.okhttp.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:176)
at com.android.okhttp.okio.RealBufferedSink.write(RealBufferedSink.java:46)
at com.android.okhttp.internal.http.HttpConnection$ChunkedSink.write(HttpConnection.java:339)
at com.android.okhttp.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:176)
at com.android.okhttp.okio.RealBufferedSink$1.write(RealBufferedSink.java:198)
at out.write(buffer, 0, bytes_read);
In the code below, i've remove some not useful part like a global try on this, or compute for the fileSize.
huc = (HttpURLConnection) new URL(upload_url).openConnection();
huc.setRequestMethod("POST");
huc.setRequestProperty("Content-Length", "" + FILE_SZ);
huc.setRequestProperty("User-Agent", "UA");
huc.setRequestProperty("Expect","100-continue");
huc.setFixedLengthStreamingMode(FILE_SZ);
huc.setUseCaches(false);
huc.setDoInput(true);
huc.setDoOutput(true);
huc.setInstanceFollowRedirects(true);
try {
// read input file chunks + publish progress
OutputStream out = new OutputStream(huc.getOutputStream());
buffer_size = 4 * 1024;
buffer = new byte[buffer_size];
total_bytes = 0;
while ((bytes_read = fis.read(buffer)) != -1) {
out.write(buffer, 0, bytes_read);
total_bytes += bytes_read;
}
out.flush();
out.close();
} catch (IOException e) {
e.printStackTrace();
}

Download file in java when Content-Length is incorrectly set?

I have an android app that downloads and uses a file at runtime. The file is valid as I can download it via the browser and open it up, etc. However my app kept reporting that the file is corrupted.
After investigation I discovered the server (which I have no control over) is returning an incorrect "Content-Length:" (~180 vs ~120000). The header is the culprit as I confirmed the issue by downloading the file with curl - which also resulted in a truncated file.
After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length. To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Anybody know of a way to download a file if the Content-Length is incorrectly set? A browser can.
Here's my latest attempt:
public static void downloadFileFromRemoteUrl(String urlString, String destination){
try {
URL url = new URL(urlString);
File file = new File(destination);
URLConnection urlConnection = url.openConnection();
InputStream inputStream = urlConnection.getInputStream();
byte[] buffer = new byte[1024];
int curLength = 0;
int newLength = 0;
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
byteArrayOutputStream.write(buffer, 0, newLength);
}
FileOutputStream fos = new FileOutputStream(file);
fos.write(byteArrayOutputStream.toByteArray());
fos.close();
android.util.Log.d("DB UPDATE", "Done downloading database. Size: " + byteArrayOutputStream.toByteArray().length);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}

After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length.
Nonsense. You are crediting those classes with paranormal powers. How could an output stream possibly become aware of the Content-length header? The URLConnection's input stream is being terminated at the content-length. Correctly.
To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Of course not.
Anybody know of a way to download a file if the Content-Length is incorrectly set?
You could use a Socket and engage in HTTP yourself, which is less trivial than it sounds. But the problem is at the server and that's where it should be fixed. Complain. Or else #Zong Yu is correct and the page is HTML containing JavaScript, say.
NB You don't need to read the entire file into memory:
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
fos.write(buffer, 0, newLength);
}

My final "solution" was to realize I was dealing with a 301 redirect response and not the actual resource! I updated the section that handles my url, checking for a 301 and if exists, update the url. The new url contained the Content-Length that corresponded with the file I was downloading.
// start by creating an http url connection object
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
// determine if this is a redirect
boolean redirect = false;
int status = httpURLConnection.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
// if it is, we need a new url
if (redirect) {
String newUrl = httpURLConnection.getHeaderField("Location");
httpURLConnection = (HttpURLConnection) new URL(newUrl).openConnection();
}

Try Fetch. Fetch is an in app download manager for Android. It's very easy to use. Find the GitHub page here. The project comes with several demos that you can try out. Disclaimer: I'm the creator of Fetch, and it is open source.

How to download a part of a file from URL in android?

I am trying to download a part of file given the download URL using setRequestProperty("Range","bytes=" + startbytes + "-" + endbytes); The following code snippet shows what I am trying to do.
protected String doInBackground(String... aurl) {
int count;
Log.d(TAG,"Entered");
try {
URL url = new URL(aurl[0]);
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
int lengthOfFile = connection.getContentLength();
Log.d(TAG,"Length of file: "+ lengthOfFile);
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
The problem is that, an exception is being raised, which says "Cannot set request property after connection is made". Please help me resolve this issue.

Option 1
If you do not need to know the content length:
[Beware, do not call the connection.getContentLength(). If you call that, you will get the exception. If you need to call it, then check the second option]
URL url = new URL(aurl[0]);
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
//Note that, response code will be 206 (Partial Content) instead of usual 200 (OK)
if(connection.getResponseCode() == HttpURLConnection.HTTP_PARTIAL){
//Your code here to read response data
}
Option 2
If you need to know the content length:
URL url = new URL(aurl[0]);
//First make a HEAD call to get the content length
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
connection.setRequestMethod("HEAD");
if(connection.getResponseCode() == HttpURLConnection.HTTP_OK){
int lengthOfFile = connection.getContentLength();
Log.d("ERF","Length of file: "+ lengthOfFile);
connection.disconnect();
//Now that we know the content lenght, make the GET call
connection =(HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
//Note that, response code will be 206 (Partial Content) instead of usual 200 (OK)
if(connection.getResponseCode() == HttpURLConnection.HTTP_PARTIAL){
//Your code here to read response data
}
}

Assuming you're using HTTP for the download, you'll want to use the HEAD http verb and RANGE http header.
HEAD will give you the filesize (if available), and then RANGE lets you download a byte range.
Once you have the filesize, divide it into roughly equal sized chunks and spawn download thread for each chunk. Once all are done, write the file chunks in the correct order.
If you don't know how to use the RANGE header, here's another SO answer that explains how: https://stackoverflow.com/a/6323043/1355166
[EDIT]
To make file into chunks use this, and start the downloading process,
private void getBytesFromFile(File file) throws IOException {
FileInputStream is = new FileInputStream(file); //videorecorder stores video to file
java.nio.channels.FileChannel fc = is.getChannel();
java.nio.ByteBuffer bb = java.nio.ByteBuffer.allocate(10000);
int chunkCount = 0;
byte[] bytes;
while(fc.read(bb) >= 0){
bb.flip();
//save the part of the file into a chunk
bytes = bb.array();
storeByteArrayToFile(bytes, mRecordingFile + "." + chunkCount);//mRecordingFile is the (String)path to file
chunkCount++;
bb.clear();
}
}
private void storeByteArrayToFile(byte[] bytesToSave, String path) throws IOException {
FileOutputStream fOut = new FileOutputStream(path);
try {
fOut.write(bytesToSave);
}
catch (Exception ex) {
Log.e("ERROR", ex.getMessage());
}
finally {
fOut.close();
}
}

Send a file with a PUT request using a HttpURLConnection

I'm trying to do something relatively simple. I need to make a simple PUT request with a file in the body in order to upload a file to a server not in my control. Here's the code I have so far:
connection = ((HttpURLConnection)new URL(ticket.getEndpoint()).openConnection());
connection.setRequestMethod("PUT");
connection.setRequestProperty("Content-Type", "video/mp4");
connection.setRequestProperty("Content-Length", String.valueOf(getStreamFile().length()));
connection.setUseCaches(false);
connection.setDoOutput(true);
connection.connect();
outputStream = connection.getOutputStream();
streamFileInputStream = new FileInputStream(getStreamFile());
streamFileBufferedInputStream = new BufferedInputStream(streamFileInputStream);
byte[] streamFileBytes = new byte[getBufferLength()];
int bytesRead = 0;
int totalBytesRead = 0;
while ((bytesRead = streamFileBufferedInputStream.read(streamFileBytes)) > 0) {
outputStream.write(streamFileBytes, 0, bytesRead);
outputStream.flush();
totalBytesRead += bytesRead;
notifyListenersOnProgress((double)totalBytesRead / (double)getStreamFile().length());
}
outputStream.close();
logger.debug("Wrote {} bytes of {}, ratio: {}",
new Object[]{totalBytesRead, getStreamFile().length(),
(double)totalBytesRead / (double)getStreamFile().length()});
I'm watching my network manager and nothing near the size of my file gets sent. In fact, I don't know if anything is being sent at all, but I don't see any errors thrown.
I need to be able to send this request and also measure the status of the upload synchronously, so as to be able to inform my listeners of the upload progress. How can I modify my existing example to just work™?

Try setting the content-type param to multipart/form-data. W3C forms.

How to manage loop of request-response dialog with server

I'm writing a simple client-server system and the question is: how to structure my client code in order to get POST request-response working in a loop?
At the moment it looks something like this (and it's is NOT a loop right now):
open HttpURLConnection
set properties
setDoOutput(true)
writing to output stream
closing output stream
new DataInputStream
reading response
exiting method
I'm not sure which objects do I have to save for the next iterations and which ones I should close.

you need to save the connection object and you should make use of setDoInput(true) for reading data but if you just want to read responseCode and responseMessage you dont need InputStream. check the code below.
HttpURLConnection connection =(HttpURLConnection)new URL("url").openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-type", "text/xml"); // depend on you
connection.setRequestProperty("Accept", "text/xml, application/xml"); // depend on you
connection.setRequestMethod("POST");
OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());
writer.write(yaml);
writer.close();
int statusCode = connection.getResponseCode();
String message = connection.getResponseMessage();
for InputStreamReader
connection.setDoInput(true);
InputStreamReader reader = new InputStreamReader(connection.getInputStream());
char[] cbuf = new char[100];
reader.read(cbuf);
// there are 3 read method you can choose as per your convenience
//and put a check for end of line in while loop for reading whole content.
reader.close();

After managing my own 'research' on this subject (thanks to Google and Nokia Developer forums) I've come to the final view of my code. It's a file upload loop:
path = Paths.get(requestString);
in = Files.newInputStream(path);
int i = 0;
while ((bytesRead = in.read(buf)) != -1) {
URL u = new URL(defaultURL);
huc =
(HttpURLConnection) u.openConnection();
huc.setRequestMethod("POST");
huc.setDoOutput(true);
huc.setDoInput(true);
os = huc.getOutputStream();
os.write(buf, 0, bytesRead);
os.flush();
os = null;
// thanks to dku.rajkumar for the following block of code !
InputStreamReader reader =
new InputStreamReader(huc.getInputStream());
char[] cbuf = new char[400];
reader.read(cbuf);
reader.close();
String s = new String(cbuf);
messagebuffer.append(s + "\n\n");
huc.disconnect();
Thread.sleep(16);
}