I am trying to upload some bytes to the server for 15 seconds.I have written the following code to write the bytes to output stream :
long uploadedBytes=0;
ByteArrayInputStream byteArrayInputStream=null;
OutputStream outputStream=null;
try {
byte[] randomData=generateBinData(5*1024);
byte[] bytes = new byte[(int) 1024 * 5];
URL url = new URL(urls[0]);
HttpURLConnection connection =
(HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setRequestMethod("POST");
connection.setRequestProperty("Connection", "Keep-Alive");
outputStream = connection.getOutputStream();
byteArrayInputStream = new ByteArrayInputStream(randomData);
long startTime=System.currentTimeMillis();
while(byteArrayInputStream.read(bytes) > 0
&& timeDiff < 15000) {
outputStream.write(bytes, 0, bytes.length);
uploadedBytes += bytes.length;
byteArrayInputStream = new ByteArrayInputStream(randomData);
timeDiff = System.currentTimeMillis() - startTime;
int progress=(int)(timeDiff *100 / 15000);
publishProgress(progress);
}
But the progress for the above upload is running very fast and it shows large amount of bytes uploaded within seconds.Which is not according to my 2g mobile network connection.
For example it shows :
uploadedBytes =9850880 and with time difference(timeDiff) = 3 sec.
if i run the same code for 15 seconds it terminates the whole application.
Please help me to find where i am goind wrong.
thanks ...waiting for reply
Unless you set chunked or streaming transfer mode, HttpURLConnection buffers all the output before sending any of it, so it can get a Content-Length. So what you're seeing is the progress of the buffering, not of the transfer. Set chunked transfer mode and you will see a difference.
Your copy loop is wrong. It should be like this:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Your code will probably work in this specific case but that's not a reason not to get it right for all cases.
check your random byte length. i think the generateBinData() method is not generating 5Kb of data.
sure the uploadedBytes is huge. say, if a write to outputstream takes 10 milisec to write 5Kb(5*1024) of data,in 3 second you should able to write only 153600 bytes.
Reason for app termination - check if any read operation throws exception.
Related
I use a Expect 100-continue in the http header like you see in the code below. The HttpUrlConnection when i get the outputstream dont wait the answer of the 100 Expectation or any other 400 error. (In the RFC they tell it's not a problem: https://www.rfc-editor.org/rfc/rfc7231#section-5.1.1).
I've looked at this post: How to wait for Expect 100-continue response in Java using HttpURLConnection in case they have more info on the problem but i have tested the setChuncked method just to see if it's make a difference but no.
The problem is when the server answer with an error and close the connection without let me finish the upload i have a Broken Pipe Exception. It's right i write bytes on a close connection but i can't handle that correctly. I can't read the server answer. I can't know that the connection is close before the write. If i try to get the ErrorStream i get a null reference same for the InputStream, ResponseCode and other methods to know what happen.
Any idea how can make it right or what i miss to make it work?
javax.net.ssl.SSLException: Write error: ssl=0x73de8c02c0: I/O error during system call, Broken pipe
at com.android.org.conscrypt.NativeCrypto.SSL_write(Native Method)
at com.android.org.conscrypt.OpenSSLSocketImpl$SSLOutputStream.write(OpenSSLSocketImpl.java:824)
at com.android.okhttp.okio.Okio$1.write(Okio.java:76)
at com.android.okhttp.okio.AsyncTimeout$1.write(AsyncTimeout.java:155)
at com.android.okhttp.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:176)
at com.android.okhttp.okio.RealBufferedSink.write(RealBufferedSink.java:46)
at com.android.okhttp.internal.http.HttpConnection$ChunkedSink.write(HttpConnection.java:339)
at com.android.okhttp.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:176)
at com.android.okhttp.okio.RealBufferedSink$1.write(RealBufferedSink.java:198)
at out.write(buffer, 0, bytes_read);
In the code below, i've remove some not useful part like a global try on this, or compute for the fileSize.
huc = (HttpURLConnection) new URL(upload_url).openConnection();
huc.setRequestMethod("POST");
huc.setRequestProperty("Content-Length", "" + FILE_SZ);
huc.setRequestProperty("User-Agent", "UA");
huc.setRequestProperty("Expect","100-continue");
huc.setFixedLengthStreamingMode(FILE_SZ);
huc.setUseCaches(false);
huc.setDoInput(true);
huc.setDoOutput(true);
huc.setInstanceFollowRedirects(true);
try {
// read input file chunks + publish progress
OutputStream out = new OutputStream(huc.getOutputStream());
buffer_size = 4 * 1024;
buffer = new byte[buffer_size];
total_bytes = 0;
while ((bytes_read = fis.read(buffer)) != -1) {
out.write(buffer, 0, bytes_read);
total_bytes += bytes_read;
}
out.flush();
out.close();
} catch (IOException e) {
e.printStackTrace();
}
This question already has answers here:
Determine the size of an InputStream
(13 answers)
Closed 3 years ago.
I'm having an InputStream from a ProcessBuilder that acutally reads the stdout stream.
Question: how can I know the size of that inmemory InputStream, so I can write it to a HttpResponse http header?
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
OutputStream out = response.getOutputStream();
int bytes;
while ((bytes = br.read()) != -1) {
out.write(bytes);
}
//how can I know the size of the inmemory stream/file written?
//response.setContentLength((int) pdfFile.length());
There is no such thing as the size of an input stream. Consider a program which never exits, or a socket peer which never stops sending. And you don't need to know to write it to an HttpResponse header. The Content-length is managed automatically for you.
Try this
InputStream is = process.getInputStream();
ByteArrayOutputStream os = new ByteArrayOutputStream();
int b;
while ((b = is.read()) != -1)
os.write(b);
response.setContentLength(os.size());
response.getOutputStream().write(os.toByteArray());
If you really want to set the content length header, you'll need to read the entire stream before writing to the response OutputStream
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] bytes = new byte[1024];
int count;
while ((count = in.read(bytes)) > 0) {
out.write(bytes, 0, count);
}
response.setContentLength(out.size();
out.writeTo(response.getOutputStream());
Note: With this approach you've now read the entire stream into memory, this will have an impact on available memory and likely won't scale well.
import org.apache.commons.io.IOUtils;
byte[] bytes = IOUtils.toByteArray(inputStream);
log.message("bytes .lenght "+bytes.length);
if (bytes.length > 400000)
//some byte range limit`enter code can add any byte range
{
throw new Exception("File Size is larger than 40 MB ..");
}
An InputStream inherently doesn't have a size. It could conceivably keep delivering bytes forever. Or the producing end could end the stream without warning.
If you must find out the length, then you have to read to the end, counting the bytes, and report the length when you finish.
You're worrying about HTTP's Content-length header, and you've got a point. The fact is that the original version of HTTP was not designed for large, dynamically generated content. The protocol inherently expects you to know the size of the content before you start writing it - yet how is that possible if it's (for example) an ongoing chat, or the output of a video camera?
The solution is HTTP's chunked transfer encoding. Here you don't set a Content-Length header. You set Transfer-Encoding: chunked, then write the content as chunks, each of which has a size header.
The HTTP RFC has details one this, or https://en.wikipedia.org/wiki/Chunked_transfer_encoding is slightly more friendly.
However most HTTP APIs hide this detail from you. Unless you are developing a web library from scratch (perhaps for academic reasons), you shouldn't have to think about Content-Length or Transfer-Encoding.
I'm trying to implement pause/resume in my download manager, I search the web and read several articles and change my code according them but resume seems not working correctly, Any ideas?
if (!downloadPath.exists())
downloadPath.mkdirs();
if (outputFileCache.exists())
{
downloadedSize = outputFileCache.length();
connection.setAllowUserInteraction(true);
connection.setRequestProperty("Range", "bytes=" + downloadedSize + "-");
connection.setConnectTimeout(14000);
connection.connect();
input = new BufferedInputStream(connection.getInputStream());
output = new FileOutputStream(outputFileCache, true);
input.skip(downloadedSize); //Skip downloaded size
}
else
{
connection.setConnectTimeout(14000);
connection.connect();
input = new BufferedInputStream(url.openStream());
output = new FileOutputStream(outputFileCache);
}
fileLength = connection.getContentLength();
byte data[] = new byte[1024];
int count = 0;
int __progress = 0;
long total = downloadedSize;
while ((count = input.read(data)) != -1 && !this.isInterrupted())
{
total += count;
output.write(data, 0, count);
__progress = (int) (total * 100 / fileLength);
}
output.flush();
output.close();
input.close();
Okay problem fixed, here is my code for other users who wants to implement pause/resume:
if (outputFileCache.exists())
{
connection.setAllowUserInteraction(true);
connection.setRequestProperty("Range", "bytes=" + outputFileCache.length() + "-");
}
connection.setConnectTimeout(14000);
connection.setReadTimeout(20000);
connection.connect();
if (connection.getResponseCode() / 100 != 2)
throw new Exception("Invalid response code!");
else
{
String connectionField = connection.getHeaderField("content-range");
if (connectionField != null)
{
String[] connectionRanges = connectionField.substring("bytes=".length()).split("-");
downloadedSize = Long.valueOf(connectionRanges[0]);
}
if (connectionField == null && outputFileCache.exists())
outputFileCache.delete();
fileLength = connection.getContentLength() + downloadedSize;
input = new BufferedInputStream(connection.getInputStream());
output = new RandomAccessFile(outputFileCache, "rw");
output.seek(downloadedSize);
byte data[] = new byte[1024];
int count = 0;
int __progress = 0;
while ((count = input.read(data, 0, 1024)) != -1
&& __progress != 100)
{
downloadedSize += count;
output.write(data, 0, count);
__progress = (int) ((downloadedSize * 100) / fileLength);
}
output.close();
input.close();
}
It is impossible to tell what is wrong without some more information, however things to note:
You must make a HTTP/1.1 request (it's hard to tell from your sample code)
The server must support HTTP/1.1
The server will tell you what it supports with an Accept-Ranges header in the response
If-Range should be the etag the server gave you for the resource, not the last modified time
You should check your range request with something simple to test the origin actually supports the Range request first (like curl or wget )
It nay be that your server is taking to long to respond (more then the timeout limit) or this is also a fact that not all servers support pause - resume.
It is also a point to ponder that weather the file is downloaded through Http, https, ftp or udp.
Pausing" could just mean reading some of the stream and writing it to disk. When resuming you would have to use the headers to specify what is left to download.
you may try something like :
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
if(ISSUE_DOWNLOAD_STATUS.intValue()==ECMConstant.ECM_DOWNLOADING){
File file=new File(DESTINATION_PATH);
if(file.exists()){
downloaded = (int) file.length();
connection.setRequestProperty("Range", "bytes="+(file.length())+"-");
}
}else{
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
}
connection.setDoInput(true);
connection.setDoOutput(true);
progressBar.setMax(connection.getContentLength());
in = new BufferedInputStream(connection.getInputStream());
fos=(downloaded==0)? new FileOutputStream(DESTINATION_PATH): new FileOutputStream(DESTINATION_PATH,true);
bout = new BufferedOutputStream(fos, 1024);
byte[] data = new byte[1024];
int x = 0;
while ((x = in.read(data, 0, 1024)) >= 0) {
bout.write(data, 0, x);
downloaded += x;
progressBar.setProgress(downloaded);
}
and please try to sync things.
I would start debugging from this line:
connection.setRequestProperty("Range", "bytes=" + downloadedSize + "-");
As from the source code it is not possible to determine what downloadedSize is, it's hard to elaborate further, but the format should be bytes=from-to.
Anyway, I would suggest you to use Apache HttpClient to avoid common pitfalls. Here is a question from someone who uses Apache HttpClient on a similar topic and some sample code is provided.
I think you just need to delete the input.skip(downloadedSize) line. Setting the HTTP header for byte range means the server will skip sending those bytes.
Say you have a file that's 20 bytes long consisting of "aaaaabbbbbcccccddddd", and suppose the transfer is paused after downloading 5 bytes. Then the Range header will cause the server to send "bbbbbcccccddddd", you should read all of this content and append it to the file -- no skip(). But the skip() call in your code will skip "bbbbb" leaving "cccccddddd" to be downloaded. If you've already downloaded at least 50% of the file, then skip() will exhaust all of the input and nothing will happen.
Also, all of the things in stringy05's post apply. You should make sure the server supports HTTP/1.1, make sure the Range header is supported for the resource (dynamically generated content may not support it), and make sure the resource isn't modified using etag and modification date.
I'm trying to do something relatively simple. I need to make a simple PUT request with a file in the body in order to upload a file to a server not in my control. Here's the code I have so far:
connection = ((HttpURLConnection)new URL(ticket.getEndpoint()).openConnection());
connection.setRequestMethod("PUT");
connection.setRequestProperty("Content-Type", "video/mp4");
connection.setRequestProperty("Content-Length", String.valueOf(getStreamFile().length()));
connection.setUseCaches(false);
connection.setDoOutput(true);
connection.connect();
outputStream = connection.getOutputStream();
streamFileInputStream = new FileInputStream(getStreamFile());
streamFileBufferedInputStream = new BufferedInputStream(streamFileInputStream);
byte[] streamFileBytes = new byte[getBufferLength()];
int bytesRead = 0;
int totalBytesRead = 0;
while ((bytesRead = streamFileBufferedInputStream.read(streamFileBytes)) > 0) {
outputStream.write(streamFileBytes, 0, bytesRead);
outputStream.flush();
totalBytesRead += bytesRead;
notifyListenersOnProgress((double)totalBytesRead / (double)getStreamFile().length());
}
outputStream.close();
logger.debug("Wrote {} bytes of {}, ratio: {}",
new Object[]{totalBytesRead, getStreamFile().length(),
(double)totalBytesRead / (double)getStreamFile().length()});
I'm watching my network manager and nothing near the size of my file gets sent. In fact, I don't know if anything is being sent at all, but I don't see any errors thrown.
I need to be able to send this request and also measure the status of the upload synchronously, so as to be able to inform my listeners of the upload progress. How can I modify my existing example to just work�
Try setting the content-type param to multipart/form-data. W3C forms.
My app needs to download large files. After some time I get
java.net.SocketException: Connection timed out
I believe it's because the device is going to sleep or wifi.
So how i should handle this ? I want that user could download a large file no matter how much time it will take.
File downloading is done using:
HttpURLConnection con = (HttpURLConnection) new URL(uriToFile).openConnection();
con.connect();
FileOutputStream fileOutput = new FileOutputStream(file);
InputStream inputStream = con.getInputStream();
byte[] buffer = new byte[1024];
int bufferLength = 0;
while ((bufferLength = inputStream.read(buffer)) > 0) {
fileOutput.write(buffer, 0, bufferLength);
}
fileOutput.close();
Thanks.
You can't handle it, other than by retrying the connection. You can lower the default connection timeout of about 75 seconds, but you can't raise it.