I found a strange hack-y way to fix my code and I was wondering if anyone could explain why it works. I am writing code that communicates with a REST API to upload a video file split into multiple HTTP requests.
I was having a problem with one of my video part requests connecting, but never responding. The program uploads the video in five parts, but it would always hang on the third part of the five parts. I decided to add a request hard timeout to force the program to skip that hanging part. Well, magically after adding that timer, there is no more hangup!
Any ideas why this is the case? The request doesn't actually timeout, yet the addition of this code keeps my program chugging.
private void uploadParts(String assetId) throws IOException {
//set up post request
HttpClient client = HttpClientBuilder.create().build();
String url = "";
//prepare video
File video = new File("files/video.mp4");
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(video));
int partMaxSize = 1024 * 1024 * 5;
byte[] buffer = new byte[partMaxSize];
double fileSize = video.length();
System.out.println(fileSize);
System.out.println(fileSize / partMaxSize);
int parts = (int) Math.ceil(fileSize / partMaxSize);
System.out.println(parts);
for(int i = 1; i < parts+1; i++) {
String partNumber = i + "";
System.out.println("part: " + partNumber);
int partSize = (int) (i < parts ? partMaxSize : fileSize);
fileSize -= partSize;
int tmp = 0;
tmp = bis.read(buffer);
url = String.format("https://www.site.com/upload/multipart/%s/%s", assetId, partNumber);
final HttpPut request = new HttpPut(url);
request.addHeader("Authorization", "Bearer " + accessToken);
request.addHeader("Content-Type", "application/octet-stream");
request.setEntity(new ByteArrayEntity(buffer));
//Magical code start
int hardTimeout = 5; // seconds
TimerTask task = new TimerTask() {
#Override
public void run() {
if (request != null) {
request.abort();
}
}
};
new Timer(true).schedule(task, hardTimeout * 1000);
//Magical code end
HttpResponse response = client.execute(request);
System.out.println(response.getStatusLine().getReasonPhrase());
}
bis.close();
}
If i leave out the magical code section, my code hangs on the third part. If I include it, the program runs through fine.
I found the answer! Turns out HttpClient only allows a certain number of connections at a time. According to my code, the default maximum connections is 2. I needed to close each connection after they were complete and the upload ran fine.
Fixed code adds request connection release.
private void uploadParts(String assetId) throws IOException {
//set up post request
HttpClient client = HttpClientBuilder.create().build();
String url = "";
//prepare video
File video = new File("files/video.mp4");
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(video));
int partMaxSize = 1024 * 1024 * 5;
byte[] buffer = new byte[partMaxSize];
double fileSize = video.length();
System.out.println(fileSize);
System.out.println(fileSize / partMaxSize);
int parts = (int) Math.ceil(fileSize / partMaxSize);
System.out.println(parts);
for(int i = 1; i < parts+1; i++) {
String partNumber = i + "";
System.out.println("part: " + partNumber);
int partSize = (int) (i < parts ? partMaxSize : fileSize);
fileSize -= partSize;
int tmp = 0;
tmp = bis.read(buffer);
url = String.format("https://www.site.com/upload/multipart/%s/%s", assetId, partNumber);
final HttpPut request = new HttpPut(url);
request.addHeader("Authorization", "Bearer " + accessToken);
request.addHeader("Content-Type", "application/octet-stream");
request.setEntity(new ByteArrayEntity(buffer));
//Magical code start
int hardTimeout = 5; // seconds
TimerTask task = new TimerTask() {
#Override
public void run() {
if (request != null) {
request.abort();
}
}
};
new Timer(true).schedule(task, hardTimeout * 1000);
//Magical code end
HttpResponse response = client.execute(request);
request.releaseConnection();
System.out.println(response.getStatusLine().getReasonPhrase());
}
bis.close();
}
The timer was working because it was closing my old connections after 10 seconds. Thank you for your input, guys.
Related
I am reading data from API response in batches of bytes which is of Content-Type = text/CSV and using Java's NIO package to transfer bytes between two Channels. I want to write API responses in batches to the same File (APPEND). With below code append doesn't seem to work correctly, it's more of overriding results.
And once all the data is written then I also want to print the number of total lines in CSV.
Version - Java 8
private void downloadFile_NIO(String encoded_token) throws Exception
{
long start_Range = 0;
long end_Range = 5000;
long batch_size = 5000;
long totalsize = 1080612174;
long counter = 0;
//Path path = Paths.get(FILE_NAME);
//long lines = 0;
while (start_Range <= totalsize)
{
URL url = new URL(FILE_URL);
HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
connection.setRequestProperty("Authorization", "Bearer " + encoded_token );
connection.setDoOutput(true);
connection.setRequestMethod("GET");
ReadableByteChannel readableByteChannel = Channels.newChannel(connection.getInputStream());
FileOutputStream fileOutputStream=new FileOutputStream(new File(FILE_NAME),true);
fileOutputStream.getChannel().transferFrom(readableByteChannel, 0, Long.MAX_VALUE);
// lines = Files.lines(path).count();
// System.out.println("lines->" + lines);
System.out.println();
fileOutputStream.close();
readableByteChannel.close();
counter = counter + 1;
if (counter < 2)
{
start_Range = start_Range + batch_size + 1;
end_Range = end_Range + batch_size;
}
else
{
start_Range = start_Range + batch_size;
end_Range = end_Range + batch_size;
}
}
}
I want to try download a file from resume in android studio, i have try my code with open stream and don't have any problem but in this code:
public void startStream2(Context context) {
try {
URL url = new URL(file.getFileUrl());
StrictMode.ThreadPolicy ploicy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
StrictMode.setThreadPolicy(ploicy);
URLConnection connection = url.openConnection();
int downloaded = 0;
BufferedOutputStream bout;
f = new File(downloadPath, file.getName());
if (f.exists()) {
downloaded = (int) f.length();
connection.setRequestProperty("Range", "bytes=" + (f.length()) + "-");
}
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
connection.setRequestProperty("Accept","*/*");
connection.connect();
int response=((HttpURLConnection)connection).getResponseCode();
Log.e(TAG, "startStream2: "+ response );
if (response>399 && response<601){
InputStreamReader sr;
sr = new InputStreamReader(((HttpURLConnection) connection).getErrorStream(), "UTF-8");
StringBuilder builder = new StringBuilder();
for (int bt = 0; (bt = sr.read()) != -1;) {
builder.append((char)bt);
}
sr.close();
Log.e(TAG, "startStream2: "+builder.toString());
}
InputStream inp=connection.getInputStream();
BufferedInputStream in = new BufferedInputStream(inp);
stream2 = (downloaded == 0) ? new FileOutputStream(f) : new FileOutputStream(f, true);
bout = new BufferedOutputStream(stream2, 1024);
byte[] data = new byte[1024];
int x = 0;
while ((x = in.read(data, 0, 1024)) >= 0) {
bout.write(data, 0, x);
downloaded += x;
int percent = ((int) downloaded * 100) / (int) Size;
//set percent progress
}
}catch (Exception e){
Log.e(TAG, "startStream2: ",e );
}
}
Error log:
startStream2: 405 startStream2:
405 Not Allowed
405 Not Allowed
nginx
startStream2:
java.io.FileNotFoundException:
at
com.android.okhttp.internal.huc.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:242)
I have test the url before and don't have any problem with download from first or resume.
How can i resolve it?
thanks.
connection.setDoOutput(true);
The problem is here. It changes the HTTP verb from GET to POST. You don't want to POST. You aren't sending any output. Remove it.
And if the response code isn't 200 you should not get the input stream, otherwise it will throw an exception. You can get the error stream if you want more info, but there isn't an input stream at this point.
You can also remove
connection.connect();
and
connection.setDoInput(true);
They don't do anything that doesn't already happen.
remove these lines
connection.setDoInput(true);
connection.setDoOutput(true);
Finally i have resolved it by this code:
public void startStream2(Context context) {
try {
URL url = new URL(file.getFileUrl());
StrictMode.ThreadPolicy ploicy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
StrictMode.setThreadPolicy(ploicy);
URLConnection connection = url.openConnection();
int downloaded = 0;
BufferedOutputStream bout;
f = new File(downloadPath, file.getName());
if (f.exists()) {
downloaded = (int) f.length();
connection.setRequestProperty("Range", "bytes=" + (f.length()) + "-");
}
connection.setDoInput(true);
connection.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
connection.setRequestProperty("Accept","*/*");
int response=((HttpURLConnection)connection).getResponseCode();
Log.e(TAG, "startStream2: "+ response );
if (response>399 && response<601){
InputStreamReader sr;
sr = new InputStreamReader(((HttpURLConnection) connection).getErrorStream(), "UTF-8");
StringBuilder builder = new StringBuilder();
for (int bt = 0; (bt = sr.read()) != -1;) {
builder.append((char)bt);
}
sr.close();
Log.e(TAG, "startStream2: "+builder.toString());
}
InputStream inp=connection.getInputStream();
BufferedInputStream in = new BufferedInputStream(inp);
stream2 = (downloaded == 0) ? new FileOutputStream(f) : new FileOutputStream(f, true);
bout = new BufferedOutputStream(stream2, 1024);
byte[] data = new byte[1024];
int x = 0;
while ((x = in.read(data, 0, 1024)) >= 0) {
bout.write(data, 0, x);
downloaded += x;
int percent = ((int) downloaded * 100) / (int) Size;
//set percent progress
}
}catch (Exception e){
Log.e(TAG, "startStream2: ",e );
}
}
Thank a lot!
I want to build android download speed test. To do that I am using TrafficStats class. Problem is that I am getting wrong results. Results are almost the same when I run the test but I put heavy load on my Internet connection before I run test. I download file for 30 seconds and after that (or when file is downloaded) and then calculate bytes using TrafficStats
Does someone knows where is the problem?
This is code that I am using:
#Override
protected String doInBackground(String... urls) {
String downloaded ="";
// String uploaded = "";
try{
long BeforeTime = System.currentTimeMillis();
long TotalTxBeforeTest = TrafficStats.getTotalTxBytes();
long TotalRxBeforeTest = TrafficStats.getTotalRxBytes();
URL url = new URL(urls[0]);
URLConnection connection = new URL(urls[0]).openConnection();
connection.setUseCaches(false);
connection.connect();
InputStream input = connection.getInputStream();
BufferedInputStream bufferedInputStream = new BufferedInputStream(input);
byte[] buffer = new byte[1024];
int n = 0;
long endLoop = BeforeTime+30000;
while(System.currentTimeMillis() < endLoop) {
/* if (bufferedInputStream.read(buffer) != -1){
break;
}*/
}
long TotalTxAfterTest = TrafficStats.getTotalTxBytes();
long TotalRxAfterTest = TrafficStats.getTotalRxBytes();
long AfterTime = System.currentTimeMillis();
double TimeDifference = AfterTime - BeforeTime;
double rxDiff = TotalRxAfterTest - TotalRxBeforeTest;
double txDiff = TotalTxAfterTest - TotalTxBeforeTest;
Log.e(TAG, "Download skinuto. "+ rxDiff);
if((rxDiff != 0) && (txDiff != 0)) {
double rxBPS = (rxDiff / (TimeDifference/1000)); // total rx bytes per second.
double txBPS = (txDiff / (TimeDifference/1000)); // total tx bytes per second.
downloaded = String.valueOf(rxBPS) + "B/s. Total rx = " + rxDiff;
// uploaded = String.valueOf(txBPS) + "B/s. Total tx = " + txDiff;
}
else {
downloaded = "No downloaded bytes.";
}
}
catch(Exception e){
Log.e(TAG, "Error while downloading. "+ e.getMessage());
}
return downloaded;
}
I tried your code - it seems to work fine for me BUT i changed
while(System.currentTimeMillis() < endLoop) {
/* if (bufferedInputStream.read(buffer) != -1) {
break;
}*/
}
to
while(System.currentTimeMillis() < endLoop) {
if (bufferedInputStream.read(buffer) == -1){
break;
}
}
since read returns -1 if the end of the stream is reached.
I would like to send big file by dividing it to small parts and send them separately.
I tried to use the hedder "Range" and got "org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity".
// create authenticate client
DefaultHttpClient client = new DefaultHttpClient();
// create HTTP put with the file
HttpPut httpPut = new HttpPut(url);
final File recordingFile = new File(mDir, mName);
long fileLength = recordingFile.length();
for (int i=0; i < fileLength; i += 4096) {
int length = Math.min(4096, (int)recordingFile.length() - i);
InputStreamEntity entity = new InputStreamEntity(inputStream, length);
httpPut.setEntity(entity);
httpPut.addHeader("Connection", "Keep-Alive");
httpPut.addHeader("Range", "bytes=" + i + "-" + (i + length));
// Execute
HttpResponse res = client.execute(httpPut);
int statusCode = res.getStatusLine().getStatusCode();
}
I also tried "Content-Range" header (instead of "Range") and I got the same exception.
httpPut.addHeader("Content-Range", "bytes=" + i + "-" + (i + length) + "/" + fileLength);
httpPut.addHeader("Accept-Ranges", "bytes");
You repeatedly send multiple of 4096 bits. E.g. let's take the first two steps:
i = 0
Send range 0-4096
i = 4096
Send range 4096-8192.
Fix this lines:
for (int i=0; i <= fileLength; i += 4097) {
int length = Math.min(4096, (int)recordingFile.length() - i + 1);
/*...*/
}
and it should work fine.
Update:
Maybe the problem is that for some reasons (e.g. authentication failure) it tries to resend the same chunk again, in which case the inputstream is already consumed.
Try using a ByteArrayEntity instead of InputStreamEntity, something like this:
ByteArrayInputStream bis = new ByteArrayInputStream(recordingFile);
for (int i=0; i <= fileLength; i += 4097) {
int length = Math.min(4096, (int)recordingFile.length() - i + 1);
byte[] bytes = new byte[length];
bis.read(bytes);
ByteArrayEntity entity = ByteArrayEntity(bytes);
/*...*/
}
to fetch() over 1M in app engine,i use the range header and then combine those pieces.and my codes:
int startpos=0;
int endpos;
int seg=1;
int len=1;
while(len>0){
endpos=startpos+seg;
httpConn = (HttpURLConnection) u.openConnection();
httpConn.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14");
con.setRequestProperty("Range", "bytes=" + startpos + "-" + endpos);
con.connect();
InputStream in=con.getInputStream();
len=con.getContentLength();
byte[] b=new byte[len];
in.read(b, 0, len);
startpos+=len;
}
but when it goes to the "InputStream in=con.getInputStream();",its debug is " URL Fetch Response too large problems"
so i don't know what the wrong with these codes.
and there are other ways to fetch() over 1M?
Not all HTTP servers support range requests, especially when it comes to frameworks serving dynamic content - they'll simply ignore the Range header and send you the whole response.
The recent release of 1.4.0 increased the URLFetch response limit to 32MB, though, so you no longer need to do this.
I had the same problem and hacked up a little class to simulate an input stream on Appengine using the HTTP range parameter. It allows you to read files bigger then the limit in a line-oriented fashion. I am attaching it below, although you may need to adapt it for your purposes:
package com.theodorebook.AEStreamer;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.util.Arrays;
import java.util.logging.Logger;
/**
* A class to simulate a stream in appengine, which insists on downloading
* an entire URL before letting you do anything with it. This enables one
* to read files larger than the size limits.
*
* #author Theodore Book (theodorebook at gmail dot com)
*
*/
public class AEStreamer {
private static final int BITE_SIZE = 0x10000; //How big a chunk to grab at a time
private static final byte TERMINATOR = '\n'; //String terminator
private int mCurrentPosition = 0; //The current position in the file
private int mOffset = -1; //The offset of the current block
private long mValidBytes = 0; //The number of valid bytes in the chunk
private byte[] mChunk = new byte[BITE_SIZE];
private boolean mComplete = false;
private String mURL;
private static final Logger log = Logger.getLogger(AEStreamer.class.getName());
public AEStreamer(String url) {
mURL = url;
}
/**
* Returns the next line from the source, or null on empty
* #return
*/
public String readLine() {
String line = "";
//See if we have something to read
if (mCurrentPosition >= mOffset + mValidBytes) {
if (mComplete)
return null;
readChunk();
}
if (mValidBytes == 0)
return null;
//Read until we reach a terminator
int endPtr = mCurrentPosition - mOffset;
while (mChunk[endPtr] != TERMINATOR) {
endPtr++;
//If we reach the end of the block
if (endPtr == mValidBytes) {
line += new String(Arrays.copyOfRange(mChunk, mCurrentPosition - mOffset, endPtr));
mCurrentPosition += (endPtr - mCurrentPosition + mOffset);
if (mComplete) {
return line;
} else {
readChunk();
endPtr = mCurrentPosition - mOffset;
}
}
}
line += new String(Arrays.copyOfRange(mChunk, mCurrentPosition - mOffset, endPtr));
mCurrentPosition += (endPtr - mCurrentPosition + mOffset);
mCurrentPosition++;
return line;
}
/**
* Reads the next chunk from the server
*/
private void readChunk() {
if (mOffset < 0)
mOffset = 0;
else
mOffset += BITE_SIZE;
try {
URL url = new URL(mURL);
URLConnection request = url.openConnection();
request.setRequestProperty("Range", "bytes=" + (mOffset + 1) + "-" + (mOffset + BITE_SIZE));
InputStream inStream = request.getInputStream();
mValidBytes = inStream.read(mChunk);
inStream.close();
} catch (Exception e) {
log.severe("Unable to read " + mURL + ": " + e.getLocalizedMessage());
mComplete = true;
mValidBytes = 0;
return;
}
if (mValidBytes < BITE_SIZE)
mComplete = true;
//log.info("Read " + mValidBytes + " bytes");
}
}