I am trying to upload a file on a server and display how many bytes are uploaded per second by this way:
public void RunUploadTest () {
try {
URL url = new URL(serverLink);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("content-type", "video/mp4");
for(int i =0; i<10; i++) {
FileInputStream fis = new FileInputStream(myFileVideo);
DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
int bytesWrite = 0;
byte[] buffer = new byte[512];
Stopwatch timer = new Stopwatch();
int read;
while ((read = fis.read(buffer)) != -1&& timer.elapsedTime()<1000) {
dos.write(buffer, 0, read);
dos.flush();
bytesWrite++;
}
Log.d("Upload", "Bytes written: " + bytesWrite*512);
}
fis.close();
} catch (IOException e){
e.printStackTrace();
}
}
The problem is that is not calculating how many bytes are uploaded as I expect.
Do you have any idea why is not working?
If you have 10 times the same bytes written, there are 2 options:
your file is written in less than 1 second;
your elapsedTime method, returns in seconds(for example) and not in milliseconds
You probably don't need that for loop too, it makes you read the file 10 times.
I would rewrite your code like this:
public void RunUploadTest () {
try {
URL url = new URL(serverLink);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("content-type", "video/mp4");
FileInputStream fis = new FileInputStream(myFileVideo);
DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
int bytesWrite = 0;
byte[] buffer = new byte[512];
int read;
while((read = fis.read(buffer))) {
Stopwatch timer = new Stopwatch();
while (timer.elapsedTime()<1) { //my guess here is that this method returns in seconds
dos.write(buffer, 0, read);
dos.flush();
bytesWrite += read; //taken from Yoni Gross answer
}
Log.d("Upload", "Bytes written: " + bytesWrite);
}
fis.close();
} catch (IOException e){
e.printStackTrace();
}
}
There are few problems with your code:
1. You are not counting right
This is a relatively minor issue, but the last buffer you read/write will probably not going to be 512 bytes long, so your calculation will not be precise.
you should change
bytesWrite++;
}
Log.d("Upload", "Bytes written: " + bytesWrite*512);
to
bytesWrite += read;
}
Log.d("Upload", "Bytes written: " + bytesWrite);
2. You are stopping the upload after 1 second anyway
The following condition:
while ((read = fis.read(buffer)) != -1&& timer.elapsedTime()<1000)
Will stop either when the file has finished reading, or 1 second elapsed. Unless a very small file, most likely file upload will stop after 1 second.
I'm not completely sure if you want an update every second, or just the average.
In case of average you should just change the while condition to:
while (read = fis.read(buffer))
And change log to:
float uploadRate = (float)(bytesWrite / timer.elapsedTime()) / 1000.0f;
String logString = String.format("Upload rate: %.2f bytes per second", uploadRate)
Log.d("Upload", logString);
In case you do want to print progress every second, you'll need to do something more fancy and expensive, like:
while (read = fis.read(buffer)) {
dos.write(buffer, 0, read);
dos.flush();
bytesWrite += read;
if (timer.shouldPrintSecondsProgress(){
Log.d("Upload", "Bytes written: " + bytesWrite);
}
}
StopWatch.shouldPrintSecondsProgress should be something like:
public boolean shouldPrintSecondsProgress(){
int currentSecond = elapsedTime() / 1000;
if (currentSecond == nextUpdate){ //I've ignored the > case, which is probably an error in our case
nextUpdate++; //this is an int field, marking the next second cause update
return true;
}
return false;
}
3. It's possible you measure only 'send' and not 'upload'
While you are sending all your data and flushing the request stream, some layer of the networking stack might have buffer/async behavior. Actually the roundtrip alone might effect calculation correctness.
It best if you close the output stream prior to measure time elapsed, and/or read the response from server (which will ensure upload is finished).
Related
I am working on a task where I need to download multiple files from a single hyperlink. When I call the one API link I want it to return multiple files.
My current code is only downloading a single file:
try {
URL url = new URL(f_url[0]);
URLConnection conection = url.openConnection();
conection.connect();
// getting file length
int lenghtOfFile = conection.getContentLength();
// input stream to read file - with 8k buffer
InputStream input = new BufferedInputStream(url.openStream(), 8192);
// Output stream to write file
OutputStream output = new FileOutputStream("/sdcard/downloadedfile.jpg");
byte data[] = new byte[1024];
long total = 0;
while ((count = input.read(data)) != -1) {
total += count;
// publishing the progress....
// After this onProgressUpdate will be called
publishProgress(""+(int)((total*100)/lenghtOfFile));
// writing data to file
output.write(data, 0, count);
}
// flushing output
output.flush();
// closing streams
output.close();
input.close();
} catch (Exception e) {
Log.e("Error: ", e.getMessage());
}
check this site . It downloads multiple files and shows its content on your screen
I'm working on a Java application which will stream video from an IP Camera. The video streams from the IP Camera in MJPEG format. The protocol is the following...
--ipcamera (\r\n)
Content-Type: image/jpeg (\r\n)
Content-Length: {length of frame} (\r\n)
(\r\n)
{frame}
(\r\n)
--ipcamera (\r\n)
etc.
I've tried using classes such as BufferedReader and Scanner to read until the "\r\n", however those are meant for text and not binary data, so it becomes corrupt. Is there any way to read the binary stream until it encounters a "\r\n"? Here is my current (broken) code.
EDIT: I've gotten it to work. I updated the code below. However, it's really slow in doing so. I'm not sure if it has anything to do with the ArrayList or not, but it could be the culprit. Any pointers to speed up the code? It's currently taking 500ms to 900ms for a single frame.
public void run() {
long startTime = System.currentTimeMillis();
try {
URLConnection urlConn = url.openConnection();
urlConn.setReadTimeout(15000);
urlConn.connect();
urlStream = urlConn.getInputStream();
DataInputStream dis = new DataInputStream(urlStream);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ArrayList<Byte> bytes = new ArrayList<Byte>();
byte cur;
int curi;
byte[] curBytes;
int length = 0;
while ((curi = dis.read()) != -1) {
cur = (byte) curi;
bytes.add(cur);
curBytes = getPrimativeArray(bytes);
String curBytesString = new String(curBytes, "UTF-8");
if (curBytesString.equals("--ipcamera\r\n")) {
bytes.clear();
continue;
} else if (curBytesString.equals("Content-Type: image/jpeg\r\n")) {
bytes.clear();
continue;
} else if (curBytesString.matches("^Content-Length: ([0-9]+)\r\n$")) {
length = Integer.parseInt(curBytesString.replace("Content-Length: ", "").trim());
bytes.clear();
continue;
} else if (curBytesString.equals("\r\n")) {
if (length == 0) {
continue;
}
byte[] frame = new byte[length];
dis.readFully(frame, 0, length);
writeFrame(frame);
bytes.clear();
break;
}
}
} catch (Exception e) {
e.printStackTrace();
}
long curTime = System.currentTimeMillis() - startTime;
System.out.println(curTime);
}
private byte[] getPrimativeArray(ArrayList<Byte> array) {
byte[] bytes = new byte[array.size()];
for (int i = 0; i < array.size(); i++) {
bytes[i] = array.get(i).byteValue();
}
return bytes;
}
private void writeFrame(byte[] bytes) throws IOException {
File file = new File("C:\\test.jpg");
FileOutputStream fos = new FileOutputStream(file);
fos.write(bytes);
fos.close();
System.out.println("done");
}
Currently you do not cope with the case when data is read in the frame part.
A rough assumption is:
Current version:
else if (line.equals("") && length != 0)
Probably more correct version:
else if (!line.equals("") && length != 0)
You cannot use BufferedReader to read binary, it will corrupt it. I you want to keep things simple, use DataInputStream.readLine(). Though not ideal, it may be the simplest in your case.
Other than using some bad practices and assuming that your URLConnection correctly delivers the data, the example you posted seems to work if you reset the length to zero after reading the frame data.
} else if (line.equals("") && length != 0) {
char[] buf = new char[length];
reader.read(buf, 0, length);
baos.write(new String(buf).getBytes());
//break;
length = 0; // <-- reset length
}
Please note this way all the frame data are written in the same ByteArrayOutputStream consecutively. If you don't want that, you should create a new ByteArrayOutputStream for every new frame you encounter.
You can't use a BufferedReader for part of the transmission and then some other stream for the rest of it. The BufferedReader will fill its buffer and steal some of the data you want to read with the other stream. Use DataInputStream.readLine(), noting that it's deprecated, or else roll your own line-reading code, using the input stream provided by the URLConnection.
Surely you don't have to? URLConnection reads the headers for you. If you want the content-length, use the API to get it. The stuff you get to read starts at the body of the transmission.
I'm trying to make a partial GET request and I expect a 206 response, but I'm still getting 200 OK. How can I make it responded with a 206?
Here is the code I wrote:
HttpURLConnection urlConn = (HttpURLConnection) url.openConnection();
int start = Integer.parseInt(args[1]);
int end = Integer.parseInt(args[2]);
urlConn.setRequestMethod("GET");
urlConn.setRequestProperty("Range", "bytes=" + start + "-" + end);
if (urlConn.getResponseCode() != HttpURLConnection.HTTP_PARTIAL)
System.out.println ("File cannot be downloaded.");
else
{
String filepath = url.getPath();
String filename = filepath.substring (filepath.lastIndexOf('/') + 1);
BufferedInputStream in = new BufferedInputStream (urlConn.getInputStream());
BufferedOutputStream out = new BufferedOutputStream (new FileOutputStream (filename));
int temp;
while ((temp = in.read()) != -1)
{
out.write ((char)temp);
}
out.flush();
out.close();
System.out.println ("File downloaded successfully.");
}
How can I make it responded with a 206?
You can't force a server to respect your Range headers. The HTTP 1.1 spec says:
"A server MAY ignore the Range header."
You comment:
I know the server does not ignore it. At least shouldn't
Well apparently, the server >>IS<< ignoring the header. Or alternatively, the range that you are requesting encompasses the entire document.
I'm trying to implement pause/resume in my download manager, I search the web and read several articles and change my code according them but resume seems not working correctly, Any ideas?
if (!downloadPath.exists())
downloadPath.mkdirs();
if (outputFileCache.exists())
{
downloadedSize = outputFileCache.length();
connection.setAllowUserInteraction(true);
connection.setRequestProperty("Range", "bytes=" + downloadedSize + "-");
connection.setConnectTimeout(14000);
connection.connect();
input = new BufferedInputStream(connection.getInputStream());
output = new FileOutputStream(outputFileCache, true);
input.skip(downloadedSize); //Skip downloaded size
}
else
{
connection.setConnectTimeout(14000);
connection.connect();
input = new BufferedInputStream(url.openStream());
output = new FileOutputStream(outputFileCache);
}
fileLength = connection.getContentLength();
byte data[] = new byte[1024];
int count = 0;
int __progress = 0;
long total = downloadedSize;
while ((count = input.read(data)) != -1 && !this.isInterrupted())
{
total += count;
output.write(data, 0, count);
__progress = (int) (total * 100 / fileLength);
}
output.flush();
output.close();
input.close();
Okay problem fixed, here is my code for other users who wants to implement pause/resume:
if (outputFileCache.exists())
{
connection.setAllowUserInteraction(true);
connection.setRequestProperty("Range", "bytes=" + outputFileCache.length() + "-");
}
connection.setConnectTimeout(14000);
connection.setReadTimeout(20000);
connection.connect();
if (connection.getResponseCode() / 100 != 2)
throw new Exception("Invalid response code!");
else
{
String connectionField = connection.getHeaderField("content-range");
if (connectionField != null)
{
String[] connectionRanges = connectionField.substring("bytes=".length()).split("-");
downloadedSize = Long.valueOf(connectionRanges[0]);
}
if (connectionField == null && outputFileCache.exists())
outputFileCache.delete();
fileLength = connection.getContentLength() + downloadedSize;
input = new BufferedInputStream(connection.getInputStream());
output = new RandomAccessFile(outputFileCache, "rw");
output.seek(downloadedSize);
byte data[] = new byte[1024];
int count = 0;
int __progress = 0;
while ((count = input.read(data, 0, 1024)) != -1
&& __progress != 100)
{
downloadedSize += count;
output.write(data, 0, count);
__progress = (int) ((downloadedSize * 100) / fileLength);
}
output.close();
input.close();
}
It is impossible to tell what is wrong without some more information, however things to note:
You must make a HTTP/1.1 request (it's hard to tell from your sample code)
The server must support HTTP/1.1
The server will tell you what it supports with an Accept-Ranges header in the response
If-Range should be the etag the server gave you for the resource, not the last modified time
You should check your range request with something simple to test the origin actually supports the Range request first (like curl or wget )
It nay be that your server is taking to long to respond (more then the timeout limit) or this is also a fact that not all servers support pause - resume.
It is also a point to ponder that weather the file is downloaded through Http, https, ftp or udp.
Pausing" could just mean reading some of the stream and writing it to disk. When resuming you would have to use the headers to specify what is left to download.
you may try something like :
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
if(ISSUE_DOWNLOAD_STATUS.intValue()==ECMConstant.ECM_DOWNLOADING){
File file=new File(DESTINATION_PATH);
if(file.exists()){
downloaded = (int) file.length();
connection.setRequestProperty("Range", "bytes="+(file.length())+"-");
}
}else{
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
}
connection.setDoInput(true);
connection.setDoOutput(true);
progressBar.setMax(connection.getContentLength());
in = new BufferedInputStream(connection.getInputStream());
fos=(downloaded==0)? new FileOutputStream(DESTINATION_PATH): new FileOutputStream(DESTINATION_PATH,true);
bout = new BufferedOutputStream(fos, 1024);
byte[] data = new byte[1024];
int x = 0;
while ((x = in.read(data, 0, 1024)) >= 0) {
bout.write(data, 0, x);
downloaded += x;
progressBar.setProgress(downloaded);
}
and please try to sync things.
I would start debugging from this line:
connection.setRequestProperty("Range", "bytes=" + downloadedSize + "-");
As from the source code it is not possible to determine what downloadedSize is, it's hard to elaborate further, but the format should be bytes=from-to.
Anyway, I would suggest you to use Apache HttpClient to avoid common pitfalls. Here is a question from someone who uses Apache HttpClient on a similar topic and some sample code is provided.
I think you just need to delete the input.skip(downloadedSize) line. Setting the HTTP header for byte range means the server will skip sending those bytes.
Say you have a file that's 20 bytes long consisting of "aaaaabbbbbcccccddddd", and suppose the transfer is paused after downloading 5 bytes. Then the Range header will cause the server to send "bbbbbcccccddddd", you should read all of this content and append it to the file -- no skip(). But the skip() call in your code will skip "bbbbb" leaving "cccccddddd" to be downloaded. If you've already downloaded at least 50% of the file, then skip() will exhaust all of the input and nothing will happen.
Also, all of the things in stringy05's post apply. You should make sure the server supports HTTP/1.1, make sure the Range header is supported for the resource (dynamically generated content may not support it), and make sure the resource isn't modified using etag and modification date.
For the second time I have this extremely anoying problem with an InputStream.
This InputStream belongs to a Socket that is supposed to receive an image. The code for reading this image is as below:
InputStream input = socket.getInputStream();
InputStreamReader reader = new InputStreamReader(input);
BufferedReader bufferedReader = new BufferedReader(reader);
int total = Integer.parseInt(bufferedReader.readLine());
int bytesRead = 0;
byte[] buffer = new byte[total]; // total is the total size of the image
while (bytesRead < total) {
int next = input.read(buffer, bytesRead, total-bytesRead);
if (next > 0) {
bytesRead += next;
System.out.println("Read: " + bytesRead);
}
}
Now the strange thing is that this code skips the first 1182 bytes of the image, and then reads the remaining part. So when the total size is 15000 bytes, it reads byte 1182-15000.
I checked Wireshark and the whole image is transmitted. The code throws no exceptions. input.read() returns -1 as usual.
Pervious data has been readed from the stream using a BufferedReader. This data is only 5 characters long so it can't contain the missing 1K, but my guess is that the BufferedReader.readLine() method reads (buffers) more bytes from the InputStream than needed. Could this be correct?
I've had the same problem a few months ago but I absolutely have no clue on how I solved it.
Hope anyone can help.
Thanks in advance.
EDIT: I can solve the problem by adding a 100 ms sleep between sending the image size and the image data. It solves the problem but I would still realy like to know a more appropriate solution
As the name **Buffererd**Reader indicates it will snarf more bytes than just the first line from the underlying reader and hence also from the stream. Otherwise if would not be called "buffered".
Unfortunately I'm not aware of any non-deprecated class in Java which allows mixing of binary and textual data in the way you want.
I suggest, that you modify your protocol and transfer the length of the image also in some binary encoding. Then you can stick to InputStream.
My guess is the BufferedReader will assume that whatever reading operations you're performing afterwards will go through it, so it will happily consume input in increments of its buffer size.
One thing you could do is use a BufferedInputStream on top of input, instantiate a DataInputStream on top of that to do the readLine(), and then use the BufferedInputStream in your loop. The documentation says readLine is deprecated because it doesn't convert bytes to characters properly, but I'm hoping that with your first line containing only decimal digits, it shouldn't run into that problem.
I've written a short test, I hope it covers your use case:
import java.net.*;
import java.io.*;
class test{
static byte[] b;
static {
b = new byte[123456];
java.util.Random r = new java.util.Random();
for (int i=0;i<b.length;i++)
b[i] = (byte)(r.nextInt());
}
static void client() throws Exception{
Socket socket = new Socket("127.0.0.1", 9000);
InputStream input = socket.getInputStream();
BufferedInputStream bis = new BufferedInputStream(input);
//read in length
DataInputStream dais = new DataInputStream(bis);
int total = Integer.parseInt(dais.readLine());
System.out.println("Total: "+total);
int bytesRead = 0;
byte[] buffer = new byte[total];
while (bytesRead < total) {
int next = bis.read(buffer, bytesRead, total-bytesRead);
if (next > 0) {
bytesRead += next;
System.out.println("Read: " + bytesRead);
}
}
for (int i=0;i<buffer.length;i++)
if (buffer[i]!=b[i]){
System.err.println("Error");
System.exit(1);
}
System.out.println("OK");
bis.close();
socket.close();
}
static void server() throws Exception{
ServerSocket srv = new ServerSocket(9000);
Socket sock = srv.accept();
OutputStream os = sock.getOutputStream();
BufferedOutputStream bos =new BufferedOutputStream(os);
DataOutputStream daos = new DataOutputStream(bos);
//we're sending the b buffer
//send the length in plain text, followed by newline
byte[]num = String.valueOf(b.length).getBytes();
daos.write(num,0,num.length);
daos.write(10);
//send actual buffer contents
bos.write(b, 0, b.length);
os.close();
srv.close();
sock.close();
}
public static void main(String[]args){
new Thread(new Runnable(){
public void run(){
try{server();}catch(Exception e){e.printStackTrace();}
}
}).start();
new Thread(new Runnable(){
public void run(){
try{client();}catch(Exception e){e.printStackTrace();}
}}).start();
}
}
Have you considered just sending the image data (without the preceding image size) and using an external library such as Apache Commons IO to handle this for you? In particular, I think that you will find IOUtils.toByteArray(InputStream input) interesting, e.g.
InputStream input = socket.getInputStream();
byte[] buffer = IOUtils.toByteArray(input);