This question already has answers here:
Determine the size of an InputStream
(13 answers)
Closed 3 years ago.
I'm having an InputStream from a ProcessBuilder that acutally reads the stdout stream.
Question: how can I know the size of that inmemory InputStream, so I can write it to a HttpResponse http header?
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
OutputStream out = response.getOutputStream();
int bytes;
while ((bytes = br.read()) != -1) {
out.write(bytes);
}
//how can I know the size of the inmemory stream/file written?
//response.setContentLength((int) pdfFile.length());
There is no such thing as the size of an input stream. Consider a program which never exits, or a socket peer which never stops sending. And you don't need to know to write it to an HttpResponse header. The Content-length is managed automatically for you.
Try this
InputStream is = process.getInputStream();
ByteArrayOutputStream os = new ByteArrayOutputStream();
int b;
while ((b = is.read()) != -1)
os.write(b);
response.setContentLength(os.size());
response.getOutputStream().write(os.toByteArray());
If you really want to set the content length header, you'll need to read the entire stream before writing to the response OutputStream
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] bytes = new byte[1024];
int count;
while ((count = in.read(bytes)) > 0) {
out.write(bytes, 0, count);
}
response.setContentLength(out.size();
out.writeTo(response.getOutputStream());
Note: With this approach you've now read the entire stream into memory, this will have an impact on available memory and likely won't scale well.
import org.apache.commons.io.IOUtils;
byte[] bytes = IOUtils.toByteArray(inputStream);
log.message("bytes .lenght "+bytes.length);
if (bytes.length > 400000)
//some byte range limit`enter code can add any byte range
{
throw new Exception("File Size is larger than 40 MB ..");
}
An InputStream inherently doesn't have a size. It could conceivably keep delivering bytes forever. Or the producing end could end the stream without warning.
If you must find out the length, then you have to read to the end, counting the bytes, and report the length when you finish.
You're worrying about HTTP's Content-length header, and you've got a point. The fact is that the original version of HTTP was not designed for large, dynamically generated content. The protocol inherently expects you to know the size of the content before you start writing it - yet how is that possible if it's (for example) an ongoing chat, or the output of a video camera?
The solution is HTTP's chunked transfer encoding. Here you don't set a Content-Length header. You set Transfer-Encoding: chunked, then write the content as chunks, each of which has a size header.
The HTTP RFC has details one this, or https://en.wikipedia.org/wiki/Chunked_transfer_encoding is slightly more friendly.
However most HTTP APIs hide this detail from you. Unless you are developing a web library from scratch (perhaps for academic reasons), you shouldn't have to think about Content-Length or Transfer-Encoding.
Related
I am trying to peek at an input stream contents from HttpClient, up to 64k bytes.
The stream comes from an HttpGet, nothing unusual about it:
HttpGet requestGet = new HttpGet(encodedUrl);
HttpResponse httpResponse = httpClient.execute(requestGet);
int status = httpResponse.getStatusLine().getStatusCode();
if (status == HttpStatus.SC_OK) {
return httpResponse.getEntity().getContent();
}
The input stream it returns is of type org.apache.http.conn.EofSensorInputStream
Our use-case is such that we need to "peek" at the first (up to 64k) bytes of the input stream. I use an algorithm described here How do I peek at the first two bytes in an InputStream?
PushbackInputStream pis = new PushbackInputStream(inputStream, DEFAULT_PEEK_BUFFER_SIZE);
byte [] peekBytes = new byte[DEFAULT_PEEK_BUFFER_SIZE];
int read = pis.read(peekBytes);
if (read < DEFAULT_PEEK_BUFFER_SIZE) {
byte[] trimmed = new byte[read];
System.arraycopy(peekBytes, 0, trimmed, 0, read);
peekBytes = trimmed;
}
pis.unread(peekBytes);
When I use a ByteArrayInputStream, this works with no problem.
The Issue: When using the org.apache.http.conn.EofSensorInputStream I only get a small number of bytes at the beginning of the stream. usually around 400 bytes. When I expected up to 64k bytes.
I also tried using a BufferedInputStream where I read up to the first 64k bytes then call a .reset() but that doesn't work either. Same issue.
Why might this be? I do not think anything is closing the stream because if you call IOUtils.toString(inputStream) I do get all the content.
See InputStream#read(byte[] b,int off,int len) contract:
Reads up to len bytes of data from the input stream into an array of
bytes. An attempt is made to read as many as len bytes, but a
smaller number may be read. The number of bytes actually read is
returned as an integer
Instead of using this method, use IOUtils.read which reads until you get the number of bytes you requested in a loop.
I'm using a InputStream buffer as following. I'd like to know when it is actually buffering (filling itself with data)... I'm feeding it with internet stream. I put Logs before and after len = in.read(buffer) but they are logged in the same time (so the process is not here).
conn = new URL(StringUrls[0]).openConnection();
conn.setReadTimeout(5000);
conn.setConnectTimeout(5000);
in = conn.getInputStream();
int len=-1;
buffer = new byte[1024];
Log.v("buffer", "buffering...");
len = in.read(buffer);
Log.v("buffer", "buffered");
http://docs.oracle.com/javase/7/docs/api/java/io/InputStream.html#read(byte[]) states:
This method blocks until input data is available, end of file is detected, or an exception is thrown.
In other words, it is being filled when it has the data to be filed with.
When you open a connection, the data are ready to be read. That is why InputStream does not have to wait for anything.
Ok, So I'm making a Java program that has a server and client and I'm sending a Zip file from server to client. I have sending the file down, almost. But recieving I've found some inconsistency. My code isn't always getting the full archive. I'm guessing it's terminating before the BufferedReader has the full thing. Here's the code for the client:
public void run(String[] args) {
try {
clientSocket = new Socket("jacob-custom-pc", 4444);
out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedInputStream(clientSocket.getInputStream());
BufferedReader inRead = new BufferedReader(new InputStreamReader(in));
int size = 0;
while(true) {
if(in.available() > 0) {
byte[] array = new byte[in.available()];
in.read(array);
System.out.println(array.length);
System.out.println("recieved file!");
FileOutputStream fileOut = new FileOutputStream("out.zip");
fileOut.write(array);
fileOut.close();
break;
}
}
}
} catch(IOException e) {
e.printStackTrace();
System.exit(-1);
}
}
So how can I be sure the full archive is there before it writes the file?
On the sending side write the file size before you start writing the file. On the reading side Read the file size so you know how many bytes to expect. Then call read until you have gotten everything you expect. With network sockets it may take more than one call to read to get everything that was sent. This is especially true as your data gets larger.
HTTP sends a content-length: x+\n in bytes. This is elegant, it might throw a TimeoutException if the conn is broken.
You are using a TCP socket. The ZIP file is probably larger than the network MTU, so it will be split up into multiple packets and reassembled at the other side. Still, something like this might happen:
client connects
server starts sending. The ZIP file is bigger than the MTU and therefore split up into multiple packets.
client busy-waits in the while (true) until it gets the first packets.
client notices that data has arrived (in.available() > 0)
client reads all available data, writes it to the file and exits
the last packets arrive
So as you can see: Unless the client machine is crazily slow and the network is crazily fast and has a huge MTU, your code simply won't receive the entire file by design. That's how you built it.
A different approach: Prefix the data with the length.
Socket clientSocket = new Socket("jacob-custom-pc", 4444);
DataInputStream dataReader = new DataInputStream(clientSocket.getInputStream());
FileOutputStream out = new FileOutputStream("out.zip");
long size = dataReader.readLong();
long chunks = size / 1024;
int lastChunk = (int)(size - (chunks * 1024));
byte[] buf = new byte[1024];
for (long i = 0; i < chunks; i++) {
dataReader.read(buf);
out.write(buf);
}
dataReader.read(buf, 0, lastChunk);
out.write(buf, 0, lastChunk);
And the server uses DataOutputStream to send the size of the file before the actual file. I didn't test this, but it should work.
How can I make sure I received whole file through socket stream?
By fixing your code. You are using InputStream.available() as a test for end of stream. That's not what it's for. Change your copy loop to this, which is also a whole lot simpler:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Use with any buffer size greater than zero, typically 8192.
In.available() just tells you that there is no data to be consumed by in.read() without blocking (waiting) at the moment but it does not mean the end of stream. But, they may arrive into your PC at any time, with TCP/IP packet. Normally, you never use in.available(). In.read() suffices everything for the reading the stream entirely. The pattern for reading the input streams is
byte[] buf;
int size;
while ((size = in.read(buf)) != -1)
process(buf, size);
// end of stream has reached
This way you will read the stream entirely, until its end.
update If you want to read multiple files, then chunk you stream into "packets" and prefix every one with an integer size. You then read until size bytes is received instead of in.read = -1.
update2 Anyway, never use in.available for demarking between the chunks of data. If you do that, you imply that there is a time delay between incoming data pieces. You can do this only in the real-time systems. But Windows, Java and TCP/IP are all these layers incompatible with real-time.
I would like to send image file from java server to android app using this code:
Server(Java):
File file = new File("./clique.jpg");
FileInputStream stream = new FileInputStream(file);
DataOutputStream writer = new DataOutputStream(socket.getOutputStream());
byte[] contextB = new byte[4096];
int n;
int i = 0;
while ( (n=stream.read(contextB))!=-1 ){
writer.write(contextB, 0, n);
writer.flush();
System.out.println(n);
i+=n;
}
writer.flush();
stream.close();
android app:
DataInputStream reader = new DataInputStream(socket.getInputStream());
byte[] buffer = new byte[4096];
ByteArrayOutputStream content = new ByteArrayOutputStream();
int n;
int i = 0;
reader = new DataInputStream(socket.getInputStream());
while ( (n=reader.read(buffer)) != null){
content.write(buffer, 0, n);
content.flush();
}
Utility.CreateImageFile(content.toByteArray());
What I noticed is that in the android app n which is the number of read bytes is not 4096 while I am sending from server byte blocks of 4096 size,also I can not get n=-1 which is the end of stream,it blocks until I close the app then I get n=-1.
Regarding the number of bytes you read at a time has nothing to do with the number of bytes you write -it very much depends on the network conditions and will be variable with every chunk (basically as many bytes managed to be transmitted in short period of time between your reads as many you will get in the read chunk.
Regarding the end of stream - in your server code you have forgotten to close the output stream (you only close the stream which is input stream - you should also close the writer which in turn will close the underlying output stream.
Two comments:
1) I would really recommend to use Buffered Readers/Writers wrapping the writes/readers - the code you will get will be nicer and you will not have to create/manage buffers yourself.
2) Use try {} finally and close your streams in finally clauses - this is the best practice that will make sure that you will close the streams and free resources even in case of problems while reading/writing.
You got a problem in your android code:
while ( (n=reader.read(buffer)) != null) {
n can not be null.
Use writer.close() instead of writer.flush() after your loop on the server.
I´m having a problem, in my server, after I send a file with X bytes, I send a string saying this file is over and another file is coming, like
FILE: a SIZE: Y\r\n
send Y bytes
FILE a FINISHED\r\n
FILE b SIZE: Z\r\n
send Z byes
FILE b FINISHED\r\n
FILES FINISHED\r\n
In my client it does not recive properly.
I use readline() to get the command lines after reading Y or Z bytes from the socket.
With one file it works fine, with multiple files it rarely works (yeah, I dont know how it worked once or twice)
Here are some code I use to transfer binary
public static void readInputStreamToFile(InputStream is, FileOutputStream fout,
long size, int bufferSize) throws Exception
{
byte[] buffer = new byte[bufferSize];
long curRead = 0;
long totalRead = 0;
long sizeToRead = size;
while(totalRead < sizeToRead)
{
if(totalRead + buffer.length <= sizeToRead)
{
curRead = is.read(buffer);
}
else
{
curRead = is.read(buffer, 0, (int)(sizeToRead - totalRead));
}
totalRead = totalRead + curRead;
fout.write(buffer, 0, (int) curRead);
}
}
public static void writeFileInputStreamToOutputStream(FileInputStream in, OutputStream out, int bufferSize) throws Exception
{
byte[] buffer = new byte[bufferSize];
int count = 0;
while((count = in.read(buffer)) != -1)
{
out.write(buffer, 0, count);
}
}
just for note I could solve replacing readline to this code:
ByteArrayOutputStream ba = new ByteArrayOutputStream();
int ch;
while(true)
{
ch = is.read();
if(ch == -1)
throw new IOException("Conecção finalizada");
if(ch == 13)
{
ch = is.read();
if(ch == 10)
return new String(ba.toByteArray(), "ISO-8859-1");
else
ba.write(13);
}
ba.write(ch);
}
PS: "is" is my input stream from socket: socket.getInputStream();
still I dont know if its the best implementation to do, im tryinf to figure out
There's no readLine() calls in the code here, but to answer your question; Yes, calling BufferedReader.readLine() might very well leave stuff around in its internal buffer. It's buffering the input.
If you wrap one of your InputStream in a BufferedReader, you can't really get much sane behavior if you read from the BufferedReader and then later on read from the InputStream.
You could read bytes from your InputStream and parse out a text line from that by looking for a pair of \r\n bytes. When you got a line saying "FILE: a SIZE: Y\r\n" , you go on as usual, except the buffer you used to parse lines might contain the first few bytes of your file, so write those bytes out first.
Or you use the idea of FTP and use one TCP stream for commands and one TCP stream for the actual transfer, reading from the command stream with a BufferedReader.readLine(), and reading the data as you already do with an InputStream.
Yes, the main point of a BufferedReader is to buffer the data. It is reading input from its underlying Reader in bigger chunks to avoid having multiple small reads.
That it has a readLine() method is just a nice bonus which is made easily possible by the buffering.
You may want to use a DataInputStream (on top of a BufferedInputStream) and it's readLine() method, if you really have to mix text and binary data over the same connection - read the data from the same DataInputStream. (But take care about the encoding here.)
Call flush() on the OutputStream after you've written data that you want to be certain has been sent. So essentially at the end of each file call flush().
I guess you must flush your output stream in order to make sure any buffered bytes are properly sent down the stream. Closing the stream will equally have this process run.
The Javadocs for flush say:
Flushes this output stream and forces
any buffered output bytes to be
written out. The general contract of
flush is that calling it is an
indication that, if any bytes
previously written have been buffered
by the implementation of the output
stream, such bytes should immediately
be written to their intended
destination.