Develop Reference

Stream isn't closing after sending file

Excuse me my understanding of streams is slowly building.
I initally tried to stream a music file over to my client but that didn't work out so I moved transferring the entire file via bytes then saving it at the client side. The problem is the input stream is still receiving bytes and so won't break out of the while loop (Where I want to play the song)
Below is part of my client code:
//This is part of a task that carries this out (Part of a UI application)
#Override
protected Void call()
{
try(Socket socket = new Socket(host,portNumber);
ObjectOutputStream toServer = new ObjectOutputStream(socket.getOutputStream());
ObjectInputStream fromServer = new ObjectInputStream(socket.getInputStream()))
{
//Client requests a song from the server (song is a String)
toServer.writeUTF(".Music."+song);
toServer.flush();
//Create a new file (tempSong is a string)
File test = new File(tempSong);
test.createNewFile();
//New file writer
BufferedOutputStream bOS = new BufferedOutputStream(new FileOutputStream(test));
byte[] buffer = new byte[4096];
int current;
/**
* Read the bytes from the server and write the file
* The file is written and I can play it (externally)
* but the while loop doesn't break after writting the file
*/
while ((current = fromServer.read(buffer)) > 0)
{
bOS.write(buffer, 0 , current);
}
System.out.println("Finished writing");
bOS.close();
/**
* down here a method is ran to play the file
* but it never happen because the task is still in the while loop
*/
}
catch (IOException e)
{
e.printStackTrace();
}
return null;
}
below is in the server where the server reads the message and sends the file
/** This is part of a task and the main call area */
#Override
public Void call ()
{
try
{
//Setup I/O
toClient = new ObjectOutputStream(socket.getOutputStream());
ObjectInputStream fromClient = new ObjectInputStream(socket.getInputStream());
while(!socket.isClosed())
{
//If server has received a message
if(fromClient.available() > 0)
{
//Reads message and objects from client
String input = fromClient.readUTF();
if (input.contains(".Music"))
{
findMusic(input, toClient);
}
/**
* more else IFs
*/
}
}
}
catch (Exception e)
{
e.printStackTrace();
}
return null;
}
/**
* this method is part of the task discussed above
*/
//The method that is called
private void findMusic(String input, ObjectOutputStream toClient)
{
logoff();
String[] names = input.split("[.]");
clientManagerTemp.logger("Someone request song: " + names[2] + ".mp3");
File musicFile = AudioUtil.getSoundFile("src/Resources/Songs/" + names[2]+ ".mp3");
byte[] buffer = new byte[(int) musicFile.length()];
try(BufferedInputStream bis = new BufferedInputStream(new FileInputStream(musicFile)))
{
bis.read(buffer, 0, buffer.length);
clientManagerTemp.logger("Sending " + "src/Resources/Songs/" + names[2]+ ".mp3" + "(" + buffer.length + " bytes)");
//write the file to the client
toClient.write(buffer,0, buffer.length);
toClient.flush();
clientManagerTemp.logger("Finished sending");
}
catch (IOException e)
{
e.printStackTrace();
}
}
So as you can see the server sends the file fine and my client receives it. It just won't stop the while loop. Can someone explain why? So I can get a better understanding of how streaming bytes over sockets work
**EDIT
When the client receives the file it can be played even after closing the client and server

You can send the file-length to the client, with this information the client knows when to exit the loop.
Server
private void findMusic(String input, ObjectOutputStream toClient)
{
logoff();
String[] names = input.split("[.]");
clientManagerTemp.logger("Someone request song: " + names[2] + ".mp3");
File musicFile = AudioUtil.getSoundFile("src/Resources/Songs/" + names[2]+ ".mp3");
int fileLength = (int) musicFile.length();
byte[] buffer = new byte[fileLength];
try(BufferedInputStream bis = new BufferedInputStream(new FileInputStream(musicFile)))
{
bis.read(buffer, 0, buffer.length);
clientManagerTemp.logger("Sending " + "src/Resources/Songs/" + names[2]+ ".mp3" + "(" + buffer.length + " bytes)");
//write the file to the client
toClient.writeInt(fileLength);
toClient.write(buffer,0, buffer.length);
toClient.flush();
clientManagerTemp.logger("Finished sending");
}
catch (IOException e)
{
e.printStackTrace();
}
}
Client
byte[] buffer = new byte[4096];
int current;
int fileLength = fromServer.readInt();
while ( fileLength > 0 && (current = fromServer.read(buffer, 0, Math.min(4096,fileLength))) > 0)
{
bOS.write(buffer, 0 , current);
fileLength -= current;
}

This is due to the nature of the read method, seen here Javadocs. This method will block until there is data, so your loop will never end. Now the reason for this is because you never close the stream on the server end, you just flush it, which forces a send of all the data currently in the buffer, but will not close the stream. If you call .close() on the stream from the server side then that should exit the while loop on the client side and continue to your playing code.
I haven't tested this, but from the documentation and a brief look at your code, that seems to be the issue.

If you don't close the connection (server side), there will not be an end-of-file/end-of-stream and your client side will run the loop forever, unless you configured a read timeout.
If you won't to close the loop automatically after sending the file is finished, send the count of bytes (size of file) first to client and then the file itself. This makes it possible to read only a exact number of bytes on client side and close the connection when finished.
Alternative you can send a special sequence and check for them at client side to mark the end of stream.

The reason you program is stuck is due to the behaviour of ObjectInputStream.read(). This method is blocking until it has read some data or the stream is closed. Since you never close the stream the 2nd condition will never be met. So there is only option 1. To return when data arrived. Unfortuenatly there is no way to recognize the end of a file unless you define it yourself e.g. by sending a unique sequenz or something that you can recognize to break the while loop.
Example:
while ((current = fromServer.read(buffer)) > 0)
{
// example value 42 could be anything else as well
if(current == -42)
{
break;
}
bOS.write(buffer, 0 , current);
}
The downside of this is, that your data could also contain this special value so you gotta think about different possibilities e.g memorize the sequence up to 3 values and compare them to an 'interrupt sequence'.

DataInputStream read not blocking

First off forgive me if I am mistaken for how blocking works, to my understanding blocking will pause the thread until it is ready, for exsample when reading user input the program will wait until the user hits return.
My problem is that instead of waiting for data to become available it reads bytes with the value 0. Is there a way to block until data becoms available?
The method readBytes is called in a loop.
public byte[] readBytes(){
try{
//read the head int that will be 4 bytes telling the number of bytes that follow containing data
byte[] rawLen = new byte[4];
socketReader.read(rawLen);
ByteBuffer bb = ByteBuffer.wrap(rawLen);
int len = bb.getInt();
byte[] data = new byte[len];
if (len > 0) {
socketReader.readFully(data);
}
return data;
} catch (Exception e){
e.printStackTrace();
logError("Failed to read data: " + socket.toString());
return null;
}
}

If read() returned -1, the peer has disconnected. You aren't handling that case. If you detect end of stream you must close the connection and stop reading. At present you have no way of doing so. You need to reconsider your method signature.
You should use readInt() instead of those four lines of code that read the length. At present you are assuming have read four bytes without actually checking. readInt() will check for you.
This way also you will never get out of sync with the sender, which at present is a serious risk.

How to read from byte stream in Java (PHP/Java socket communication)

So. I have a PHP socket server and Java socket client.
Maybe it's a stupid question... but i didn't find an answer.
In client, i need to read incoming bytes from input stream and put them into byte array.
I tried to do so:
[CLIENT (Java)]
public static byte[] read(BufferedInputStream in)
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[32768];
while (true) {
int readBytesCount = in.read(buffer);
if (readBytesCount == -1) {
break;
}
if (readBytesCount > 0) {
baos.write(buffer, 0, readBytesCount);
}
}
baos.flush();
baos.close();
return baos.toByteArray();
}
[SERVER (PHP)]
function write(&$client, $message)
{
$message = explode("\n", $message);
foreach ($message as $line)
{
socket_write($client['sock'], $line."\0");
}
}
But when it read all bytes, in.read() doesn't return -1, so the cycle doesn't stop.
One time it returns 13 (length) and then - nothing.
Any ideas?
SOLVED!
[CLIENT (Java)]
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[32768];
while (true) {
int readBytesCount = in.read(buffer);
if (getString(buffer).contains("#!EOS!#")) {
baos.flush();
baos.close();
return baos.toByteArray();
}
if (readBytesCount > 0) {
baos.write(buffer, 0, readBytesCount - 1);
}
}
[SERVER (PHP)]
function write(&$client, $message)
{
$message = explode("\n", $message);
$message = str_replace("\r", "", $message);
foreach ($message as $line)
{
rLog("Write to ".$client['ip'].": ".$line);
socket_write($client['sock'], $line."\0") or die("[".date('Y-m-d H:i:s')."] Could not write to socket\n");
}
socket_write($client['sock'], "#!EOS!#");
}

If your socket is still opened I believe that read will never retun -1. You do not close the trasmission ( in terms of file an EOF is not reached ). I believe that your read call returns 0 when your message is end ( instead of -1 ).
Java docs:
Read returns:
* The specified number of bytes have been read,
* The read method of the underlying stream returns -1, indicating end-of-file, or
* The available method of the underlying stream returns zero, indicating that further input requests would block.
So I believe you need to send a terminating character sequence, or the number of characters you will send, in order to identify on client-side the boundaries of your chunk of data ( the message ) inside the streaming of data.

I agree with Stefano. No EOF or terminate signal is happening on the php server side. socket_write is just writing $message to the client side. If your intent is to terminate the connection after the entire message has been written, then you might have to explicitly use socket_shutdown/socket_close.
You can also try socket_send with flags. Here is the doc for that http://www.php.net/manual/en/function.socket-send.php

Is Socket.getInputStream().read(byte[]) guaranteed to not block after at least some data is read?

The JavaDoc for the class InputStream says the following:
Reads up to len bytes of data from the input stream into an array of
bytes. An attempt is made to read as many as len bytes, but a smaller
number may be read. The number of bytes actually read is returned as
an integer. This method blocks until input data is available, end of
file is detected, or an exception is thrown.
This corresponds to my experience as well. See for instance the example code below:
Client:
Socket socket = new Socket("localhost", PORT);
OutputStream out = socket.getOutputStream();
byte[] b = { 0, 0 };
Thread.sleep(5000);
out.write(b);
Thread.sleep(5000);
out.write(b);
Server:
ServerSocket server = new ServerSocket(PORT);
Socket socket = server.accept();
InputStream in = socket.getInputStream();
byte[] buffer = new byte[4];
System.out.println(in.read(buffer));
System.out.println(in.read(buffer));
Output:
2 // Two bytes read five seconds after Client is started.
2 // Two bytes read ten seconds after Client is started.
The first call to read(buffer) blocks until input data is available. However the method returns after two bytes are read, even though there is still room in the byte buffer, which corresponds with the JavaDoc stating that 'An attempt is made to read as many as len bytes, but a smaller number may be read'. However, is it guaranteed that the method will not block once at least one byte of data is read when the input stream comes from a socket?
The reason I ask is that I saw the following code in the small Java web server NanoHTTPD, and I wondered if a HTTP Request smaller than 8k bytes (which most requests are) potientially could make the thread block indefinately unless there is a guarantee that it won't block once some data is read.
InputStream is = mySocket.getInputStream();
// Read the first 8192 bytes. The full header should fit in here.
byte[] buf = new byte[8192];
int rlen = is.read(buf, 0, bufsize);
Edit:
Let me try to illustrate once more with a relatively similar code example. EJP says that the method blocks until either EOS is signalled or at least one byte of data has arrived, in which case it reads however many bytes of data have arrived, without blocking again, and returns that number, which corresponds to the JavaDoc for method read(byte[], int, int) in the class InputStream. However, if one actually looks at the source code it is clear that the method indeed blocks until the buffer is full. I've tested it by using the same Client as above and copying the InputStream-code to a static method in my server example.
public static void main(String[] args) throws Exception {
ServerSocket server = new ServerSocket(PORT);
Socket socket = server.accept();
InputStream in = socket.getInputStream();
byte[] buffer = new byte[4];
System.out.println(read(in, buffer, 0, buffer.length));
}
public static int read(InputStream in, byte b[], int off, int len) throws IOException {
if (b == null) {
throw new NullPointerException();
}
else if (off < 0 || len < 0 || len > b.length - off) {
throw new IndexOutOfBoundsException();
}
else if (len == 0) {
return 0;
}
int c = in.read();
if (c == -1) {
return -1;
}
b[off] = (byte)c;
int i = 1;
try {
for (; i < len; i++) {
c = in.read();
if (c == -1) {
break;
}
b[off + i] = (byte)c;
}
}
catch (IOException ee) {
}
return i;
}
This code will have as its output:
4 // Four bytes read ten seconds after Client is started.
Now clearly there is data available after 5 seconds, however the method still blocks trying to fill the entire buffer. This doesn't seem to be the case with the input stream that Socket.getInputStream() returns, but is it guaranteed that it will never block once data is available, like the JavaDoc says but not like the source code shows?

However, is it guaranteed that the method will not block once at least one byte of data is read when the input stream comes from a socket?
I don't think this question means anything. The method blocks until either EOS is signalled or at least one byte of data has arrived, in which case it reads however many bytes of data have arrived, without blocking again, and returns that number.
I saw the following code in the small Java web server NanoHTTPD
The code is wrong. It makes the invalid assumption that the entire header will be delivered in the first read. I would expect to see a loop here, that loops until a blank line is detected.
I wondered if a HTTP Request smaller than 8k bytes (which most requests are) potientially could make the thread block indefinitely unless there is a guarantee that it won't block once some data is read.
Again I don't think this means anything. The method will block until at least one byte has arrived, or EOS. Period.

Java -- How to read an unknown number of bytes from an inputStream (socket/socketServer)?

Looking to read in some bytes over a socket using an inputStream. The bytes sent by the server may be of variable quantity, and the client doesn't know in advance the length of the byte array. How may this be accomplished?
byte b[];
sock.getInputStream().read(b);
This causes a 'might not be initialized error' from the Net BzEAnSZ. Help.

You need to expand the buffer as needed, by reading in chunks of bytes, 1024 at a time as in this example code I wrote some time ago
byte[] resultBuff = new byte[0];
byte[] buff = new byte[1024];
int k = -1;
while((k = sock.getInputStream().read(buff, 0, buff.length)) > -1) {
byte[] tbuff = new byte[resultBuff.length + k]; // temp buffer size = bytes already read + bytes last read
System.arraycopy(resultBuff, 0, tbuff, 0, resultBuff.length); // copy previous bytes
System.arraycopy(buff, 0, tbuff, resultBuff.length, k); // copy current lot
resultBuff = tbuff; // call the temp buffer as your result buff
}
System.out.println(resultBuff.length + " bytes read.");
return resultBuff;

Assuming the sender closes the stream at the end of the data:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buf = new byte[4096];
while(true) {
int n = is.read(buf);
if( n < 0 ) break;
baos.write(buf,0,n);
}
byte data[] = baos.toByteArray();

Read an int, which is the size of the next segment of data being received. Create a buffer with that size, or use a roomy pre-existing buffer. Read into the buffer, making sure it is limited to the aforeread size. Rinse and repeat :)
If you really don't know the size in advance as you said, read into an expanding ByteArrayOutputStream as the other answers have mentioned. However, the size method really is the most reliable.

Without re-inventing the wheel, using Apache Commons:
IOUtils.toByteArray(inputStream);
For example, complete code with error handling:
public static byte[] readInputStreamToByteArray(InputStream inputStream) {
if (inputStream == null) {
// normally, the caller should check for null after getting the InputStream object from a resource
throw new FileProcessingException("Cannot read from InputStream that is NULL. The resource requested by the caller may not exist or was not looked up correctly.");
}
try {
return IOUtils.toByteArray(inputStream);
} catch (IOException e) {
throw new FileProcessingException("Error reading input stream.", e);
} finally {
closeStream(inputStream);
}
}
private static void closeStream(Closeable closeable) {
try {
if (closeable != null) {
closeable.close();
}
} catch (Exception e) {
throw new FileProcessingException("IO Error closing a stream.", e);
}
}
Where FileProcessingException is your app-specific meaningful RT exception that will travel uninterrupted to your proper handler w/o polluting the code in between.

The simple answer is:
byte b[] = new byte[BIG_ENOUGH];
int nosRead = sock.getInputStream().read(b);
where BIG_ENOUGH is big enough.
But in general there is a big problem with this. A single read call is not guaranteed to return all that the other end has written.
If the nosRead value is BIG_ENOUGH, your application has no way of knowing for sure if there are more bytes to come; the other end may have sent exactly BIG_ENOUGH bytes ... or more than BIG_ENOUGH bytes. In the former case, you application will block (for ever) if you try to read. In the latter case, your application has to do (at least) another read to get the rest of the data.
If the nosRead value is less than BIG_ENOUGH, your application still doesn't know. It might have received everything there is, part of the data may have been delayed (due to network packet fragmentation, network packet loss, network partition, etc), or the other end might have blocked or crashed part way through sending the data.
The best answer is that EITHER your application needs to know beforehand how many bytes to expect, OR the application protocol needs to somehow tell the application how many bytes to expect or when all bytes have been sent.
Possible approaches are:
the application protocol uses fixed message sizes (not applicable to your example)
the application protocol message sizes are specified in message headers
the application protocol uses end-of-message markers
the application protocol is not message based, and the other end closes the connection to say that is the end.
Without one of these strategies, your application is left to guess, and is liable to get it wrong occasionally.
Then you use multiple read calls and (maybe) multiple buffers.

Stream all Input data into Output stream. Here is working example:
InputStream inputStream = null;
byte[] tempStorage = new byte[1024];//try to read 1Kb at time
int bLength;
try{
ByteArrayOutputStream outputByteArrayStream = new ByteArrayOutputStream();
if (fileName.startsWith("http"))
inputStream = new URL(fileName).openStream();
else
inputStream = new FileInputStream(fileName);
while ((bLength = inputStream.read(tempStorage)) != -1) {
outputByteArrayStream.write(tempStorage, 0, bLength);
}
outputByteArrayStream.flush();
//Here is the byte array at the end
byte[] finalByteArray = outputByteArrayStream.toByteArray();
outputByteArrayStream.close();
inputStream.close();
}catch(Exception e){
e.printStackTrace();
if (inputStream != null) inputStream.close();
}

Either:
Have the sender close the socket after transferring the bytes. Then at the receiver just keep reading until EOS.
Have the sender prefix a length word as per Chris's suggestion, then read that many bytes.
Use a self-describing protocol such as XML, Serialization, ...

Use BufferedInputStream, and use the available() method which returns the size of bytes available for reading, and then construct a byte[] with that size. Problem solved. :)
BufferedInputStream buf = new BufferedInputStream(is);
int size = buf.available();

Here is a simpler example using ByteArrayOutputStream...
socketInputStream = socket.getInputStream();
int expectedDataLength = 128; //todo - set accordingly/experiment. Does not have to be precise value.
ByteArrayOutputStream baos = new ByteArrayOutputStream(expectedDataLength);
byte[] chunk = new byte[expectedDataLength];
int numBytesJustRead;
while((numBytesJustRead = socketInputStream.read(chunk)) != -1) {
baos.write(chunk, 0, numBytesJustRead);
}
return baos.toString("UTF-8");
However, if the server does not return a -1, you will need to detect the end of the data some other way - e.g., maybe the returned content always ends with a certain marker (e.g., ""), or you could possibly solve using socket.setSoTimeout(). (Mentioning this as it is seems to be a common problem.)

This is both a late answer and self-advertising, but anyone checking out this question may want to take a look here:
https://github.com/GregoryConrad/SmartSocket

This question is 7 years old, but i had a similiar problem, while making a NIO and OIO compatible system (Client and Server might be whatever they want, OIO or NIO).
This was quit the challenge, because of the blocking InputStreams.
I found a way, which makes it possible and i want to post it, to help people with similiar problems.
Reading a byte array of dynamic sice is done here with the DataInputStream, which kann be simply wrapped around the socketInputStream. Also, i do not want to introduce a specific communication protocoll (like first sending the size of bytes, that will be send), because i want to make this as vanilla as possible. First of, i have a simple utility Buffer class, which looks like this:
import java.util.ArrayList;
import java.util.List;
public class Buffer {
private byte[] core;
private int capacity;
public Buffer(int size){
this.capacity = size;
clear();
}
public List<Byte> list() {
final List<Byte> result = new ArrayList<>();
for(byte b : core) {
result.add(b);
}
return result;
}
public void reallocate(int capacity) {
this.capacity = capacity;
}
public void teardown() {
this.core = null;
}
public void clear() {
core = new byte[capacity];
}
public byte[] array() {
return core;
}
}
This class only exists, because of the dumb way, byte <=> Byte autoboxing in Java works with this List. This is not realy needed at all in this example, but i did not want to leave something out of this explanation.
Next up, the 2 simple, core methods. In those, a StringBuilder is used as a "callback". It will be filled with the result which has been read and the amount of bytes read will be returned. This might be done different of course.
private int readNext(StringBuilder stringBuilder, Buffer buffer) throws IOException {
// Attempt to read up to the buffers size
int read = in.read(buffer.array());
// If EOF is reached (-1 read)
// we disconnect, because the
// other end disconnected.
if(read == -1) {
disconnect();
return -1;
}
// Add the read byte[] as
// a String to the stringBuilder.
stringBuilder.append(new String(buffer.array()).trim());
buffer.clear();
return read;
}
private Optional<String> readBlocking() throws IOException {
final Buffer buffer = new Buffer(256);
final StringBuilder stringBuilder = new StringBuilder();
// This call blocks. Therefor
// if we continue past this point
// we WILL have some sort of
// result. This might be -1, which
// means, EOF (disconnect.)
if(readNext(stringBuilder, buffer) == -1) {
return Optional.empty();
}
while(in.available() > 0) {
buffer.reallocate(in.available());
if(readNext(stringBuilder, buffer) == -1) {
return Optional.empty();
}
}
buffer.teardown();
return Optional.of(stringBuilder.toString());
}
The first method readNext will fill the buffer, with byte[] from the DataInputStream and return the amount bytes read this way.
In the secon method, readBlocking, i utilized the blocking nature, not to worry about consumer-producer-problems. Simply readBlocking will block, untill a new byte-array is received. Before we call this blocking method, we allocate a Buffer-size. Note, i called reallocate after the first read (inside the while loop). This is not needed. You can safely delete this line and the code will still work. I did it, because of the uniqueness of my problem.
The 2 things, i did not explain in more detail are:
1. in (the DataInputStream and the only short varaible here, sorry for that)
2. disconnect (your disconnect routine)
All in all, you can now use it, this way:
// The in has to be an attribute, or an parameter to the readBlocking method
DataInputStream in = new DataInputStream(socket.getInputStream());
final Optional<String> rawDataOptional = readBlocking();
rawDataOptional.ifPresent(string -> threadPool.execute(() -> handle(string)));
This will provide you with a way of reading byte arrays of any shape or form over a socket (or any InputStream realy). Hope this helps!