I have a socketChannel configured as blocking, but when reading byte buffers of 5K from this socket, I get an incomplete buffer sometimes.
ByteBuffer messageBody = ByteBuffer.allocate(5*1024);
messageBody.mark();
messageBody.order(ByteOrder.BIG_ENDIAN);
int msgByteCount = channel.read(messageBody);
Ocasionally, messageBody is not completely filled and channel.read() does not return -1 or an exception, but the actual number of bytes read (which is less than 5k).
Has anyone experienced a similar problem?
That's how reads work. The SocketChannel documentation says:
A read operation might not fill the buffer, and in fact it might not read any bytes at all. [...] It is guaranteed, however, that if a channel is in blocking mode and there is at least one byte remaining in the buffer then this method will block until at least one byte is read [emphasis added].
When you use sockets you must anticipate that the socket might transfer fewer bytes than you expect. You must loop on the .read method to get the remainder of the bytes.
This is also true when you send bytes through a socket. You must check how many bytes were sent, and loop on the send until all bytes have been sent.
This behavior is due to the network layers splitting the messages into multiple packets. If your messages are short, then you are less likely to encounter this. But you should always code for it.
With 5k bytes per buffer you are very likely to see the sender's message spit into multiple packets. Each read operation will receive one packet, which is only part of your message.
TCP/IP sends the information in packets, they are not always all available when you do the read, therefore you must do the read in a loop.
char [] buffer = new char[1024];
int chars_read;
try
{
while((chars_read = from_server.read(buffer)) != -1)
{
to_user.write(buffer,0,chars_read);
to_user.flush();
}
}
catch(IOException e)
{
to_user.println(e);
}
See this post
Related
Right now, I'm trying to write a GUI based Java tic-tac-toe game that functions over a network connection. It essentially works at this point, however I have an intermittent error in which several chars sent over the network connection are lost during gameplay. One case looked like this, when println statements were added to message sends/reads:
Player 1:
Just sent ROW 14 COLUMN 11 GAMEOVER true
Player 2:
Just received ROW 14 COLUMN 11 GAMEOV
Im pretty sure the error is happening when I read over the network. The read takes place in its own thread, with a BufferedReader wrapped around the socket's InputStream, and looks like this:
try {
int input;
while((input = dataIn.read()) != -1 ){
char msgChar = (char)input;
String message = msgChar + "";
while(dataIn.ready()){
msgChar = (char)dataIn.read();
message+= msgChar;
}
System.out.println("Just received " + message);
this.processMessage(message);
}
this.sock.close();
}
My sendMessage method is pretty simple, (just a write over a DataOutputStream wrapped around the socket's outputstream) so I don't think the problem is happening there:
try {
dataOut.writeBytes(message);
System.out.println("Just sent " + message);
}
Any thoughts would be highly appreciated. Thanks!
As it turns out, the ready() method guaruntees only that the next read WON'T block. Consequently, !ready() does not guaruntee that the next read WILL block. Just that it could.
I believe that the problem here had to do with the TCP stack itself. Being stream-oriented, when bytes were written to the socket, TCP makes no guarantees as to the order or grouping of the bytes it sends. I suspect that the TCP stack was breaking up the sent string in a way that made sense to it, and that in the process, the ready() method must detect some sort of underlying break in the stream, and return false, in spite of the fact that more information is available.
I refactored the code to add a newline character to every message send, then simply performed a readLine() instead. This allowed my network protocol to be dependent on the newline character as a message delimiter, rather than the ready() method. I'm happy to say this fixed the problem.
Thanks for all your input!
Try flushing the OutputStream on the sender side. The last bytes might remain in some intenal buffers.
It is really important what types of streamed objects you use to operate with data. It seems to me that this troubleshooting is created by the fact that you use DataOutputStream for sending info, but something else for receiving. Try to send and receive info by DataOutputStream and DataInputStream respectively.
Matter fact, if you send something by calling dataOut.writeBoolean(b)
but trying to receive this thing by calling dataIn.readString(), you will eventually get nothing. DataInputStream and DataOutputStream are type-sensitive. Try to refactor your code keeping it in mind.
Moreover, some input streams return on invocation of read() a single byte. Here you try to convert this one single byte into char, while in java char by default consists of two bytes.
msgChar = (char)dataIn.read();
Check whether it is a reason of data loss.
I have a client-server application communicating over Java NIO sockets which makes use of the SocketChannelImpl class underneath. When from the senders side, I send a ByteBuffer array of length n elements, the client always receives ByteBuffer array of length n+1 elements with the last ByteBuffer always being of the size 0.
I was wondering if that is some sort of an EOF indicator automatically sent by the SocketChannelImpl class to indicate a completed send of an array of ByteBuffers so that when the receiving side receives an array ByteBuffers followed by a BytBuffer of size 0 it knows that it has correctly received the ByteBuffer array that was sent from the senders side?
Update: Adding example to further elaborate on my question:
Whenever a call to SocketChannel.write(ByteBuffer[] srcs, int offset, int length) is made on the senders side and SocketChannel.read(ByteBuffer srcs) is made on the receivers side, I log the length of the array and the size of its element. Following is a set of logs from senders and receiver side:
Sender:
Writing using:write(ByteBuffer[] srcs, int offset, int length)
srcs.length == 2, totalBytesWritten = 1326
Receiver:
Read using read(ByteBuffer src)
totalBytesRead:4
totalBytesRead:1322
totalBytesRead:0
What I am asking is, why on the receiver side even though the amount of data that was sent by the sender(1326 bytes) is received by the client(4+1322), there is an extra call to the read(ByteBuffer src) is made which ends of reading 0 bytes. My (ill informed) guess was that was some kind of an EOF indicator, but from the comments on the question it looks like that has to do something with how the application itself goes about reading data from the channels.
SocketChannelImpl.write(ByteBuffer[] srcs, int offset, int length) appends ByteBuffer of size 0
No it doesn't.
I have a client-server application communicating over Java NIO sockets which makes use of the SocketChannelImpl class underneath.
Forget about what's underneath. You are using java.nio.channels.SocketChannel.
When from the senders side, I send a ByteBuffer array of length n elements
You don't. You send the contents. The sending ByteBuffer[] array is gathered into a byte stream and sent as bytes.
the client always receives ByteBuffer array of length n+1 elements with the last ByteBuffer always being of the size 0.
No it doesn't. The received bytes are scattered into another ByteBuffer[] array, if that's what you use to read the channel. If you use gather-write and scatter-read there is no relationship between the sending and receiving ByteBuffer arrays other than the actual bytes received.
I was wondering if that is some sort of an EOF indicator automatically sent by the SocketChannelImpl class to indicate a completed send of an array of ByteBuffers
No.
so that when the receiving side receives an array ByteBuffers
It doesn't. It receives a byte stream, into an array of ByteBuffers that it provided itself.
followed by a ByteBuffer of size 0 it knows that it has correctly received the ByteBuffer array that was sent from the senders side?
This doesn't begin to make sense. ByteBuffers are neither send nor received, either individually or as arrays.
Update: Adding example to further elaborate on my question:
There is no example here, only some output. An example would be useful but it would have to consist of Java code.
Whenever a call to SocketChannel.write(ByteBuffer[] srcs, int offset, int length) is made on the senders side and SocketChannel.read(ByteBuffer srcs)
Can we assume this should be SocketChannel.read(ByteBuffer[] srcs)? It doesn't make sense otherwise.
is made on the receivers side, I log the length of the array and the size of its element. Following is a set of logs from senders and receiver side:
Sender:
Writing using:write(ByteBuffer[] srcs, int offset, int length)
srcs.length == 2, totalBytesWritten = 1326
Receiver:
Read using read(ByteBuffer src)
totalBytesRead:4
totalBytesRead:1322
totalBytesRead:0
This can only mean two things:
You used read(ByteBuffer src) and called it three times; the peer had only sent 1326 bytes; so the third read returned zero, meaning no data was available to be read.
OR
You used read(ByteBuffer[] srcs), where srcs contained three ByteBuffers: one of length exactly 4, one of length >= 1322, and one of length > 0; only 1326 bytes were received, so the third ByteBuffer was cleared, or unaffected.
What I am asking is, why on the receiver side even though the amount of data that was sent by the sender(1326 bytes) is received by the client(4+1322), there is an extra call to the read(ByteBuffer src) is made which ends of reading 0 bytes.
There is no such call, unless you made it, and if you made it when there was no data available to be read you got zero bytes. Why is this surprising?
My (ill informed) guess was that was some kind of an EOF indicator
No. End of stream is signalled by read() returning -1.
but from the comments on the question it looks like that has to do something with how the application itself goes about reading data from the channels.
Correct. You tried to read beyond the end of the data that was sent, or with more ByteBuffers than were required to hold the data that was received. Simple as that.
Socket in OS (operating system) is a buffer with some limited capacity so if you write for example 10 Kb of the data that is not OS write this 10 Kb to the socket as the one chunk. OS for example write 10 chunk's by 1 Kb or another value, this is depends on the socket configuration in OS.
So we may read different number of the bytes at each read() method call. And the main thing in this that read() call depends on the type of the socket read method: Synchronous or Asynchronous. Difficult reading in asynchronous mode - if socket is empty at this time this is not means that we read the all data it's just means that in this time there are no data in channel but in the next read we can read for example 10 bytes and next 98 bytes and next 1098.
So in asynchronous mode we must send some bytes of the header metadata at the begining with for example data size we vill be transferring and the client will be know how many read() invocation it must call before data will be ended.
I'm working on the server for a game right now. The server's packet reading loop is blocking, and typically waits until a packet is received to continue through the loop. However, if the client disconnects, the DataInputStream returns a single byte (-1) and the loop is executed in rapid succession, as is expected. However, I don't use the DataInputStream's read() method to read one byte at a time, I use the read(byte[]) method to read them all at once into a byte array. As such, I can't easily detect if the stream is returning a single byte valued at -1.
Possible Solution: I could check if the first byte of the array is -1, and if so loop through the array to see if the rest of the array is nothing but zeroes. Doing this seems extremely inefficient however, and I feel that it would affect performance as client count increases.
Here's a simplified version of my packet-reading loop:
while (!thread.isInterrupted() && !isDisconnected())
{
try
{
byte[] data = new byte[26];
data = new byte[26];
input.read(data);
//Need to check if end of stream here somehow
Packet rawPacket = Packet.extractPacketFromData(data); //Constructs packet from the received data
if(rawPacket instanceof SomePacket)
{
//Do stuff with packet
}
}
catch(IOException e)
{
disconnectClient(); //Toggles flag showing client has disconnected
}
}
Your understanding of read(byte[]) is incorrect. It doesn't set a value in your array to -1.
The Javadoc says:
Returns:
the total number of bytes read into the buffer, or -1 if there is no more data because the end of the stream has been reached.
You need to check the return value:
int bytesRead = input.read(data);
if (bytesRead == -1)
{
// it's end of stream
}
As a side note, even when just reading data normally, you do need to check the number of bytes read was the number of bytes you requested. The call to read is not guaranteed to actually fill your array.
You should take a look at readFully() which does read fully, and throws an EOFException for end of stream:
Reads some bytes from an input stream and stores them into the buffer array b. The number of bytes read is equal to the length of b.
This method blocks until one of the following conditions occurs:
b.length bytes of input data are available, in which case a normal return is made.
End of file is detected, in which case an EOFException is thrown.
An I/O error occurs, in which case an IOException other than EOFException is thrown.
I have a small TCP server program and a corresponding client, and they communicate via ServerSocket and Socket classes and DataInputStream/DataOutputStream. And I have a problem with sending Strings to the server.
connection = new Socket("localhost", 2233);
outStream = new DataOutputStream(connection.getOutputStream());
outStream.writeBytes(fileName);
fileName is, at this point in time, a hard-coded String with the value "listener.jardesc". The server reads the string with the following code:
inStream = new DataInputStream(connection.getInputStream());
String fileName = inStream.readLine();
The string is received properly, but three zero-value bytes have been added to the end. Why is that and how can I stop it from happening? (I could, of course, trim the received string or somehow else stop this problem from mattering, but I'd rather prevent the problem completely)
I'm just going to throw this out there. You're using the readLine() method which has been deprecated in Java 5, 6 & 7. The API docs state quite clearly that this method "does not properly convert bytes to characters". I would read it as bytes or use a Buffered Reader.
http://docs.oracle.com/javase/1.5.0/docs/api/java/io/DataInputStream.html#readLine%28%29
writeBytes() does not add extra bytes.
The code you've written is invalid, as you aren't writing a newline. Therefore it doesn't work, and blocks forever in readLine().
In trying to debug this you appear to have read the bytes some other way, probably with read(); and to have ignored the return value returned by read, and to have concluded that read() filled the buffer you provided, when it didn't, leaving three bytes in their initial state, which is zero.
My goal is to send different kind of messages from client to server, and it will be text based. The thing I am uncertain of is how to del with partial reads here. I will have to be sure that I get a whole message and nothing more.
Do anyone have experience with that?
Here is what I have so far:
private void handleNewClientMessage(SelectionKey key) throws IOException {
SocketChannel sendingChannel = (SocketChannel) key.channel();
ByteBuffer receivingBuffer = ByteBuffer.allocate(2048);
int bytesRead = sendingChannel.read(receivingBuffer);
if (bytesRead > 0) {
receivingBuffer.flip();
byte[] array = new byte[receivingBuffer.limit()];
receivingBuffer.get(array);
String message = new String(array);
System.out.println("Server received " +message);
}
selector.wakeup();
}
But I have no way of "ending" the message and be certain to have one full message.
Best regards,
O
You can never be sure you won't read more than one message unless you only read one byte at a time. (Which I don't suggest).
Instead I would read as much as you can into a ByteBuffer and then parse it to find the end of the message e.g. a newline for text.
When you find the end of a line extract it and convert it to a String and process it. repeat until you have a partial message (or nothing left)
If you find you have only part of a message, you compact() (if position() > 0) when you have and try to read() some more.
This will allows you to read as many messages at once as you can but can also handle incomplete messages.
Note: You will need to keep the ByteBuffer for a connection so you know what partial messages you have read before.
Note: this is will not work if you have a message which is larger than your buffer size. I suggest using a recycled direct ByteBuffer of say 1+ MB. With direct ByteBuffers only the pages of the ByteBuffer which are used get allocated to real memory.
If you are concerned about performance I would re-use your byte[] where possible. You only need to re-allocate it if you need more space than you have already.
BTW, You might find using a BufferedReader with Plain IO is much simpler to use, but still performance well enough.