My goal is to send different kind of messages from client to server, and it will be text based. The thing I am uncertain of is how to del with partial reads here. I will have to be sure that I get a whole message and nothing more.
Do anyone have experience with that?
Here is what I have so far:
private void handleNewClientMessage(SelectionKey key) throws IOException {
SocketChannel sendingChannel = (SocketChannel) key.channel();
ByteBuffer receivingBuffer = ByteBuffer.allocate(2048);
int bytesRead = sendingChannel.read(receivingBuffer);
if (bytesRead > 0) {
receivingBuffer.flip();
byte[] array = new byte[receivingBuffer.limit()];
receivingBuffer.get(array);
String message = new String(array);
System.out.println("Server received " +message);
}
selector.wakeup();
}
But I have no way of "ending" the message and be certain to have one full message.
Best regards,
O
You can never be sure you won't read more than one message unless you only read one byte at a time. (Which I don't suggest).
Instead I would read as much as you can into a ByteBuffer and then parse it to find the end of the message e.g. a newline for text.
When you find the end of a line extract it and convert it to a String and process it. repeat until you have a partial message (or nothing left)
If you find you have only part of a message, you compact() (if position() > 0) when you have and try to read() some more.
This will allows you to read as many messages at once as you can but can also handle incomplete messages.
Note: You will need to keep the ByteBuffer for a connection so you know what partial messages you have read before.
Note: this is will not work if you have a message which is larger than your buffer size. I suggest using a recycled direct ByteBuffer of say 1+ MB. With direct ByteBuffers only the pages of the ByteBuffer which are used get allocated to real memory.
If you are concerned about performance I would re-use your byte[] where possible. You only need to re-allocate it if you need more space than you have already.
BTW, You might find using a BufferedReader with Plain IO is much simpler to use, but still performance well enough.
Related
Right now, I'm trying to write a GUI based Java tic-tac-toe game that functions over a network connection. It essentially works at this point, however I have an intermittent error in which several chars sent over the network connection are lost during gameplay. One case looked like this, when println statements were added to message sends/reads:
Player 1:
Just sent ROW 14 COLUMN 11 GAMEOVER true
Player 2:
Just received ROW 14 COLUMN 11 GAMEOV
Im pretty sure the error is happening when I read over the network. The read takes place in its own thread, with a BufferedReader wrapped around the socket's InputStream, and looks like this:
try {
int input;
while((input = dataIn.read()) != -1 ){
char msgChar = (char)input;
String message = msgChar + "";
while(dataIn.ready()){
msgChar = (char)dataIn.read();
message+= msgChar;
}
System.out.println("Just received " + message);
this.processMessage(message);
}
this.sock.close();
}
My sendMessage method is pretty simple, (just a write over a DataOutputStream wrapped around the socket's outputstream) so I don't think the problem is happening there:
try {
dataOut.writeBytes(message);
System.out.println("Just sent " + message);
}
Any thoughts would be highly appreciated. Thanks!
As it turns out, the ready() method guaruntees only that the next read WON'T block. Consequently, !ready() does not guaruntee that the next read WILL block. Just that it could.
I believe that the problem here had to do with the TCP stack itself. Being stream-oriented, when bytes were written to the socket, TCP makes no guarantees as to the order or grouping of the bytes it sends. I suspect that the TCP stack was breaking up the sent string in a way that made sense to it, and that in the process, the ready() method must detect some sort of underlying break in the stream, and return false, in spite of the fact that more information is available.
I refactored the code to add a newline character to every message send, then simply performed a readLine() instead. This allowed my network protocol to be dependent on the newline character as a message delimiter, rather than the ready() method. I'm happy to say this fixed the problem.
Thanks for all your input!
Try flushing the OutputStream on the sender side. The last bytes might remain in some intenal buffers.
It is really important what types of streamed objects you use to operate with data. It seems to me that this troubleshooting is created by the fact that you use DataOutputStream for sending info, but something else for receiving. Try to send and receive info by DataOutputStream and DataInputStream respectively.
Matter fact, if you send something by calling dataOut.writeBoolean(b)
but trying to receive this thing by calling dataIn.readString(), you will eventually get nothing. DataInputStream and DataOutputStream are type-sensitive. Try to refactor your code keeping it in mind.
Moreover, some input streams return on invocation of read() a single byte. Here you try to convert this one single byte into char, while in java char by default consists of two bytes.
msgChar = (char)dataIn.read();
Check whether it is a reason of data loss.
We have a Java code talking to external system over TCP connections with xml messages encoded in UTF-8.
The message received begin with '?'. SO the XML received is
?<begin>message</begin>
There is a real doubt if the first character is indeed '?'. At the moment, we cannot ask the external system if/what.
The code snippet for reading the stream is as below.
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, Charset.forName("UTF-8")));
int readByte = reader.read();
if (readByte <= 0) {
inputStream.close();
}
builder.append((char) readByte);
We are currently trying to log the raw bytes int readByte = inputStream.read(). The logs will take few days to be received.
In the mean time, I was wondering how we could ascertain at our end if it was truly a '?' and not a decoding issue?
I suspect strongly you have a byte-order-mark at the beginning of your doc. That won't render as a valid character, and consequently could appear as a question mark. Can you dump the raw bytes out and check for that sequence ?
Your question seems to boil down to this:
Can we ascertain the real value of the first byte of the message without actually looking at it.
The answer is "No, you can't". (Obviously!)
...
However, if you could intercept the TCP/IP traffic from the external system with a packet sniffer (aka traffic monitoring tool), then dumping the first byte or bytes of the message would be simple ... requiring no code changes.
Is logging the int returned by inputStream.read() the correct way to to analyse the bytes received. Or does the word length of the OS or other environment variables come into picture.
The InputStream.read() method returns either a single (unsigned) byte of data (in the range 0 to 255 inclusive) or -1 to indicate "end of stream". It is not sensitive to the "word length" or anything else.
In short, provided you treat the results appropriately, calling read() should give you the data you need to see what the bytes in the stream really are.
I'm making a Server that gets packages at 64 kb size.
int length = 65536;
byte[] bytes = new byte[length];
int pos = 0;
while(pos < length -1)
{
System.out.println("Before read");
pos += dis.read(bytes, pos, length-pos);
System.out.println(""+pos+" >> "+ length);
}
This is the code I use to read all bytes from the socket. Dis is a InputStream.
When I run the code 1 out of n goes wrong. The code only receives 52964 bytes instead of 65536 bytes.
I also checked the C code and it says it send 65536 bytes.
Does someone know what I'm doing wrong?
This is yet another case where Jakarta Commons IOUtils is a better choice than writing it yourself. It's one line of code, and it's fully tested. I recommend IOUtils.readFully() in this case.
If it does not read the entire buffer, then you know that you're not sending all the content. Perhaps you're missing a flush on the server side.
InputStream.read() returns the number of bytes read or -1 if the end of the stream has been reached. You need to check for that error condition. Also, I suspect your while(..) loop is the problem. Why are you calling it pos as in position? You may be terminating prematurely. Also, ensure that your C code, whatever it is doing, is sending properly. You can examine the network traffic with a tool like Wireshark to be sure.
What do you mean it "goes wrong"? What is the output? It can't be exiting the loop before reading the full 64 KB, so what really happens?
Also, it's better to save the return value of the I/O call separately and inspect it, before assuming the I/O was successful. If that's DataInputStream.read(), it returns -1 on error.
Your code is incorrect as it doesn't check for -1.
This is a case for using DataInputStream.readFully() rather than coding it yourself and getting it wrong.
I have some code that tries to read in a Google Protocol Buffer message from a socket in Java. However, the mergeDelimitedFrom() method can throw an IOException if it reads in invalid data or if the socket connection is reset (and probably other reasons). If the connection is reset I would like to exit out of the loop, but if it is just a invalid message I would like to continue running. One thought is to just have some sort of exception counter and exit after X consecutive failures, but I was hoping to be able to figure out what type of error occurs instead of being in the dark.
This is basically the code I have:
while (m_Running)
{
SomeMessage message = null;
try
{
final Builder builder = SomeMessage.newBuilder();
if (builder.mergeDelimitedFrom(m_InputStream))
{
message = builder.build();
}
else
{
// Google protocol buffers doesn't document it very well
// but if mergeDelimietedFrom returns false then it has
// reached the end of the input stream. For a socket, no
// more data will be coming so exit from the thread
m_Running = false;
}
}
catch (final IOException e)
{
// what should really be done here ???
}
}
Just don't do it. If you are reading protocol buffer objects directly off a socket, then you are effectively defining your own application protocol. It's harder than you might think to do it right - there is a good general description of some of the problems at On the Design of Application Protocols. It's important to understand framing - determining where one message ends and another begins.
Which leads us to some advice from the inventors of protobuf at https://developers.google.com/protocol-buffers/docs/techniques. The key piece of advice is this:
If you want to write multiple messages to a single file or stream, it is up to you to keep track of where one message ends and the next begins.
I recommend that you decide on a framing protocol to divide the stream into messages, then write some custom socket code to handle the work or reading bytes of the sockets, dividing them into byte arrays where each byte array is known to contain exactly one message, then finally use protobuf to deserialize each message-byte-array into an object. Guaranteed no IOException protobuf deserialization.
You'll still have to deal with IOExceptions but it will be at a lower level where you are just reading byte arrays and you'll know exactly how much data has been deserialized when the error occurs.
Also consider using something like netty to help with the socket code.
I have a socketChannel configured as blocking, but when reading byte buffers of 5K from this socket, I get an incomplete buffer sometimes.
ByteBuffer messageBody = ByteBuffer.allocate(5*1024);
messageBody.mark();
messageBody.order(ByteOrder.BIG_ENDIAN);
int msgByteCount = channel.read(messageBody);
Ocasionally, messageBody is not completely filled and channel.read() does not return -1 or an exception, but the actual number of bytes read (which is less than 5k).
Has anyone experienced a similar problem?
That's how reads work. The SocketChannel documentation says:
A read operation might not fill the buffer, and in fact it might not read any bytes at all. [...] It is guaranteed, however, that if a channel is in blocking mode and there is at least one byte remaining in the buffer then this method will block until at least one byte is read [emphasis added].
When you use sockets you must anticipate that the socket might transfer fewer bytes than you expect. You must loop on the .read method to get the remainder of the bytes.
This is also true when you send bytes through a socket. You must check how many bytes were sent, and loop on the send until all bytes have been sent.
This behavior is due to the network layers splitting the messages into multiple packets. If your messages are short, then you are less likely to encounter this. But you should always code for it.
With 5k bytes per buffer you are very likely to see the sender's message spit into multiple packets. Each read operation will receive one packet, which is only part of your message.
TCP/IP sends the information in packets, they are not always all available when you do the read, therefore you must do the read in a loop.
char [] buffer = new char[1024];
int chars_read;
try
{
while((chars_read = from_server.read(buffer)) != -1)
{
to_user.write(buffer,0,chars_read);
to_user.flush();
}
}
catch(IOException e)
{
to_user.println(e);
}
See this post