Develop Reference

HTTP Webserver ignoring last line of POST request [duplicate]

Right now, I'm trying to write a GUI based Java tic-tac-toe game that functions over a network connection. It essentially works at this point, however I have an intermittent error in which several chars sent over the network connection are lost during gameplay. One case looked like this, when println statements were added to message sends/reads:
Player 1:
Just sent ROW 14 COLUMN 11 GAMEOVER true
Player 2:
Just received ROW 14 COLUMN 11 GAMEOV
Im pretty sure the error is happening when I read over the network. The read takes place in its own thread, with a BufferedReader wrapped around the socket's InputStream, and looks like this:
try {
int input;
while((input = dataIn.read()) != -1 ){
char msgChar = (char)input;
String message = msgChar + "";
while(dataIn.ready()){
msgChar = (char)dataIn.read();
message+= msgChar;
}
System.out.println("Just received " + message);
this.processMessage(message);
}
this.sock.close();
}
My sendMessage method is pretty simple, (just a write over a DataOutputStream wrapped around the socket's outputstream) so I don't think the problem is happening there:
try {
dataOut.writeBytes(message);
System.out.println("Just sent " + message);
}
Any thoughts would be highly appreciated. Thanks!

As it turns out, the ready() method guaruntees only that the next read WON'T block. Consequently, !ready() does not guaruntee that the next read WILL block. Just that it could.
I believe that the problem here had to do with the TCP stack itself. Being stream-oriented, when bytes were written to the socket, TCP makes no guarantees as to the order or grouping of the bytes it sends. I suspect that the TCP stack was breaking up the sent string in a way that made sense to it, and that in the process, the ready() method must detect some sort of underlying break in the stream, and return false, in spite of the fact that more information is available.
I refactored the code to add a newline character to every message send, then simply performed a readLine() instead. This allowed my network protocol to be dependent on the newline character as a message delimiter, rather than the ready() method. I'm happy to say this fixed the problem.
Thanks for all your input!

Try flushing the OutputStream on the sender side. The last bytes might remain in some intenal buffers.

It is really important what types of streamed objects you use to operate with data. It seems to me that this troubleshooting is created by the fact that you use DataOutputStream for sending info, but something else for receiving. Try to send and receive info by DataOutputStream and DataInputStream respectively.
Matter fact, if you send something by calling dataOut.writeBoolean(b)
but trying to receive this thing by calling dataIn.readString(), you will eventually get nothing. DataInputStream and DataOutputStream are type-sensitive. Try to refactor your code keeping it in mind.
Moreover, some input streams return on invocation of read() a single byte. Here you try to convert this one single byte into char, while in java char by default consists of two bytes.
msgChar = (char)dataIn.read();
Check whether it is a reason of data loss.

Debugging if UTF-8 decoding is done correctly?

We have a Java code talking to external system over TCP connections with xml messages encoded in UTF-8.
The message received begin with '?'. SO the XML received is
?<begin>message</begin>
There is a real doubt if the first character is indeed '?'. At the moment, we cannot ask the external system if/what.
The code snippet for reading the stream is as below.
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, Charset.forName("UTF-8")));
int readByte = reader.read();
if (readByte <= 0) {
inputStream.close();
}
builder.append((char) readByte);
We are currently trying to log the raw bytes int readByte = inputStream.read(). The logs will take few days to be received.
In the mean time, I was wondering how we could ascertain at our end if it was truly a '?' and not a decoding issue?

I suspect strongly you have a byte-order-mark at the beginning of your doc. That won't render as a valid character, and consequently could appear as a question mark. Can you dump the raw bytes out and check for that sequence ?

Your question seems to boil down to this:
Can we ascertain the real value of the first byte of the message without actually looking at it.
The answer is "No, you can't". (Obviously!)
...
However, if you could intercept the TCP/IP traffic from the external system with a packet sniffer (aka traffic monitoring tool), then dumping the first byte or bytes of the message would be simple ... requiring no code changes.
Is logging the int returned by inputStream.read() the correct way to to analyse the bytes received. Or does the word length of the OS or other environment variables come into picture.
The InputStream.read() method returns either a single (unsigned) byte of data (in the range 0 to 255 inclusive) or -1 to indicate "end of stream". It is not sensitive to the "word length" or anything else.
In short, provided you treat the results appropriately, calling read() should give you the data you need to see what the bytes in the stream really are.

Java nio partial read

My goal is to send different kind of messages from client to server, and it will be text based. The thing I am uncertain of is how to del with partial reads here. I will have to be sure that I get a whole message and nothing more.
Do anyone have experience with that?
Here is what I have so far:
private void handleNewClientMessage(SelectionKey key) throws IOException {
SocketChannel sendingChannel = (SocketChannel) key.channel();
ByteBuffer receivingBuffer = ByteBuffer.allocate(2048);
int bytesRead = sendingChannel.read(receivingBuffer);
if (bytesRead > 0) {
receivingBuffer.flip();
byte[] array = new byte[receivingBuffer.limit()];
receivingBuffer.get(array);
String message = new String(array);
System.out.println("Server received " +message);
}
selector.wakeup();
}
But I have no way of "ending" the message and be certain to have one full message.
Best regards,
O

You can never be sure you won't read more than one message unless you only read one byte at a time. (Which I don't suggest).
Instead I would read as much as you can into a ByteBuffer and then parse it to find the end of the message e.g. a newline for text.
When you find the end of a line extract it and convert it to a String and process it. repeat until you have a partial message (or nothing left)
If you find you have only part of a message, you compact() (if position() > 0) when you have and try to read() some more.
This will allows you to read as many messages at once as you can but can also handle incomplete messages.
Note: You will need to keep the ByteBuffer for a connection so you know what partial messages you have read before.
Note: this is will not work if you have a message which is larger than your buffer size. I suggest using a recycled direct ByteBuffer of say 1+ MB. With direct ByteBuffers only the pages of the ByteBuffer which are used get allocated to real memory.
If you are concerned about performance I would re-use your byte[] where possible. You only need to re-allocate it if you need more space than you have already.
BTW, You might find using a BufferedReader with Plain IO is much simpler to use, but still performance well enough.

Read binary data from a socket

I'm trying to connect to a server, and then send it a HTTP request (GET in this case). The idea is request a file and then receive it from the server.
It should work with both text files and binary files (imgs for example). I have no problem with text files, it works perfect, but I'm having some troubles with binary files.
First, I declare a BufferedReader (for reading header and textfile) and a DataInput Stream:
BufferedReader in_text = new BufferedReader(
new InputStreamReader(socket.getInputStream()));
DataInputStream in_binary = new DataInputStream(
new BufferedInputStream(socket.getInputStream()));
Then, I read the header with in_text and discover if it's a textfile or binary file. In case it's a textfile, I read it correctly in a StringBuilder. In case it's a binary file, I declare a byte[filesize] and store the following content of in_binary.
byte[] bindata = new byte[filesize];
in_binary.readFully(bindata);
And it doesn't work. I get a EOFException.
I thought that maybe in_binary is still in the first position of the stream, so it hasn't read the header yet. So I captured the length of the header and skip that bytes in in_binary.
byte[] bindata = new byte[filesize];
in_binary.reset();
in_binary.skip(headersize);
in_binary.readFully(bindata);
And still the same.
What could be happening?
Thanks!
PD: I know I could use URLConnection and all of that. That's not the problem.

BufferedReader buffers data (hence the name) - it will almost certainly have read more data from the socket than just the header. Therefore, when you try to read the actual data some has already been read from the socket. If you try reading just a few bytes you'll probably see that they aren't the first bytes of the actual response data.
If you know how to use URLConnection, I have to wonder what reason you have for not using it.

As soon as you use any subclass of Reader, you aren't reading binary. You are converting from bytes to characters, using the default encoding of the JVM. If you really want bytes of binary, you need to stick to streams, not readers. Creating both stacks at once is asking for trouble.
Use Apache Commons IO: IOUtils.toByteArray() to read the entire content into memory as a byte[], and then decide what to do with it, unless you have a gigantic amount of data, in which case you should set up the buffered input stream, decide what to do, and only construct the reader after you push back.

Java blocking socket returning incomplete ByteBuffer

I have a socketChannel configured as blocking, but when reading byte buffers of 5K from this socket, I get an incomplete buffer sometimes.
ByteBuffer messageBody = ByteBuffer.allocate(5*1024);
messageBody.mark();
messageBody.order(ByteOrder.BIG_ENDIAN);
int msgByteCount = channel.read(messageBody);
Ocasionally, messageBody is not completely filled and channel.read() does not return -1 or an exception, but the actual number of bytes read (which is less than 5k).
Has anyone experienced a similar problem?

That's how reads work. The SocketChannel documentation says:
A read operation might not fill the buffer, and in fact it might not read any bytes at all. [...] It is guaranteed, however, that if a channel is in blocking mode and there is at least one byte remaining in the buffer then this method will block until at least one byte is read [emphasis added].

When you use sockets you must anticipate that the socket might transfer fewer bytes than you expect. You must loop on the .read method to get the remainder of the bytes.
This is also true when you send bytes through a socket. You must check how many bytes were sent, and loop on the send until all bytes have been sent.
This behavior is due to the network layers splitting the messages into multiple packets. If your messages are short, then you are less likely to encounter this. But you should always code for it.
With 5k bytes per buffer you are very likely to see the sender's message spit into multiple packets. Each read operation will receive one packet, which is only part of your message.

TCP/IP sends the information in packets, they are not always all available when you do the read, therefore you must do the read in a loop.
char [] buffer = new char[1024];
int chars_read;
try
{
while((chars_read = from_server.read(buffer)) != -1)
{
to_user.write(buffer,0,chars_read);
to_user.flush();
}
}
catch(IOException e)
{
to_user.println(e);
}
See this post