Develop Reference

Second packet read from input stream has incomplete data,

I a simple server that receives bytes using TCP and then saves them to a file stream. Through many tests I have seen that the first packet received is always just the filename with no other data. The second packet received only has one byte and it is the first letter of the input text file. After this all packets are sent correctly, but I can't seem to figure out what is messing up the second packet. It also appears that the last packet is written twice. Can anyone see what I am doing wrong? Here is an example Input/Output: https://www.diffchecker.com/srclclrx
InputStream in = clntSock.getInputStream(); //server's input stream - gets data from the client
OutputStream out = clntSock.getOutputStream(); //server's output stream - server sends data to the client
byte[] byteBuffer = new byte[BUFSIZE];
int count = in.read(byteBuffer, 0, BUFSIZE);
String firstRead = new String(byteBuffer, 0, count);
int fileNameEnd = firstRead.indexOf("\r\n");
String fileName = firstRead.substring(0, fileNameEnd);
FileOutputStream fout = new FileOutputStream(fileName); //unzipped file output stream
int contentBegin = fileNameEnd+2;
byte[] oldBuffer = Arrays.copyOfRange(byteBuffer, contentBegin, count);
int oldCount = count-contentBegin;
String oldString = new String(byteBuffer, contentBegin, count-contentBegin, "US-ASCII");
while((count = in.read(byteBuffer, 0, BUFSIZE)) != -1) { // read from origin's buffer into byteBuffer until origin is out of data
String newString = new String(byteBuffer, 0, count, "US-ASCII");
String combinedString = oldString + newString;
int index = combinedString.indexOf("--------MagicStringCSE283Miami");
if(index != -1){
System.out.println("Final Print");
byte[] combinedBuffer = concat(oldBuffer, byteBuffer);
for(int i=0; i<index; i++){
System.out.print((char)combinedBuffer[i]);
}
System.out.println("");
fout.write(combinedBuffer, 0, index);
fout.flush();
fout.close();
break;
}
System.out.println(+ oldCount);
fout.write(oldBuffer, 0, oldCount); //write the byteBuffer's data to the client via the zip output stream
fout.flush(); //push all data out of the zipOutputStream before continuing
if(count == 1){
for(int i=0; i<count; i++){
System.out.println((char)byteBuffer[i]);
}
}
oldBuffer = byteBuffer;
oldCount = count;
oldString = newString;
}
Edit: Another peculiarity to me is that the second to last packet is always just "-" and then the last packet has the remainder of the magic string which terminates the file output stream.

Are you really sure that you are taking the full content of the data you receive ?
while((count = in.read(byteBuffer, 0, BUFSIZE)) != -1) { // read from origin's buffer into byteBuffer until origin is out of data
add logic here to print count
add logic here to print the content of the byteBuffer
}
It is very likely that in your logic you are mistreating what you receive and somehow loose part of the data.
For instance your second packet where you claim to receive only '-' is the count then just equal to 1 ? It is possible that indeed this is the case with TCP, but you really have to verify that you are indeed processing everything that you receive. Based on your explanation I think you are dropping data, well not processing it correctly really.

There are no messages in TCP and no guarantees about how much data each read() will return. It is only specified to transfer at least one byte in blocking mode, unless an error occurs.
One of the consequences of this is that it is impossible to write correct networking code without storing the result of read() in a variable, testing it, and then using it to delimit the amount of data processed.
Your expectations are at fault.

How to validate Integer received from a socket stream in java

In my application, clients sends file to the server in this format,
4 bytes of file length + actual file content
So the server reads the first 4 bytes to get the length of message and then it reads upto that message length.
The server code will be like this,
ByteArrayOutputStream lengthBuf = new ByteArrayOutputStream(4);
byte[] output = new byte[4];
//Reading first 4 bytes from InputStream
int readLength = inputStream.read(output);
lengthBuf.write(outpu, 0, readLength);
//Converting to integer
int length = ByteBuffer.wrap(lengthBuf.toByteArray()).getInt();
It works fine for valid cases. But in case if the client fails to append the length in the first 4 bytes, then this length got some junk value (1481988969). How to validate that the first 4 bytes of data is a valid integer ?

As long as you are fixed to 4 bytes and all values of the integer can be valid, there is no way.
However if you can more bytes. You can add a fifth byte and e.g. use CRC or other error-detecting codes to check if the value has correctly been transmitted.
Another option would be if you actually do not need all bits of the four transmitted bytes, that you use some bits of them.
Both actually needs access to the Server-Implementation of the Socket. If you don't have this access, it will not be possible to check if the integer is correct. You only can exclude file sizes which you know that they never occur. E.g.:
if (integer < MAX_FILE_SIZE and integer > 0) valid();
// or integer >= 0 if empty files are allowed.
else invalid();

I would do it this way.
DataInputStream dis = new DataInputStream(inputStream);
int len = dis.readInt(); // read exactly 4 bytes
byte[] bytes = new byte[len];
dis.readFully(bytes);
When you do you first read, you could be getting less than 4 bytes. When you read the actual data, you might have to read many times to get it all. read(byte[]) can return as soon as it get 1 or more bytes.

Why don;t you initalize it with at least size four then , when the client fails to append the size >
if(length==null) {
size = 4 ;
}
ByteArrayOutputStream lengthBuf = new ByteArrayOutputStream(size);
int length = ByteBuffer.wrap(lengthBuf.toByteArray()).getInt();

InputStream misses byte while reading in a loop

I'm trying to write a java program to read from a COM port. There are 266 bytes to read, and since the 266 bytes are not generated all together, which means that the input stream can be empty at sometime, I used a while loop to read all 266 bytes. The problem is that SOMETIMES one byte may be missed (only one byte), according to my checking the received bytes one by one. Here are the codes:
While(numOfBytes < 266) {
if(!(inputStream.available() > 0)) continue;
inputStream.read(buffer);
data[numOfBytes] = buffer[0];
numOfBytes++;
}

You give input stream an array to store data in (is.read(buffer)), but regardless of how much it reads, storing only 1 byte, and incrementing number of bytes by 1.
Try instead something like:
While(numOfBytes < 266) {
if(!(inputStream.available() > 0)) continue;
int b = inputStream.read();
if(b >= 0){
data[numOfBytes] = (byte) b ;
numOfBytes++;
}
}

normally I would do it like this
byte[] in = new byte[4196];
int bytesRead = 0;
while ((bytesRead = is.read(in)) != -1) {
// add to a StringBuffer maybe
}

difference between input.read and input.read(array, offset, length)

I'm trying to understand how inputstreams work. The following block of code is one of the many ways to read data from a text file:-
File file = new File("./src/test.txt");
InputStream input = new BufferedInputStream (new FileInputStream(file));
int data = 0;
while (data != -1) (-1 means we reached the end of the file)
{
data = input.read(); //if a character was read, it'll be turned to a bite and we get the integer representation of it so a is 97 b is 98
System.out.println(data + (char)data); //this will print the numbers followed by space then the character
}
input.close();
Now to use input.read(byte, offset, length) i have this code. I got it from here
File file = new File("./src/test.txt");
InputStream input = new BufferedInputStream (new FileInputStream(file));
int totalBytesRead = 0, bytesRemaining, bytesRead;
byte[] result = new byte[ ( int ) file.length()];
while ( totalBytesRead < result.length )
{
bytesRemaining = result.length - totalBytesRead;
bytesRead = input.read ( result, totalBytesRead, bytesRemaining );
if ( bytesRead > 0 )
totalBytesRead = totalBytesRead + bytesRead;
//printing integer version of bytes read
for (int i = 0; i < bytesRead; i++)
System.out.print(result[i] + " ");
System.out.println();
//printing character version of bytes read
for (int i = 0; i < bytesRead; i++)
System.out.print((char)result[i]);
}
input.close();
I'm assuming that based on the name BYTESREAD, this read method is returning the number of bytes read. In the documentation, it says that the function will try to read as many as possible. So there might be a reason why it wouldn't.
My first question is: What are these reasons?
I could replace that entire while loop with one line of code: input.read(result, 0, result.length)
I'm sure the creator of the article thought about this. It's not about the output because I get the same output in both cases. So there has to be a reason. At least one. What is it?

The documentation of read(byte[],int,int says that it:
Reads up to len bytes of data.
An attempt is made to read as many as len bytes
A smaller number may be read.
Since we are working with files that are right there in our hard disk, it seems reasonable to expect that the attempt will read the whole file, but input.read(result, 0, result.length) is not guaranteed to read the whole file (it's not said anywhere in the documentation). Relying in undocumented behaviors is a source for bugs when the undocumented behavior change.
For instance, the file stream may be implemented differently in other JVMs, some OS may impose a limit on the number of bytes that you may read at once, the file may be located in the network, or you may later use that piece of code with another implementation of stream, which doesn't behave in that way.
Alternatively, if you are reading the whole file in an array, perhaps you could use DataInputStream.readFully
About the loop with read(), it reads a single byte each time. That reduces performance if you are reading a big chunk of data, since each call to read() will perform several tests (has the stream ended? etc) and may ask the OS for one byte. Since you already know that you want file.length() bytes, there is no reason for not using the other more efficient forms.

Imagine you are reading from a network socket, not from a file. In this case you don't have any information about the total amount of bytes in the stream. You would allocate a buffer of fixed size and read from the stream in a loop. During one iteration of the loop you can't expect there are BUFFERSIZE bytes available in the stream. So you would fill the buffer as much as possible and iterate again, until the buffer is full. This can be useful, if you have data blocks of fixed size, for example serialized object.
ArrayList<MyObject> list = new ArrayList<MyObject>();
try {
InputStream input = socket.getInputStream();
byte[] buffer = new byte[1024];
int bytesRead;
int off = 0;
int len = 1024;
while(true) {
bytesRead = input.read(buffer, off, len);
if(bytesRead == len) {
list.add(createMyObject(buffer));
// reset variables
off = 0;
len = 1024;
continue;
}
if(bytesRead == -1) break;
// buffer is not full, adjust size
off += bytesRead;
len -= bytesRead;
}
} catch(IOException io) {
// stream was closed
}
ps. Code is not tested and should only point out, how this function can be useful.

You specify the amount of bytes to read because you might not want to read the entire file at once or maybe you couldn't or might not want to create a buffer as large as the file.

SocketChannel.write(ByteBuffer[]) "corrupting" data

Problem: Corrupt TCP segment.
I send a sequence of ByteBuffers over and over in a SocketChannel. The sequence is the following:
\r\n
length of chunk (example: fff)
\r\n
chunk data (rubbish, a 1000 - 5000 character long string)
\r\n
length of next chunk (example: fff)
\r\n
next chunk data (rubbish, a 1000 - 5000 character long string)
...
I hope you see the pattern. The MTU on network level is about 1500, so it'll create TCP segments to send over the "chunk data".
The problem in the segments is: Somehow(?), randomly(?), a segment (its payload) starts with \r\n instead of the remaining bytes from the "chunk data" first.
So you get for example:
(segment 1)
\r\n
length of chunk (example: fff)
\r\n
chunk data (456 bytes)
(segment 2)
\r\n
chunk data (remaining 156 bytes)
length of next
\r\n
Instead of:
(segment 1)
\r\n
length of chunk (example: fff)
\r\n
chunk data (456 bytes)
(segment 2)
chunk data (remaining 156 bytes)
\r\n
length of next
\r\n
I'd like to know if Java code is even able to cause that, knowing that my "chunk data" ByteBuffer sent correctly, except for the ByteBuffer containing \r\n that joins in...
Any help is welcome, thank you for your time!
Andrew

I will bet that you are ignoring the result of a read or write. TCP does not lose or corrupt data and neither do the Socket APIs or the Java networking libraries. At least I've never seen it in about 22 years of network programming and 14 years of Java.

It is not because of network issue but with the way we coded. If we are reading and writing the data in chunks it may lead to data corruption because of last chunk. It may be possible the last data chunk read is partially filled and having default values as 0 for byte array.
Following example shows the solution for it
ObjectOutputStream out = new ObjectOutputStream(
socket.getOutputStream());
ObjectInputStream in = new ObjectInputStream(socket.getInputStream());
ObjectOutputStream out = new ObjectOutputStream(socket.getOutputStream());
//Something local returns DataInputStream from server
InputStream dataInputStream = local.downloadFile(fileToBeRead);
int chunkSize = 10000;
byte[] chunkByteArray = new byte[chunkSize];
int bytesRead = -1;
while ((bytesRead = dataInputStream.read(chunkByteArray)) != -1) {
if (bytesRead < chunkSize) {
System.out.println("Last Chunk is " + bytesRead);
chunkByteArray = getLastChunkByteArray(bytesRead,chunkByteArray);
}
out.write(chunkByteArray);
}
dataInputStream.close();
And the method
private byte[] getLastChunkByteArray(int noOfBytesRead,
byte[] partialFilledChunk) {
byte[] lastChunk = new byte[noOfBytesRead];
for (int i = 0; i < noOfBytesRead; i++) {
lastChunk[i] = partialFilledChunk[i];
}
return lastChunk;
}