I'm trying to implement a communication system with an automatic repeat request strategy. I use three classes: Transmitter, Channel, Receiver. I have a maximum number of byte for message (window). But when I receive the byte sent, sometimes I receive less bytes than window. Why?
My code is this:
Transmitter
int n = 0;
int remaining, length;
while (n<channelBytes.length) {
remaining = channelBytes.length-n;
length = (remaining<window)? remaining : window;
outputStream.write(channelBytes,n,length);
// wait for the ack
byte[] b = new byte[4];
channel.socket().setSoTimeout(2000);
inputStream.read(b);
n += ByteBuffer.wrap(b).getInt();
}
Channel
bytes = new byte[SystemModel.WINDOW];
while(true) {
// receive from Tx
upInputStream.read(bytes);
// insert channel error
insertError(bytes);
Thread.sleep(propagationDelay + transmissionDelay);
// send bytes to Rx
downOutputStream.write(bytes);
// wait for the ack from Rx
clientChannelDown.socket().setSoTimeout(2000);
byte[] ack = new byte[4];
downInputStream.read(ack);
// send ack to Tx
upOutputStream.write(ByteBuffer.allocate(4).put(ack).array());
}
Receiver
byte[] b = new byte[SystemModel.WINDOW];
while (true) {
try {
int received = inputStream.read(b);
channelCoding.decodePartially(b);
}catch (SocketTimeoutException te){
break;
} catch (IOException e) {
e.printStackTrace();
} catch (DataFormatException e) {
// send ack
int ack = Integer.parseInt(e.getMessage());
try {
outputStream.write(ByteBuffer.allocate(4).putInt(ack).array());
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
In the Receiver, the array of bytes "b" is not always the length of the window.
Invalid code. See the Javadoc. InputStream.read(byte[] [,...]) isn't obliged to transfer more than one byte, and it is never valid to call it without storing the result into a variable. If you're expecting more than one byte, you have to loop, or use DataInputStream.readFully().
The canonical way to copy streams in Java is as follows:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
For ByteBuffers with Channels it is as follows:
while (in.read(buffer) > 0 || buffer.position() > 0)
{
buffer.flip();
out.write(buffer);
buffer.compact();
}
If you code correctly there is no need to insert sleeps into network code.
E&OE
I currently try to send a big Message over a Socket-Connection. My Message has something about 1.3 MB. Here is my code:
public void send(String message) throws IOException {
byte[] bytes = message.getBytes(encoding);
ByteBuffer byteBuffer = ByteBuffer.wrap(bytes);
while (socketChannel.write(byteBuffer) > 0) {
}
}
On the other side of the Client is a Sever, which just prints out the read data. It always stops reading after something about 260 kB. It seems to me that some Buffer is full?
What can I do to make this Socket-Connection working?
while (socketChannel.write(byteBuffer) > 0) {
}
If you aren't in blocking mode, this loop will terminate as soon as the local socket send buffer fills up, which causes write() to return zero.
A simple but poor quality fix would be to change the loop condition from > to >=. However if write() returns zero you should really be using a Selector and OP_WRITE to detect when the channel becomes writable again.
I think it would be better to allocate enough space for your ByteBuffer and flip it before you send it, because you want to read from it from now on. Maybe this works for you.
final byte[] bytesToSend = message.getBytes(encoding);
ByteBuffer buffer = ByteBuffer.allocate(bytesToSend.length);
buffer.clear();
buffer.put(bytesToSend);
buffer.flip();
while (buffer.hasRemaining()) {
try {
socketChannel.write(buf);
} catch (IOException e) {
e.printStackTrace();
}
}
I have searched and searched and everything I have found has been helpful but I keep getting an out of memory error. The images I send are .06 MB so I know the problem isn't from decoding the byte[] into a bitmap. When I remove the while loops this works like a charm for one frame but I want multiple frames. I am getting a byte[] and sending it to a different device using sockets but I am at a loss how to do this. My problem is that I don't send and receive the correct byte[] length. This is what i am doing currently.
while (count != -1) {
//first send the byte[] length
dataOutputStream.writeInt(sendPackage.length);
//pass a byte array
publishProgress("sending file to client");
showMyToastOnUiThread(String.valueOf(sendPackage.length));
outputStream.write(sendPackage, 0, sendPackage.length);
outputStream.flush();
}
Receive byte[] on different device:
int count = inputStream.read();
while (count != -1) {
int byteArrayLength = dataInputStream.readInt();
Log.i(MainActivity.TAG, "Starting convert to byte array");
byte[] receivedBytes = convertInputStreamToByteArray(inputStream, byteArrayLength);
Bitmap bitmap = BitmapFactory.decodeByteArray(receivedBytes, 0, receivedBytes.length);
publishProgress(bitmap);
}
//convert inputstream to byte[]
public byte[] convertInputStreamToByteArray(InputStream inputStream, int readLength) {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] data = new byte[readLength];
try {
Log.i(MainActivity.TAG, "Starting convert to byte array while loop");
int readTotal = 0;
int count = 0;
while (count >= 0 && readTotal < readLength) {
count = inputStream.read(data, readTotal, readLength - readTotal);
if (readLength > 0) {
readTotal += count;
}
}
Log.i(MainActivity.TAG, "Finished convert to byte array while loop");
} catch (IOException e) {
Log.e(MainActivity.TAG, "error: " + e.getMessage());
e.printStackTrace();
}
return data;
}
This is the problem:
int count = inputStream.read();
while (count != -1) {
You're consuming a byte and then ignoring it. That means the next value you read (the size) will be incorrect. You need a different way of telling whether you're at the end of the stream. Some options:
Send a -1 when you're finished; that way you can stop as soon as readInt returns -1
If you know it, send the number of images you're going to send before you start sending them
Use mark(1), then read(), then reset() - if your stream supports marking. I don't know whether it will or not. You could always wrap it in BufferedInputStream if not.
Reimplement DataInputStream.readInt yourself in a way which detects the end of the stream as being an expected possibility instead of throwing an exception
Just catch an exception in readInt (not nice - getting to the end of the stream isn't really exceptional)
I make a java socket server and c socket client, here are the code
Java Socket Server:
int Send_Request(String s) {
try {
os = socket.getOutputStream();
os.write(s.getBytes());
os.flush();
Log.d(tag,"Data = " + s);
return 0;
} catch (IOException e) {
// TODO Auto-generated catch block
Log.e(tag,"Send Request Error");
return -1;
}
C socket client:
void* recv_request()
{
int i,in;
char buf[1024];
while(1)
{
if ( ( in = read(sockfd, buf, strlen(buf)) ) != -1 )
{
LOGD("Received = %s ...",buf);
sendServerCutText(buf);
memset(buf,0,strlen(buf));
}
}
}
The problem is .. when i send from the server, it is blocking on flush(), the c client cannot receive until there is another Send_Request called.
where is the problem??
There is a conceptual problem. You think TCP is message oriented (sending messages with specified length). It's not like that, it is only providing a stream of bytes.
For sending a message with a specified length a common technique is to send first the length (encoded in fixed length, eg 4 byte integer in network byte order) and then the actual message.
There's also an implementation problem, the 3rd argument for read should be the maximum read length, which should be 1024, executing strlen on a non-initialized local char array is clearly undefined-behaviour.
Looking to read in some bytes over a socket using an inputStream. The bytes sent by the server may be of variable quantity, and the client doesn't know in advance the length of the byte array. How may this be accomplished?
byte b[];
sock.getInputStream().read(b);
This causes a 'might not be initialized error' from the Net BzEAnSZ. Help.
You need to expand the buffer as needed, by reading in chunks of bytes, 1024 at a time as in this example code I wrote some time ago
byte[] resultBuff = new byte[0];
byte[] buff = new byte[1024];
int k = -1;
while((k = sock.getInputStream().read(buff, 0, buff.length)) > -1) {
byte[] tbuff = new byte[resultBuff.length + k]; // temp buffer size = bytes already read + bytes last read
System.arraycopy(resultBuff, 0, tbuff, 0, resultBuff.length); // copy previous bytes
System.arraycopy(buff, 0, tbuff, resultBuff.length, k); // copy current lot
resultBuff = tbuff; // call the temp buffer as your result buff
}
System.out.println(resultBuff.length + " bytes read.");
return resultBuff;
Assuming the sender closes the stream at the end of the data:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buf = new byte[4096];
while(true) {
int n = is.read(buf);
if( n < 0 ) break;
baos.write(buf,0,n);
}
byte data[] = baos.toByteArray();
Read an int, which is the size of the next segment of data being received. Create a buffer with that size, or use a roomy pre-existing buffer. Read into the buffer, making sure it is limited to the aforeread size. Rinse and repeat :)
If you really don't know the size in advance as you said, read into an expanding ByteArrayOutputStream as the other answers have mentioned. However, the size method really is the most reliable.
Without re-inventing the wheel, using Apache Commons:
IOUtils.toByteArray(inputStream);
For example, complete code with error handling:
public static byte[] readInputStreamToByteArray(InputStream inputStream) {
if (inputStream == null) {
// normally, the caller should check for null after getting the InputStream object from a resource
throw new FileProcessingException("Cannot read from InputStream that is NULL. The resource requested by the caller may not exist or was not looked up correctly.");
}
try {
return IOUtils.toByteArray(inputStream);
} catch (IOException e) {
throw new FileProcessingException("Error reading input stream.", e);
} finally {
closeStream(inputStream);
}
}
private static void closeStream(Closeable closeable) {
try {
if (closeable != null) {
closeable.close();
}
} catch (Exception e) {
throw new FileProcessingException("IO Error closing a stream.", e);
}
}
Where FileProcessingException is your app-specific meaningful RT exception that will travel uninterrupted to your proper handler w/o polluting the code in between.
The simple answer is:
byte b[] = new byte[BIG_ENOUGH];
int nosRead = sock.getInputStream().read(b);
where BIG_ENOUGH is big enough.
But in general there is a big problem with this. A single read call is not guaranteed to return all that the other end has written.
If the nosRead value is BIG_ENOUGH, your application has no way of knowing for sure if there are more bytes to come; the other end may have sent exactly BIG_ENOUGH bytes ... or more than BIG_ENOUGH bytes. In the former case, you application will block (for ever) if you try to read. In the latter case, your application has to do (at least) another read to get the rest of the data.
If the nosRead value is less than BIG_ENOUGH, your application still doesn't know. It might have received everything there is, part of the data may have been delayed (due to network packet fragmentation, network packet loss, network partition, etc), or the other end might have blocked or crashed part way through sending the data.
The best answer is that EITHER your application needs to know beforehand how many bytes to expect, OR the application protocol needs to somehow tell the application how many bytes to expect or when all bytes have been sent.
Possible approaches are:
the application protocol uses fixed message sizes (not applicable to your example)
the application protocol message sizes are specified in message headers
the application protocol uses end-of-message markers
the application protocol is not message based, and the other end closes the connection to say that is the end.
Without one of these strategies, your application is left to guess, and is liable to get it wrong occasionally.
Then you use multiple read calls and (maybe) multiple buffers.
Stream all Input data into Output stream. Here is working example:
InputStream inputStream = null;
byte[] tempStorage = new byte[1024];//try to read 1Kb at time
int bLength;
try{
ByteArrayOutputStream outputByteArrayStream = new ByteArrayOutputStream();
if (fileName.startsWith("http"))
inputStream = new URL(fileName).openStream();
else
inputStream = new FileInputStream(fileName);
while ((bLength = inputStream.read(tempStorage)) != -1) {
outputByteArrayStream.write(tempStorage, 0, bLength);
}
outputByteArrayStream.flush();
//Here is the byte array at the end
byte[] finalByteArray = outputByteArrayStream.toByteArray();
outputByteArrayStream.close();
inputStream.close();
}catch(Exception e){
e.printStackTrace();
if (inputStream != null) inputStream.close();
}
Either:
Have the sender close the socket after transferring the bytes. Then at the receiver just keep reading until EOS.
Have the sender prefix a length word as per Chris's suggestion, then read that many bytes.
Use a self-describing protocol such as XML, Serialization, ...
Use BufferedInputStream, and use the available() method which returns the size of bytes available for reading, and then construct a byte[] with that size. Problem solved. :)
BufferedInputStream buf = new BufferedInputStream(is);
int size = buf.available();
Here is a simpler example using ByteArrayOutputStream...
socketInputStream = socket.getInputStream();
int expectedDataLength = 128; //todo - set accordingly/experiment. Does not have to be precise value.
ByteArrayOutputStream baos = new ByteArrayOutputStream(expectedDataLength);
byte[] chunk = new byte[expectedDataLength];
int numBytesJustRead;
while((numBytesJustRead = socketInputStream.read(chunk)) != -1) {
baos.write(chunk, 0, numBytesJustRead);
}
return baos.toString("UTF-8");
However, if the server does not return a -1, you will need to detect the end of the data some other way - e.g., maybe the returned content always ends with a certain marker (e.g., ""), or you could possibly solve using socket.setSoTimeout(). (Mentioning this as it is seems to be a common problem.)
This is both a late answer and self-advertising, but anyone checking out this question may want to take a look here:
https://github.com/GregoryConrad/SmartSocket
This question is 7 years old, but i had a similiar problem, while making a NIO and OIO compatible system (Client and Server might be whatever they want, OIO or NIO).
This was quit the challenge, because of the blocking InputStreams.
I found a way, which makes it possible and i want to post it, to help people with similiar problems.
Reading a byte array of dynamic sice is done here with the DataInputStream, which kann be simply wrapped around the socketInputStream. Also, i do not want to introduce a specific communication protocoll (like first sending the size of bytes, that will be send), because i want to make this as vanilla as possible. First of, i have a simple utility Buffer class, which looks like this:
import java.util.ArrayList;
import java.util.List;
public class Buffer {
private byte[] core;
private int capacity;
public Buffer(int size){
this.capacity = size;
clear();
}
public List<Byte> list() {
final List<Byte> result = new ArrayList<>();
for(byte b : core) {
result.add(b);
}
return result;
}
public void reallocate(int capacity) {
this.capacity = capacity;
}
public void teardown() {
this.core = null;
}
public void clear() {
core = new byte[capacity];
}
public byte[] array() {
return core;
}
}
This class only exists, because of the dumb way, byte <=> Byte autoboxing in Java works with this List. This is not realy needed at all in this example, but i did not want to leave something out of this explanation.
Next up, the 2 simple, core methods. In those, a StringBuilder is used as a "callback". It will be filled with the result which has been read and the amount of bytes read will be returned. This might be done different of course.
private int readNext(StringBuilder stringBuilder, Buffer buffer) throws IOException {
// Attempt to read up to the buffers size
int read = in.read(buffer.array());
// If EOF is reached (-1 read)
// we disconnect, because the
// other end disconnected.
if(read == -1) {
disconnect();
return -1;
}
// Add the read byte[] as
// a String to the stringBuilder.
stringBuilder.append(new String(buffer.array()).trim());
buffer.clear();
return read;
}
private Optional<String> readBlocking() throws IOException {
final Buffer buffer = new Buffer(256);
final StringBuilder stringBuilder = new StringBuilder();
// This call blocks. Therefor
// if we continue past this point
// we WILL have some sort of
// result. This might be -1, which
// means, EOF (disconnect.)
if(readNext(stringBuilder, buffer) == -1) {
return Optional.empty();
}
while(in.available() > 0) {
buffer.reallocate(in.available());
if(readNext(stringBuilder, buffer) == -1) {
return Optional.empty();
}
}
buffer.teardown();
return Optional.of(stringBuilder.toString());
}
The first method readNext will fill the buffer, with byte[] from the DataInputStream and return the amount bytes read this way.
In the secon method, readBlocking, i utilized the blocking nature, not to worry about consumer-producer-problems. Simply readBlocking will block, untill a new byte-array is received. Before we call this blocking method, we allocate a Buffer-size. Note, i called reallocate after the first read (inside the while loop). This is not needed. You can safely delete this line and the code will still work. I did it, because of the uniqueness of my problem.
The 2 things, i did not explain in more detail are:
1. in (the DataInputStream and the only short varaible here, sorry for that)
2. disconnect (your disconnect routine)
All in all, you can now use it, this way:
// The in has to be an attribute, or an parameter to the readBlocking method
DataInputStream in = new DataInputStream(socket.getInputStream());
final Optional<String> rawDataOptional = readBlocking();
rawDataOptional.ifPresent(string -> threadPool.execute(() -> handle(string)));
This will provide you with a way of reading byte arrays of any shape or form over a socket (or any InputStream realy). Hope this helps!