I'm implementing sending data through Bluetooth in Android between two Android devices. For simplicity, I'll mostly likely pass strings and parse them as JSON or other string format.
Now I'm wondering, how should I read the data to be sure, that I've received all of them? Currently I'm taking the following approach:
byte[] buffer = new byte[1024];
while (!finished) {
// My class
MemoryStream ms = new MemoryStream();
int bytesRead = 0;
do {
bytesRead = input.read(buffer, 0, buffer.length);
ms.write(buffer, 0, bytesRead);
} while (bytesRead == buffer.length);
// Now process data
}
However, this will work only if inputStream.read() will always return as many bytes as were sent on the other device.
For example, I'm assuming, that if first device sends 1234 bytes, first call to read will return 1024 and second 210. Or if first device sends 1024 bytes, first call to read will return 1024 bytes and second - -1 (stream end).
Am I right? Or should I implement my own mechanism of determining, whether all sent data was received (or should I wait for more to complete current chunk)?
The answer is: no. It is possible, that transmission will end (in terms of input.read) and not the whole sent buffer is transferred.
One has to guard transmission, preferably by preceeding the data with their size in bytes and then read data until all of them are transferred.
Related
I am making a chat program in which I have to separate a file into 1024 Bytes chunk and send it over Socket in this format:
<DataPacket>Binary_data<DataPacket\>
Currently, my only idea is to send a string
"<Packet>"
then a Byte[1024] and then a string
"<Packet\>".
So my question is:
Is there a more convenience way to do this?
Which Java class for input/output (DataInputStream, BufferedOutputStream,... ) is most suitable and a small example code of doing this?
Additional Info:
I must use Java library (JRE8), no Apache,...
I understand that I dont have to separate the data in TCP, it just be a must.
It would be very good if all the code can be run from a function like:
void SendFile(Socket sendingSocket, File fileToSend);
void ReceiveFile(Socket receivingSocket);
To chunk binary data, it's usually better to send the number of bytes first and then the raw data as bytes. So your chat client should send the string "Packet 711\n" first, then 711 bytes of the file (most file sizes are not multiples of 1024!).
That way, you can send the data in chunks and make the last chunk as small as necessary in order to avoid to corrupt the file. The line feed after the size makes it easier for the recipient to determine where the number ends and where the real raw data starts.
[EDIT] If you can't change the protocol much, maybe this works: <DataPacket>711\n ... 711 bytes binary data ... </DataPacket>
So the code to send the data would look like this:
void SendFile(Socket sendingSocket, File fileToSend) {
OutputStream stream = sendingSocket.getOutputStream();
InputStream input = new FileInputStream(fileToSend);
byte[] buffer = new byte[1024];
int len;
while(true) {
len = input.read(buffer);
if(len < 0) break;
String header = "<DataPacket>" + len + "\n";
stream.write(header.getBytes('ASCII'));
stream.write(buffer, 0, len);
String footer = "<DataPacket\>";
stream.write(footer.getBytes('ASCII'));
}
input.close();
stream.close();
}
On the receiving side, you need to parse the header (which includes the number of bytes to expect). Then you can allocate a buffer and read the bytes.
This is the subsequent question of my previous one:
Java UDP send - receive packet one by one
As I indicated there, basically, I want to receive a packet one by one as it is via UDP.
Here's an example code:
ds = new DatagramSocket(localPort);
byte[] buffer1 = new byte[1024];
DatagramPacket packet = new DatagramPacket(buffer1, buffer1.length);
ds.receive(packet);
Log.d("UDP-receiver", packet.getLength()
+ " bytes of the actual packet received");
Here, the actual packet size is say, 300bytes, but the buffer1 is allocated as 1024 byte, and to me, it's something wrong with to deal with buffer1.
How to obtain the actual packet size byte[] array from here?
and, more fundamentally, why do we need to preallocate the buffer size to receive UDP packet in Java like this? ( node.js doesn't do this )
Is there any way not to pre-allocate the buffer size and directly receive the UDP packet as it is?
Thanks for your thought.
You've answered your own question. packet.getLength() returns the actual number of bytes in the received datagram. So, you just have to use buffer[] from index 0 to index packet.getLength()-1.
Note that this means that if you're calling receive() in a loop, you have to recreate the DatagramPacket each time around the loop, or reset its length to the maximum before the receive. Otherwise getLength() keeps shrinking to the size of the smallest datagram received so far.
self answer. I did as follows:
int len = 1024;
byte[] buffer2 = new byte[len];
DatagramPacket packet;
byte[] data;
while (isPlaying)
{
try
{
packet = new DatagramPacket(buffer2, len);
ds.receive(packet);
data = new byte[packet.getLength()];
System.arraycopy(packet.getData(), packet.getOffset(), data, 0, packet.getLength());
Log.d("UDPserver", data.length + " bytes received");
}
catch()//...........
//...........
Ok, So I'm making a Java program that has a server and client and I'm sending a Zip file from server to client. I have sending the file down, almost. But recieving I've found some inconsistency. My code isn't always getting the full archive. I'm guessing it's terminating before the BufferedReader has the full thing. Here's the code for the client:
public void run(String[] args) {
try {
clientSocket = new Socket("jacob-custom-pc", 4444);
out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedInputStream(clientSocket.getInputStream());
BufferedReader inRead = new BufferedReader(new InputStreamReader(in));
int size = 0;
while(true) {
if(in.available() > 0) {
byte[] array = new byte[in.available()];
in.read(array);
System.out.println(array.length);
System.out.println("recieved file!");
FileOutputStream fileOut = new FileOutputStream("out.zip");
fileOut.write(array);
fileOut.close();
break;
}
}
}
} catch(IOException e) {
e.printStackTrace();
System.exit(-1);
}
}
So how can I be sure the full archive is there before it writes the file?
On the sending side write the file size before you start writing the file. On the reading side Read the file size so you know how many bytes to expect. Then call read until you have gotten everything you expect. With network sockets it may take more than one call to read to get everything that was sent. This is especially true as your data gets larger.
HTTP sends a content-length: x+\n in bytes. This is elegant, it might throw a TimeoutException if the conn is broken.
You are using a TCP socket. The ZIP file is probably larger than the network MTU, so it will be split up into multiple packets and reassembled at the other side. Still, something like this might happen:
client connects
server starts sending. The ZIP file is bigger than the MTU and therefore split up into multiple packets.
client busy-waits in the while (true) until it gets the first packets.
client notices that data has arrived (in.available() > 0)
client reads all available data, writes it to the file and exits
the last packets arrive
So as you can see: Unless the client machine is crazily slow and the network is crazily fast and has a huge MTU, your code simply won't receive the entire file by design. That's how you built it.
A different approach: Prefix the data with the length.
Socket clientSocket = new Socket("jacob-custom-pc", 4444);
DataInputStream dataReader = new DataInputStream(clientSocket.getInputStream());
FileOutputStream out = new FileOutputStream("out.zip");
long size = dataReader.readLong();
long chunks = size / 1024;
int lastChunk = (int)(size - (chunks * 1024));
byte[] buf = new byte[1024];
for (long i = 0; i < chunks; i++) {
dataReader.read(buf);
out.write(buf);
}
dataReader.read(buf, 0, lastChunk);
out.write(buf, 0, lastChunk);
And the server uses DataOutputStream to send the size of the file before the actual file. I didn't test this, but it should work.
How can I make sure I received whole file through socket stream?
By fixing your code. You are using InputStream.available() as a test for end of stream. That's not what it's for. Change your copy loop to this, which is also a whole lot simpler:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Use with any buffer size greater than zero, typically 8192.
In.available() just tells you that there is no data to be consumed by in.read() without blocking (waiting) at the moment but it does not mean the end of stream. But, they may arrive into your PC at any time, with TCP/IP packet. Normally, you never use in.available(). In.read() suffices everything for the reading the stream entirely. The pattern for reading the input streams is
byte[] buf;
int size;
while ((size = in.read(buf)) != -1)
process(buf, size);
// end of stream has reached
This way you will read the stream entirely, until its end.
update If you want to read multiple files, then chunk you stream into "packets" and prefix every one with an integer size. You then read until size bytes is received instead of in.read = -1.
update2 Anyway, never use in.available for demarking between the chunks of data. If you do that, you imply that there is a time delay between incoming data pieces. You can do this only in the real-time systems. But Windows, Java and TCP/IP are all these layers incompatible with real-time.
Problem: Corrupt TCP segment.
I send a sequence of ByteBuffers over and over in a SocketChannel. The sequence is the following:
\r\n
length of chunk (example: fff)
\r\n
chunk data (rubbish, a 1000 - 5000 character long string)
\r\n
length of next chunk (example: fff)
\r\n
next chunk data (rubbish, a 1000 - 5000 character long string)
...
I hope you see the pattern. The MTU on network level is about 1500, so it'll create TCP segments to send over the "chunk data".
The problem in the segments is: Somehow(?), randomly(?), a segment (its payload) starts with \r\n instead of the remaining bytes from the "chunk data" first.
So you get for example:
(segment 1)
\r\n
length of chunk (example: fff)
\r\n
chunk data (456 bytes)
(segment 2)
\r\n
chunk data (remaining 156 bytes)
length of next
\r\n
Instead of:
(segment 1)
\r\n
length of chunk (example: fff)
\r\n
chunk data (456 bytes)
(segment 2)
chunk data (remaining 156 bytes)
\r\n
length of next
\r\n
I'd like to know if Java code is even able to cause that, knowing that my "chunk data" ByteBuffer sent correctly, except for the ByteBuffer containing \r\n that joins in...
Any help is welcome, thank you for your time!
Andrew
I will bet that you are ignoring the result of a read or write. TCP does not lose or corrupt data and neither do the Socket APIs or the Java networking libraries. At least I've never seen it in about 22 years of network programming and 14 years of Java.
It is not because of network issue but with the way we coded. If we are reading and writing the data in chunks it may lead to data corruption because of last chunk. It may be possible the last data chunk read is partially filled and having default values as 0 for byte array.
Following example shows the solution for it
ObjectOutputStream out = new ObjectOutputStream(
socket.getOutputStream());
ObjectInputStream in = new ObjectInputStream(socket.getInputStream());
ObjectOutputStream out = new ObjectOutputStream(socket.getOutputStream());
//Something local returns DataInputStream from server
InputStream dataInputStream = local.downloadFile(fileToBeRead);
int chunkSize = 10000;
byte[] chunkByteArray = new byte[chunkSize];
int bytesRead = -1;
while ((bytesRead = dataInputStream.read(chunkByteArray)) != -1) {
if (bytesRead < chunkSize) {
System.out.println("Last Chunk is " + bytesRead);
chunkByteArray = getLastChunkByteArray(bytesRead,chunkByteArray);
}
out.write(chunkByteArray);
}
dataInputStream.close();
And the method
private byte[] getLastChunkByteArray(int noOfBytesRead,
byte[] partialFilledChunk) {
byte[] lastChunk = new byte[noOfBytesRead];
for (int i = 0; i < noOfBytesRead; i++) {
lastChunk[i] = partialFilledChunk[i];
}
return lastChunk;
}
Im trying to create a Speex Voip client and server. I have the basics down and its working OK on the local machine over UDP. I am using JSpeex for portability. Im looking for tips on creating the client and server. What are your thoughts?
The JSpeex library can only encode 320 bytes per call so the packets sent to the server are tiny (in my case ~244 bytes). Would it be better for the client to wait until about 1 or 2 KB of encoded data is ready before sending or let the server handle buffering the packets?
Also, any help on how to implement buffering the data would be nice.
Some of what I have that works on the local machine.
Client:
public void run() {
int nBytesToRead = (m_inputAudioFormat.getFrameSize() * 160);
int nAvailable = 0;
byte[] abPCMData = new byte[nBytesToRead];
byte[] abSpeexData = null;
UserSpeexPacket userSpeexPacket = new UserSpeexPacket("Xiphias3", "TheLounge", null, 0);
while (m_captureThread != null) {
nAvailable = m_line.available();
if (nAvailable >= nBytesToRead) {
int nBytesRead = m_line.read(abPCMData, 0, nBytesToRead);
if (nBytesRead == -1) break;
if (nBytesRead < nBytesToRead)
Arrays.fill(abPCMData, nBytesRead, abPCMData.length, (byte) 0);
abSpeexData = createSpeexPacketFromPCM(abPCMData, 0, abPCMData.length);
//DatagramPacket packet = new DatagramPacket(abSpeexData, 0, abSpeexData.length, m_connection.getInetAddress(), m_nServerPort);
userSpeexPacket.setSpeexData(abSpeexData);
userSpeexPacket.incrementPacketNumber();
DatagramPacket packet = UserSpeexPacket.userSpeexPacketToDatagramPacket(m_connection.getInetAddress(), m_connection.getPort(), userSpeexPacket);
try {
m_connection.send(packet);
}
catch(IOException iox) {
System.out.println("Connection to server lost: " + iox.getMessage());
break;
}
}
}
closeLine();
disconnect();
}
public byte[] createSpeexPacketFromPCM(byte[] abPCMData, int nOffset, int nLength)
{
byte[] abEncodedData = null;
m_speexEncoder.processData(abPCMData, nOffset, nLength);
abEncodedData = new byte[m_speexEncoder.getProcessedDataByteSize()];
m_speexEncoder.getProcessedData(abEncodedData, 0);
return abEncodedData;
}
Server:
DatagramPacket packet = new DatagramPacket(new byte[2048], 0, 2048);
byte[] abPCMData = null;
long lPrevVolPrintTime = 0;
while (m_bServerRunning) {
try {
m_serverSocket.receive(packet);
//System.out.println("Packet size is " + packet.getData().length);
//System.out.println("Got packet from " + packet.getAddress().getHostAddress());
//abPCMData = decodeSpeexPacket(packet.getData(), 0, packet.getLength());
UserSpeexPacket usp = UserSpeexPacket.datagramPacketToUserSpeexPacket(packet);
abPCMData = decodeSpeexPacket(usp.getSpeexData(), 0, usp.getSpeexData().length);
m_srcDataLine.write(abPCMData, 0, abPCMData.length);
if (System.currentTimeMillis() >= (lPrevVolPrintTime + 500)) {
//System.out.println("Current volume: " + AudioUtil.getVolumeLevelForPCM22050Hz16Bit1Channel(abPCMData, 0, abPCMData.length));
lPrevVolPrintTime = System.currentTimeMillis();
}
}
catch (IOException iox) {
if (m_bServerRunning) {
System.out.println("Server socket broke: " + iox.getMessage());
stopServer();
}
}
}
I am working on a similar project. From everything I've read, and personal experience, your best option is to work with small bits of data and send them as soon as you can. You want any jitter buffering to be done on the side of the receiver.
It is typical for a VoIP application to send 50-100 packets per second. For uLaw encoding at 8000Hz, this would result in a packet size of 80-160 bytes. The reasoning for this is that some packets will inevitably be dropped, and you want the impact to the receiver to be as small as possible. So with 10ms or 20ms of audio data per packet, a dropped packet may result in a small hiccup, but not nearly as bad as losing 2k of audio data (~250ms).
Additionally, with a large packet size, you must accumulate all of the data at the sender before sending it. So given a typical network latency of 50ms, with 20ms of audio data per packet, the receiver is not going to hear what the sender says for a minimum of 70ms. Now imagine what happens when 250ms of audio is being sent at once. 270ms will elapse between the sender speaking and the receiver playing that audio.
Users seem to be more forgiving of packet loss here and there, which results in sub-par audio quality, because the audio quality of most telephones isn't that great to begin with. However, users are also used to very low latency on modern telephone circuits, so introducing a round-trip delay of even 250ms can be extremely frustrating.
Now, as far as implementing buffering, I have found a good strategy to use a Queue (whoops, using .NET here :)), and then wrap that in a class that tracks the desired minimum and maximum number of packets in the Queue. Use rigorous locking since you will most likely be accessing it from multiple threads. If the Queue "bottoms out" and has zero packets in it (buffer underrun), set a flag and return null until the packet count reaches your desired minimum. Your consumer will have to check for null being returned and not queue anything into the output buffer, however. Alternatively your consumer could keep track of the last packet and repeatedly enqueue it, which may cause looping audio, but in some cases that may "sound" better than silence. You will have to do this until the producer puts enough packets into the queue to reach the minimum. This will result in a longer period of silence for the user, but that is generally better accepted than short, frequent periods of silence (choppiness). If you get a burst of packets and the producer fills up the queue (reaching the desired maximum), you can either start ignoring the new packets, or drop enough packets out of the front of the queue to return to the minimum.
Picking those min/max values is tough though. You are trying to balance smooth audio (no underruns) with minimum latency between sender and receiver. VoIP is fun but it sure can be frustrating! Good luck!