Develop Reference

Handling binary packets

I coded this packet handler but I can imagine scenarios in which it will get stuck or won't be able to read incomplete data. My questions are:
Should I use two buffers, one for the current incoming data and other to append incomplete data to?
I'm being stupidly over-complicated?
Code:
byte[] buffer;
int bufferLength;
int bytesRead;
buffer = new byte[1024];
while (bluetoothConnected) {
try {
// Wait for packet header
if (mmInStream.available() >= 8) {
bufferLength = mmInStream.read(buffer);
bytesRead = 0;
// Parse every packet
while (true) {
int commandType = ByteBuffer.wrap(buffer, 0, 2).order(ByteOrder.LITTLE_ENDIAN).getShort();
int payloadSize = ByteBuffer.wrap(buffer, 2, 2).order(ByteOrder.LITTLE_ENDIAN).getShort();
int packetSize = PACKET_HEADER_SIZE + payloadSize;
// Break if payload is incomplete
if (bufferLength < (bytesRead + packetSize)) {
// Append to other buffer
break;
}
byte[] packet = new byte[packetSize];
System.arraycopy(buffer, bytesRead, packet, 0, packetSize);
parsePacketSequence(socket, packet);
bytesRead += packetSize;
// Break if all bytes are read
if (bufferLength == bytesRead)
{
break;
}
// Break if more bytes are needed
// Packet header incomplete
if ((bufferLength - bytesRead) < PACKET_HEADER_SIZE)
{
// Append to other buffer
break;
}
}
}
}
catch (IOException e) {
bluetoothConnected = false;
Log.d(TAG, "Error " + e);
break;
}
}

Should I use two buffers, one for the current incoming data and other to append incomplete data to?
No.
I'm being stupidly over-complicated?
Yes.
Here's a simple version using DataInputStream:
DataInputStream din = new DataInputStream(mmInStream);
while (bluetoothConnected) {
try {
// Read packet header
int commandType = swap(din.readShort());
int payloadSize = swap(din.readShort());
int packetSize = PACKET_HEADER_SIZE + payloadSize;
byte[] packet = new byte[packetSize];
din.readFully(packet);
parsePacketSequence(socket, packet);
}
catch (IOException e) {
bluetoothConnected = false;
Log.d(TAG, "Error " + e);
break;
}
}
The swap() method which converts a short in litte-endian byte order to Java byte order is left as an exercise for the reader.
NB I don't see how parsePacketSequence() can work if it doesn't know commandType.
E&OE

Java sending handshake packets to minecraft server

I have been working on a java program that basically acts like Minechat(text-based app to just view chat.) I have never really worked with networking too much, so the issue is figuring out how to send packets correctly. I am currently at the position of creating the handshake with the server. After hours of research, I have come up with the following code, but it always runs into the "Failed! (Exception)" message. To me, everything looks correct, but for all I know it could be 100% wrong. If someone could point out what I'm doing wrong here, I'd really appreciate it.
For reference, feel free to use this and this.
public static void main(String[] args) throws IOException {
host = new InetSocketAddress("162.244.165.111", 48040);
socket = new Socket();
System.out.println("Connecting...");
socket.connect(host, 3000);
System.out.println("Done!");
System.out.println("Making streams...");
output = new DataOutputStream(socket.getOutputStream());
input = new DataInputStream(socket.getInputStream());
System.out.println("Done!");
System.out.println("Attempting handshake... "+host.getAddress().toString().substring(1));
byte[] msg = ("47;"+host.getAddress().toString().substring(1)+";"+host.getPort()+";2;").getBytes(Charset.forName("UTF-16"));
output.writeInt(msg.length+Integer.valueOf(0x00));
output.writeByte(0x00);
output.write(msg);
output.flush();
try {
if (input.readByte() != 0x02)
System.out.println("Failed!");
else
System.out.println("Done!");
} catch (EOFException e) {
System.out.println("Failed! (Exception)");
}
}
EDIT:
More research suggests I use a Byte array, but this confuses me on how to represent a string and using strings is required?

Looking at this page http://wiki.vg/Protocol it looks like your not writing enough data nor in the right order. You also need to be using varint which is a special type of data representation of an integer.
Relevant links to this issue:
Handshake Protocol
Packet format
Server Ping Explanation and Example (which involves handshake)
The status ping works as follows:
C->S : Handshake State=1
C->S : Request
S->C : Response
C->S : Ping
S->C : Pong
C is client and S is server
Using the wiki and the provided code samples I modified your code to follow the entire status request.
public static void main(String [] args) throws IOException {
String address = "162.244.165.111";
int port = 48040;
InetSocketAddress host = new InetSocketAddress(address, port);
Socket socket = new Socket();
System.out.println("Connecting...");
socket.connect(host, 3000);
System.out.println("Done!");
System.out.println("Making streams...");
DataOutputStream output = new DataOutputStream(socket.getOutputStream());
DataInputStream input = new DataInputStream(socket.getInputStream());
System.out.println("Done!");
System.out.println("Attempting handshake... "+host.getAddress().toString());
byte [] handshakeMessage = createHandshakeMessage(address, port);
// C->S : Handshake State=1
// send packet length and packet
writeVarInt(output, handshakeMessage.length);
output.write(handshakeMessage);
// C->S : Request
output.writeByte(0x01); //size is only 1
output.writeByte(0x00); //packet id for ping
// S->C : Response
int size = readVarInt(input);
int packetId = readVarInt(input);
if (packetId == -1) {
throw new IOException("Premature end of stream.");
}
if (packetId != 0x00) { //we want a status response
throw new IOException("Invalid packetID");
}
int length = readVarInt(input); //length of json string
if (length == -1) {
throw new IOException("Premature end of stream.");
}
if (length == 0) {
throw new IOException("Invalid string length.");
}
byte[] in = new byte[length];
input.readFully(in); //read json string
String json = new String(in);
// C->S : Ping
long now = System.currentTimeMillis();
output.writeByte(0x09); //size of packet
output.writeByte(0x01); //0x01 for ping
output.writeLong(now); //time!?
// S->C : Pong
readVarInt(input);
packetId = readVarInt(input);
if (packetId == -1) {
throw new IOException("Premature end of stream.");
}
if (packetId != 0x01) {
throw new IOException("Invalid packetID");
}
long pingtime = input.readLong(); //read response
// print out server info
System.out.println(json);
System.out.println("Done!");
}
public static byte [] createHandshakeMessage(String host, int port) throws IOException {
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
DataOutputStream handshake = new DataOutputStream(buffer);
handshake.writeByte(0x00); //packet id for handshake
writeVarInt(handshake, 4); //protocol version
writeString(handshake, host, StandardCharsets.UTF_8);
handshake.writeShort(port); //port
writeVarInt(handshake, 1); //state (1 for handshake)
return buffer.toByteArray();
}
public static void writeString(DataOutputStream out, String string, Charset charset) throws IOException {
byte [] bytes = string.getBytes(charset);
writeVarInt(out, bytes.length);
out.write(bytes);
}
public static void writeVarInt(DataOutputStream out, int paramInt) throws IOException {
while (true) {
if ((paramInt & 0xFFFFFF80) == 0) {
out.writeByte(paramInt);
return;
}
out.writeByte(paramInt & 0x7F | 0x80);
paramInt >>>= 7;
}
}
public static int readVarInt(DataInputStream in) throws IOException {
int i = 0;
int j = 0;
while (true) {
int k = in.readByte();
i |= (k & 0x7F) << j++ * 7;
if (j > 5) throw new RuntimeException("VarInt too big");
if ((k & 0x80) != 128) break;
}
return i;
}

How to read all of Inputstream in Server Socket JAVA

I am using Java.net at one of my project.
and I wrote a App Server that gets inputStream from a client.
But some times my (buffered)InputStream can not get all of OutputStream that client sent to my server.
How can I write a wait or some thing like that, that my InputStream gets all of the OutputStream of client?
(My InputStream is not a String)
private Socket clientSocket;
private ServerSocket server;
private BufferedOutputStream outputS;
private BufferedInputStream inputS;
private InputStream inBS;
private OutputStream outBS;
server = new ServerSocket(30501, 100);
clientSocket = server.accept();
public void getStreamFromClient() {
try {
outBS = clientSocket.getOutputStream();
outputS = new BufferedOutputStream( outBS);
outputS.flush();
inBS = clientSocket.getInputStream();
inputS = new BufferedInputStream( inBS );
} catch (Exception e) {
e.printStackTrace();
}
}
Thanks.

The problem you have is related to TCP streaming nature.
The fact that you sent 100 Bytes (for example) from the server doesn't mean you will read 100 Bytes in the client the first time you read. Maybe the bytes sent from the server arrive in several TCP segments to the client.
You need to implement a loop in which you read until the whole message was received.
Let me provide an example with DataInputStream instead of BufferedinputStream. Something very simple to give you just an example.
Let's suppose you know beforehand the server is to send 100 Bytes of data.
In client you need to write:
byte[] messageByte = new byte[1000];
boolean end = false;
String dataString = "";
try
{
DataInputStream in = new DataInputStream(clientSocket.getInputStream());
while(!end)
{
int bytesRead = in.read(messageByte);
dataString += new String(messageByte, 0, bytesRead);
if (dataString.length == 100)
{
end = true;
}
}
System.out.println("MESSAGE: " + dataString);
}
catch (Exception e)
{
e.printStackTrace();
}
Now, typically the data size sent by one node (the server here) is not known beforehand. Then you need to define your own small protocol for the communication between server and client (or any two nodes) communicating with TCP.
The most common and simple is to define TLV: Type, Length, Value. So you define that every message sent form server to client comes with:
1 Byte indicating type (For example, it could also be 2 or whatever).
1 Byte (or whatever) for length of message
N Bytes for the value (N is indicated in length).
So you know you have to receive a minimum of 2 Bytes and with the second Byte you know how many following Bytes you need to read.
This is just a suggestion of a possible protocol. You could also get rid of "Type".
So it would be something like:
byte[] messageByte = new byte[1000];
boolean end = false;
String dataString = "";
try
{
DataInputStream in = new DataInputStream(clientSocket.getInputStream());
int bytesRead = 0;
messageByte[0] = in.readByte();
messageByte[1] = in.readByte();
int bytesToRead = messageByte[1];
while(!end)
{
bytesRead = in.read(messageByte);
dataString += new String(messageByte, 0, bytesRead);
if (dataString.length == bytesToRead )
{
end = true;
}
}
System.out.println("MESSAGE: " + dataString);
}
catch (Exception e)
{
e.printStackTrace();
}
The following code compiles and looks better. It assumes the first two bytes providing the length arrive in binary format, in network endianship (big endian). No focus on different encoding types for the rest of the message.
import java.nio.ByteBuffer;
import java.io.DataInputStream;
import java.net.ServerSocket;
import java.net.Socket;
class Test
{
public static void main(String[] args)
{
byte[] messageByte = new byte[1000];
boolean end = false;
String dataString = "";
try
{
Socket clientSocket;
ServerSocket server;
server = new ServerSocket(30501, 100);
clientSocket = server.accept();
DataInputStream in = new DataInputStream(clientSocket.getInputStream());
int bytesRead = 0;
messageByte[0] = in.readByte();
messageByte[1] = in.readByte();
ByteBuffer byteBuffer = ByteBuffer.wrap(messageByte, 0, 2);
int bytesToRead = byteBuffer.getShort();
System.out.println("About to read " + bytesToRead + " octets");
//The following code shows in detail how to read from a TCP socket
while(!end)
{
bytesRead = in.read(messageByte);
dataString += new String(messageByte, 0, bytesRead);
if (dataString.length() == bytesToRead )
{
end = true;
}
}
//All the code in the loop can be replaced by these two lines
//in.readFully(messageByte, 0, bytesToRead);
//dataString = new String(messageByte, 0, bytesToRead);
System.out.println("MESSAGE: " + dataString);
}
catch (Exception e)
{
e.printStackTrace();
}
}
}

You can read your BufferedInputStream like this. It will read data till it reaches end of stream which is indicated by -1.
inputS = new BufferedInputStream(inBS);
byte[] buffer = new byte[1024]; //If you handle larger data use a bigger buffer size
int read;
while((read = inputS.read(buffer)) != -1) {
System.out.println(read);
// Your code to handle the data
}

int c;
String raw = "";
do {
c = inputstream.read();
raw+=(char)c;
} while(inputstream.available()>0);
InputStream.available() shows the available bytes only after one byte is read, hence do .. while

Bluetooth-connection; can't send strings properly

I have troubles with my program when i need to send Strings from my server bluetooth-socket to my client bluetooth-socket.
Everything works fine as long as I am only sending one String at a time (for example chatting) but if I need to write more Strings at a short period of time (to interchange informations), the Strings will not get seperated from the client code. For example if I'm sending "FirstUser" and right after that "SecondUser" the client does not read "FirstUser" and then "SecondUser". It will read "FirstUserSecondUser". How can I avoid this behaviour?
Edit: If I let the Thread sleep before it is able to send a new message, it reads the right strings but this solution is not working fine for my need.
Server-Code: sending to all clients(edited)
public synchronized void sendToAll(String message)
{
try {
Thread.sleep(100);
} catch (InterruptedException e1) {
e1.printStackTrace();
}
publishProgress(message);
for(OutputStream writer:outputList) {
try {
writer.write(message.getBytes());
writer.flush();
} catch (IOException e) {
System.out.println("Some-Error-Code");
}
}
}
Server-Code: reading from a client:
public void run() {
String nachricht;
int numRead;
byte[] buffer = new byte[1024];
while (runningFlag)
{
try {
if((numRead = inputStream.read(buffer)) >= 0) {
nachricht = new String(buffer, 0, numRead);
serverThread.handleMessage(nachricht);
}
}
catch (IOException e) {
this.cancel();
e.printStackTrace();
}
}
}
Client-Code: reading from server(edited)
#Override
protected Void doInBackground(Integer... ints) {
String nachricht = new String();
byte[] buffer = new byte[1024];
int numRead;
while (runningFlag)
{
try {
if(((numRead = inputStream.read(buffer)) >= 0)) {
nachricht = new String(buffer, 0, numRead);
publishProgress(nachricht);
}
}
catch (IOException e) {
clientGame.finish();
e.printStackTrace();
}
}
return null;
}
Client-Code: writing to server
public synchronized void write(String nachricht)
{
try {
Thread.sleep(100);
} catch (InterruptedException e1) {
e1.printStackTrace();
}
try {
outputStream.write(nachricht.getBytes());
outputStream.flush();
} catch (IOException e) {
this.cancel();
e.printStackTrace();
}
}
I appreciate every little help :) .

You need to encapsulate your data item to avoid concatenation.
It means that you have to write and read a whole data item before continuing.
You should have some utility methods to do that instead of directly using methods of OutputStream and InputStream :
public static void writeItem(OutputStream out, String s) throws IOException
{
// Get the array of bytes for the string item:
byte[] bs = s.getBytes(); // as bytes
// Encapsulate by sending first the total length on 4 bytes :
// - bits 7..0 of length
out.write(bs.length); // modulo 256 done by write method
// - bits 15..8 of length
out.write(bs.length>>>8); // modulo 256 done by write method
// - bits 23..16 of length
out.write(bs.length>>>16); // modulo 256 done by write method
// - bits 31..24 of length
out.write(bs.length>>>24); // modulo 256 done by write method
// Write the array content now:
out.write(bs); // Send the bytes
out.flush();
}
public static String readItem(InputStream in) throws IOException
{
// first, read the total length on 4 bytes
// - if first byte is missing, end of stream reached
int len = in.read(); // 1 byte
if (len<0) throw new IOException("end of stream");
// - the other 3 bytes of length are mandatory
for(int i=1;i<4;i++) // need 3 more bytes:
{
int n = in.read();
if (n<0) throw new IOException("partial data");
len |= n << (i<<3); // shift by 8,16,24
}
// Create the array to receive len bytes:
byte[] bs = new byte[len];
// Read the len bytes into the created array
int ofs = 0;
while (len>0) // while there is some byte to read
{
int n = in.read(bs, ofs, len); // number of bytes actually read
if (n<0) throw new IOException("partial data");
ofs += n; // update offset
len -= n; // update remaining number of bytes to read
}
// Transform bytes into String item:
return new String(bs);
}
Then you use these methods both for server & client to read and write your String items.

Java stream socket can only send once

I'm writing a simple download accelerator. The problem is I can send and receive messages once. The next time I try to send and receive message, I get no response froms server. I'm not even sure if I am able to send the second message.
The first message is something like;
*HEAD /TIPS/LAWLER/PANOHOW2.PDF HTTP/1.0\r\n
HTTP/1.0\r\n
Connection: close\r\n
\r\n*
and response is;
*HTTP/1.1 200 OK
Date: Mon, 24 Jan 2011 10:53:38 GMT
Server: Apache
Last-Modified: Tue,
22 Sep 1998 13:19:52 GMT
ETag: "1968013-2b4f4-3386e15b6ee00"
Accept-Ranges: bytes
Content-Length: 177396
Connection: close
Content-Type: application/pdf*
When i attemp to sen message;
GET /TIPS/LAWLER/hedeh/PANOHOW2.PDF HTTP/1.0\r\n
Range: bytes=0-44349\r\n
Connection: close\r\n
\r\n
I get nothing.
What is wrong with my code?
public class Main {
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
//Parse URL
String cmd = "http://www.imaging-resource.com"; //Host Name
if (cmd.contains("http://"))
{
cmd = cmd.substring(7); //
if (cmd.contains("/"))
{
int index = cmd.indexOf("/");
cmd = cmd.substring(0, index);
System.out.println(cmd);
}
}
String str = "HEAD /TIPS/LAWLER/PANOHOW2.PDF HTTP/1.0\r\nConnection: close\r\n\r\n"; //First message to send
//Create socket, connect, initialize read and write handlers
//in, out
Socket socket = null; //Create a client socket
SocketAddress sockaddr = null;
InetAddress address = null;
InputStream input = null; //Input handler
OutputStream output = null; //Output handler
try
{
address = InetAddress.getByName(cmd); //Get ip using host name
socket = new Socket(); //Contrusct Socket
sockaddr = new InetSocketAddress(address, 80);
//socket.setTcpNoDelay(false);
socket.connect(sockaddr, 2000); //Connect to server set and timeout to 2 sec
} //End of try Block
catch (Exception ex)
{
Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
System.out.println(ex);
} //End of catch Block
if (!socket.isConnected())
{
System.out.println("not connected");
System.exit(-1);
}
//Sending package here
try
{
int c;
byte[] buf = new byte[65535];
char[] chr = new char[65535];
input = socket.getInputStream(); //Input handler is created
output = socket.getOutputStream(); //Output handler is created
buf = str.getBytes(); //HEAD message converted into byte array
output.write(buf); //Sending message to server
output.flush();
int counter = 0;
while ((c = input.read()) != -1) //Reading received package
chr[counter++]=(char)c;
//input.reset();
str = new String(chr); //For better manipulation, server message is converted to string
System.out.println(str);
} catch (IOException e)
{
System.err.print(e);
} //End of catch
int index = str.indexOf("Content-Length"); //Look for "Content-Length" in response
str = str.substring(index); //Using its beginning index create an substring
index = str.indexOf("\r\n"); //Search for end of line
str = str.substring(0, index); //Erase end if line chars - \r\n
str = str.substring(16, str.length()); //"Content-Length: " 16 chars
int fileSize = Integer.parseInt(str); //Lentgh of file is converted to Integer
int[][] parts = new int[4][2]; //Beginning and en of jobs for threads will be stored here
int remainder = fileSize; //Bytes left to split for rest of the threads will be stored here
int start = 0;
int finish = 0;
for (int i = 0; i < 4; i++) //Number of threads many times
{
parts[i][0] = start; //*******Each threads job Interval(eg. 0-108)
//System.out.print(parts[i][0] + "-"); //******
finish += remainder / 4 - i; //*****
parts[i][1] = finish; //****
start = finish + 1; //***
if (i + 1 == 4)
parts[i][1] = fileSize; //*
}
str = "GET /TIPS/LAWLER/hedeh/PANOHOW2.PDF HTTP/1.0\r\nRange: bytes=" + parts[0][0] + "-" + parts[0][1] + "\r\nConnection: close\r\n\r\n";
//System.out.println(str);
if(!socket.isConnected())
{
System.out.println("closed");
try
{
socket.connect(sockaddr, 2000);
}//End od try
catch(Exception e){
System.err.print(e);
}//End of catch
}//End of If
System.out.println("Is Outputhandler closed :"+socket.isOutputShutdown());
System.out.println("Is Inputhandler closed :"+socket.isInputShutdown());
try
{
int c;
byte[] buf = new byte[65535];
char[] chr = new char[65535];
buf = str.getBytes(); //Output handler is created
output.write(buf); //Sending message to server
output.flush();
int counter = 0;
if((c = input.read()) != -1)
{
chr[counter++] = (char) c;
while ((c = input.read()) != -1) //Reading received package
{
System.out.println("response is not -1");
chr[counter++]=(char)c;
}
str = new String(chr); //For better manipulation, serve message is converted to string
System.out.println("Response "+str);
}//End of If
else System.out.println("No Response!");
}catch(Exception e)
{System.err.print(e);}
//Closing open stuff
try {
output.close();
input.close();
socket.close();
} catch (Exception e) {
System.out.println(e);
}
}// End of main method
}//End of class definition

The first message is something like;
HTTP/1.0\r\n
You have to use HTTP version 1.1 to use multiple requests on a single TCP connection.
From the Wikipedia article on HTTP:
In HTTP/0.9 and 1.0, the connection is closed after a single request/response pair. In HTTP/1.1 a keep-alive-mechanism was introduced, where a connection could be reused for more than one request.
Also, as #Joachim Sauer points out in the comments, you're explicitly saying Connection: close in your header. :-)

I think that the problem is that you are trying to connect to HTTP server using plain TCP socket. Yes, HTTP is on top of TCP but it is complicated protocol that requires a lot of things to know. I'd suggest you to work with higher level API that implements HTTP protocol and provides you more convenient API.
The simplest example is URL+URLConnection from JDK. Probably better is HttpClient from Jakarta.