Develop Reference

How to create IPC through sockets with a Java client and a Python server?

I am trying to make two processes communicate through local sockets: a Python server and a Java client. The data I want to pass between both consists of the bytes of a Protobuf object, with variable size. I want the connection to remain open and be used until the end of the program, because I'm passing a lot of objects that need to be processed.
Because Protobuf objects have variable size, I am sending the size of the message/response before sending the true message/response containing the object.
Currently, I am using a TCPServer from the socketserver library on the Python side. I have the following handler implemented:
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The request handler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def recv_all(self, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = self.request.recv(n - len(data))
if not packet:
return None
data += packet
return data
def handle(self):
logger.debug("Beginning of handle cycle for client: {}.".format(self.client_address))
while True:
if True: # please disregard this if condition
# Receive 4 bytes (1 int) denoting the size of the message
data_length_bytes: bytes = self.recv_all(4)
logger.debug('Received data_length: {}'.format(data_length_bytes))
# If recv read an empty request b'', then client has closed the connection
if not data_length_bytes:
break
data_length: int = int.from_bytes(data_length_bytes.strip(), byteorder='big')
data: bytes = self.recv_all(data_length).strip()
response: bytes = data.upper()
# Send length of response first
self.request.sendall(len(response).to_bytes(4, byteorder='big'))
# Send response
self.request.sendall(response)
logger.debug(
'Sent response to: {}. Size of response: {} bytes. Response: {}.'.format(self.client_address,
len(response),
response))
logger.debug("End of handle cycle for client: {}.".format(self.client_address))
And the following client:
class SocketClient
{
private static Socket socket;
private int port;
private DataOutputStream out;
private DataInputStream in;
SocketClient(int port)
{
this.port = port;
this.createSocket();
}
private void createSocket() {
InetAddress address;
try {
address = InetAddress.getByName("localhost");
socket = new Socket(address, port);
this.out = new DataOutputStream(socket.getOutputStream());
this.in = new DataInputStream(socket.getInputStream());
} catch (IOException e) {
e.printStackTrace();
}
}
byte[] sendMessageAndReceiveResponse(byte[] messageToSend){
try {
if(true) { // again, please disregard this condition
//Send the size of the message to the server
this.out.writeInt(messageToSend.length);
out.flush();
this.out.write(messageToSend);
out.flush();
//Get the response message from the server
int length = in.readInt(); // read length of incoming message
byte[] buffer = null;
if(length>=0) {
buffer = new byte[length];
in.readFully(buffer, 0, buffer.length); // read the message
}
return buffer;
}
}
catch (ConnectException exception) {
System.out.println("ATTENTION! Could not connect to socket. Nothing was retrieved from the Python module.");
exception.printStackTrace();
return null;
}
catch (Exception exception)
{
exception.printStackTrace();
return null;
}
}
void close(){
//Closing the socket
try
{
in.close();
out.close();
socket.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
I run the following experiment after starting the Python server:
SocketClient socketClient = new SocketClient(5000);
byte[] response;
// Case 1
response = socketClient.sendMessageAndReceiveResponse("12345678".getBytes());
System.out.println(new String(response));
// Case 2
response = socketClient.sendMessageAndReceiveResponse("123456781".getBytes());
System.out.println(new String(response));
// Case 3
response = socketClient.sendMessageAndReceiveResponse("12345678123456781".getBytes());
System.out.println(new String(response));
socketClient.close();
Case 1 and case 3 work well. However, when I run case 2, on the Python server side, I get the following log:
DEBUG -- [handle()] Received data_length: b'\x00\x00\x00\t' # The '\t' shouldn't be here. A '\x09' should.
And then the server throws and exception and exits the connection. This happens with every string with 8 < length < 14. What am I doing wrong, and is there an easier way to achieve what I want?

I figured out why I was having problems with messages of 8 < length < 14.
I was getting the \t character when length was equal to 9. I noticed that if I changed the length to 10, it would become \n. And to 13, \r. I realized that there wasn't any \t magically appearing. Python was for some reason converting \x09 to \t, because the horizontal tab character \t has an ASCII code equal to 9!
And when I applied the strip() function in this line:
data_length: int = int.from_bytes(data_length_bytes.strip(), byteorder='big')
, Python deleted my \t, which was actually my \x09. My problem was logging the value before stripping it, and so I took a long time to figure out my mistake.
Therefore the solution was to simply not use strip(). I leave here my current working code (at least for my tests), for someone to use:
Python server handler:
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The request handler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def recv_all(self, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = self.request.recv(n - len(data))
if not packet:
return None
data += packet
return data
def handle(self):
while True:
data_length_bytes: bytes = self.recv_all(4)
# If recv read an empty request b'', then client has closed the connection
if not data_length_bytes:
break
# DON'T DO strip() ON THE DATA_LENGTH PACKET. It might delete what Python thinks is whitespace but
# it actually is a byte that makes part of the integer.
data_length: int = int.from_bytes(data_length_bytes, byteorder='big')
# Don't do strip() on data either (be sure to check if there is some error if you do use)
data: bytes = self.recv_all(data_length)
response: bytes = data.upper()
self.request.sendall(len(response).to_bytes(4, byteorder='big'))
self.request.sendall(response)
The Java client remained the same, but without that if(true) condition that I was using for debug reasons.

Java TCP Socket Byte Heap Memory Issue

I have a Java TCP Server Socket program that is expecting about 64 bytes of data from a piece of remote hardware. The Server code is:
public void run () throws Exception
{
//Open a socket on localhost at port 11111
ServerSocket welcomeSocket = new ServerSocket(11111);
while(true) {
//Open and Accept on Socket
Socket connectionSocket = welcomeSocket.accept();
DataInputStream dIn = new DataInputStream(connectionSocket.getInputStream());
int msgLen = dIn.readInt();
System.out.println("RX Reported Length: "+ msgLen);
byte[] msg = new byte[msgLen];
if(msgLen > 0 ) {
dIn.readFully(msg);
System.out.println("Message Length: "+ msg.length);
System.out.println("Recv[HEX]: " + StringTools.toHexString(msg));
}
}
}
This works correctly as I am able to test locally with a simple ACK program:
public class ACK_TEST {
public static void main (String[] args)
{
System.out.println("Byte Sender Running");
try
{
ACK_TEST obj = new ACK_TEST ();
obj.run();
}
catch (Exception e)
{
e.printStackTrace ();
}
}
public void run () throws Exception
{
Socket clientSocket = new Socket("localhost", 11111);
DataOutputStream dOut = new DataOutputStream(clientSocket.getOutputStream());
byte rtn[] = null;
rtn = new byte[1];
rtn[0] = 0x06; // ACK
dOut.writeInt(rtn.length); // write length of the message
dOut.write(rtn); // write the message
System.out.println("Byte Sent");
clientSocket.close();
}
}
And this correctly produces this output from the Server side:
However, when I deploy the same Server code on the Raspberry Pi and the hardware sends data to it, the data length is far greater and causes a heap memory issue (Even with the Heap pre-set at 512MB, which is definitely incorrect and unnecessary)
My presumption is I am reading the data wrong from the TCP socket as from the debug from the hardware, it's certainly not sending packets of this size.
Update: I have no access to the Client source code. I do however need to take the input TCP data stream, place it into a byte array, and then another function (Not shown) parses out some known HEX codes. That function expects a byte array input.
Update: I reviewed the packet documentation. It is a 10 byte header. The first Byte is a protocol identifier. The next 2 bytes is the Packet Length (Total number of bytes in the packet, including all the header bytes and checksum) and the last 7 are a Unique ID. Therefore, I need to read those 2 bytes and create a byte array that size.

Apparently the length from the header is about 1GB. Looks like the problem on the other end. Don't you mix low/big endian encoding?

datainputstream missing bytes in.read(b)

I'm trying to write a program which acts as a server that will read bytes from a client that is written in PHP - sends request via socket (which i cannot recode due to policy) Here is the server code:
The server runs in: Red Hat Enterprise Linux Server release 6.2 (Santiago)
public void run() {
try {
serverSocket = new ServerSocket(port);
serverSocket.setSoTimeout(0);
while(!isInterrupted) {
try {
Socket server = serverSocket.accept();
LOG.info("Request received from : " + server.getRemoteSocketAddress());
DataInputStream in = new DataInputStream(server.getInputStream());
// DataInputStream in = new DataInputStream(
// new BufferedInputStream(server.getInputStream(), 10000));
byte[] bytes = new byte[10000];
int byteDupLength = in.read(t_bytes);
// in.readFully(bytes); // I tried this but to no avail
// int byteDupLength = bytes.length;
LOG.info(byteDupLength);
byte[] byteDup = new byte[byteDupLength];
System.arraycopy(bytes, 4, byteDup, 0, byteDupLength);
// FOR INFORMATION ONLY
/*for (byte b : byteDup){
LOG.info(b);
}*/
ByteBuffer buffer = ByteBuffer.wrap(byteDup);
LOG.info(buffer);
forwardRequest(byteDup);
server.close();
}
catch(SocketTimeoutException s) {
LOG.error("Socket timed out!", s);
break;
}
catch(IOException e)
{
LOG.error("IOException:", e);
break;
}
}
}
catch (IOException ex) {
LOG.error("Server socket is null", ex);
}
LOG.fatal("ReceiverEngine interrupted!");
}
I encountered a problem when the client sends request consisting of 4948 bytes. The only bytes the server can read is 2090.
Another thing that seems a mystery to me is that, when I run the server via Netbeans in my local (which is a Windows 7 Pro), it works as expected. I dont know what is wrong. Please help.. :)
Thanks!

TCP is a byte stream protocol.
The read() method isn't guaranteed to fill the buffer.
Therefore if you don't receive the expected number of bytes in a single read, you have to loop until you do receive them.
readFully() would have worked if the buffer size agreed with the size of what was sent. In your case you specified a buffer of 10,000 bytes, which weren't sent, so it would have blocked waiting for the other 10000-4948 bytes.

Socket connectivity loss when sending live streaming data

I'm struggling here...
I'm trying to determine if data was successfully sent to the server through a TCP socket using the OutputStream object. For testing on emulator socket communications is loss after 30 sec. For write data OutputStream.write(); its doesn't throw an exception , and local server continuously running its not crashing, only tcp socket connection is loss after some time. All the methods in the socket class return as though the socket is active and working. Is there anything I'm doing wrong here? Is there any socket implementation or stream implementation I can use to get an exception or error when the stream/ socket doesn't actually send the data in the buffer? Also setting setSoTimeout() on the socket doesn't seem to do anything.
Please guide me...
Here is my code:
private void sendRec() {
int lstream;
int port = 1012;
byte[] byterecv = new byte[1040];
while (true) {
System.out.println("POOL-2");
synchronized (recSendThread) {
try {
recSendThread.wait(20);
} catch (InterruptedException e) {
}
}
if (stopcall == true) {
// break;
}
try {
// Provides a client-side TCP socket
Socket clientRec = new Socket();
// serverSocket = new ServerSocket(port);
// serverSocket.setSoTimeout(5000);
// Connects this socket to the given remote host address and
// port
clientRec.connect(new InetSocketAddress("192.168.1.36", port));
System.out.println("Just connected to "
+ clientRec.getRemoteSocketAddress());
System.out.println("SENTS Rec BEFORE");
// output streams that write data to the network
OutputStream outToServerRec = clientRec.getOutputStream();
DataOutputStream outStreamRec = new DataOutputStream(
outToServerRec);
outStreamRec.write(bData);
System.out.println("SENTS Rec AFTER");
// input streams that read data from network
InputStream inFromServerRec = clientRec.getInputStream();
// clientRec.setSoTimeout(5000);
DataInputStream inStreamRec = new DataInputStream(
inFromServerRec);
while ((lstream = inStreamRec.read(byterecv)) != -1) {
System.out.println("startrec bytearray -- "
+ byterecv.length);
bos1.write(byterecv, 0, lstream);
}
inStreamRec.close();// for closing dataouputstream
clientRec.close();// for closing socket connection
} catch (IOException e) {
e.printStackTrace();
}
}
}
Here is my receiver and player code..
/**
* start receiving the voice data from server
* */
protected void startplay() {
System.arraycopy(frndid, 0, playByteData, 0, 4);
System.arraycopy(userid, 0, playByteData, 4, 4);
ByteBuffer.wrap(sessionid).order(ByteOrder.LITTLE_ENDIAN).asIntBuffer().
put(call_sessionid);
System.arraycopy(sessionid, 0, playByteData, 8, 4);
int lstream;
int port = 1014;
while (true) {
System.out.println("POOL-3");
try {
if (stopcall == true) {
System.out.println("BREAKEDDDD1111");
//break;
}
// Host name
// port++;
InetAddress addressPlay = InetAddress.getByName("192.168.1.36");
// Creates a new streaming socket connected to the target host
Socket clientPlay = new Socket(addressPlay, port);
System.out.println("Just connected to play : " +
clientPlay.getRemoteSocketAddress());
System.out.println("SENTS Play BEFORE");
// output streams that write data
OutputStream outToServer = clientPlay.getOutputStream();
DataOutputStream outStreamPlay = new DataOutputStream(outToServer);
outStreamPlay.write(playByteData);
System.out.println("SENTS Play after");
// input streams that read data
InputStream inFromServerPlay = clientPlay.getInputStream();
DataInputStream inStreamPlay = new DataInputStream(inFromServerPlay);
//clientPlay.setSoTimeout(5000);
while ((lstream = inStreamPlay.read(byteArray)) != -1) {
System.out.println("startplay() bytearray -- " +
byteArray.length);
bos.write(byteArray, 0, lstream);
}
inStreamPlay.close();
clientPlay.close();// for closing play socket connection
responseBuffer = bos.toByteArray();
System.out.println("BAOSSIZE " + bos.size());
bos.reset();
bos.flush();
bos.close();
playing = true;
System.out.println("res length -- " + responseBuffer.length);
rcvbb=ByteBuffer.wrap(responseBuffer).order(ByteOrder.LITTLE_ENDIAN).
asShortBuffer().get(playShortData);
playVoiceReceived();// plays received data
} catch (ParseException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
/**
* start playing received the voice data from server
* */
public void playVoiceReceived() {
System.out.println("POOL-4");
try {
if (at != null) {
if (at.getPlayState() != AudioTrack.PLAYSTATE_PLAYING) {
at.play();// starts playing
} else {
System.out.println("Play BEFORE WRITE");
// Writes the audio data to the audio hardware for playback.
at.write(playShortData, 0, BufferElements2Play);
System.out.println("Play AFTER WRITE");
at.flush();
}
}
} catch (Exception e) {
e.printStackTrace();
}
}

The socket has sent the data ... to the local socket send buffer. What happens after that is up to the local TCP stack, the network, the remote TCP stack, and the remote application. If you want to know whether the remote application got the data, it will have to send you a reply.

Operator write does not check whether data was delivered because otherwise it would have to wait for too long time. If network connection is actually down, TCP layer of operating system, will try to send data anyway, but will detect problems somewhat later, i.e. 1 minute later, because it will not receive acknowledgement messages from the opposite side. It will then try to resend data several times, see that problem persists and only then will report exception condition on the socket. To know that socket is in exception condition, you need to perform some operator on socket, i.e. another write attempt. Try doing write in a loop like this:
while (true)
{
outStreamRec.write (data);
Thread.sleep (1000L);
}
It should throw an error about 2 minutes after network will be down.
Note, that in opposite to write operation, operation connect is synchronous, so it actually waits for response from the opposite side, and if there is not respose, it will throw an exception.

Java SocketChannel read the messages of unknown length

SocketChannel channel = (SocketChannel) key.channel();
ByteBuffer buffer = ByteBuffer.allocate(1024);
int numRead = -1;
try {
numRead = channel.read(buffer);
System.out.println("numRead: " + numRead);
}
catch (IOException e) { e.printStackTrace();}
if (numRead == -1) {
this.dataMap.remove(channel);
Socket socket = channel.socket();
SocketAddress remoteAddr = socket.getRemoteSocketAddress();
System.out.println("Connection closed by client: " + remoteAddr);
channel.close();
key.cancel();
return;
}
System.out.println("Got: " + new String(buffer.array(), "windows-1251"));
From the socket reads 1024 bytes of data. In this case all messages are combined, and the last message does not come fully. How do I read the data into a buffer before the message separator '|' ? I want to receive each message individually.

Depends on the protocol, but if the protocol is based on a message separator character, your only option is to read as much as you can without blocking, then scan through the content you read to find the separator. You need to do this in a loop until the connection is closed (or until your side decides to close the connection).
Of course you can read multiple messages at once and even partial messages, so you have to make sure you handle those cases adequately.

There is no such thing as a message in TCP. It is a byte-stream API. If you want messages you must implement them yourself, via a separator, STX/ETX pair, length word prefix, self-describing protocol, etc.