Develop Reference

How to create IPC through sockets with a Java client and a Python server?

I am trying to make two processes communicate through local sockets: a Python server and a Java client. The data I want to pass between both consists of the bytes of a Protobuf object, with variable size. I want the connection to remain open and be used until the end of the program, because I'm passing a lot of objects that need to be processed.
Because Protobuf objects have variable size, I am sending the size of the message/response before sending the true message/response containing the object.
Currently, I am using a TCPServer from the socketserver library on the Python side. I have the following handler implemented:
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The request handler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def recv_all(self, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = self.request.recv(n - len(data))
if not packet:
return None
data += packet
return data
def handle(self):
logger.debug("Beginning of handle cycle for client: {}.".format(self.client_address))
while True:
if True: # please disregard this if condition
# Receive 4 bytes (1 int) denoting the size of the message
data_length_bytes: bytes = self.recv_all(4)
logger.debug('Received data_length: {}'.format(data_length_bytes))
# If recv read an empty request b'', then client has closed the connection
if not data_length_bytes:
break
data_length: int = int.from_bytes(data_length_bytes.strip(), byteorder='big')
data: bytes = self.recv_all(data_length).strip()
response: bytes = data.upper()
# Send length of response first
self.request.sendall(len(response).to_bytes(4, byteorder='big'))
# Send response
self.request.sendall(response)
logger.debug(
'Sent response to: {}. Size of response: {} bytes. Response: {}.'.format(self.client_address,
len(response),
response))
logger.debug("End of handle cycle for client: {}.".format(self.client_address))
And the following client:
class SocketClient
{
private static Socket socket;
private int port;
private DataOutputStream out;
private DataInputStream in;
SocketClient(int port)
{
this.port = port;
this.createSocket();
}
private void createSocket() {
InetAddress address;
try {
address = InetAddress.getByName("localhost");
socket = new Socket(address, port);
this.out = new DataOutputStream(socket.getOutputStream());
this.in = new DataInputStream(socket.getInputStream());
} catch (IOException e) {
e.printStackTrace();
}
}
byte[] sendMessageAndReceiveResponse(byte[] messageToSend){
try {
if(true) { // again, please disregard this condition
//Send the size of the message to the server
this.out.writeInt(messageToSend.length);
out.flush();
this.out.write(messageToSend);
out.flush();
//Get the response message from the server
int length = in.readInt(); // read length of incoming message
byte[] buffer = null;
if(length>=0) {
buffer = new byte[length];
in.readFully(buffer, 0, buffer.length); // read the message
}
return buffer;
}
}
catch (ConnectException exception) {
System.out.println("ATTENTION! Could not connect to socket. Nothing was retrieved from the Python module.");
exception.printStackTrace();
return null;
}
catch (Exception exception)
{
exception.printStackTrace();
return null;
}
}
void close(){
//Closing the socket
try
{
in.close();
out.close();
socket.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
I run the following experiment after starting the Python server:
SocketClient socketClient = new SocketClient(5000);
byte[] response;
// Case 1
response = socketClient.sendMessageAndReceiveResponse("12345678".getBytes());
System.out.println(new String(response));
// Case 2
response = socketClient.sendMessageAndReceiveResponse("123456781".getBytes());
System.out.println(new String(response));
// Case 3
response = socketClient.sendMessageAndReceiveResponse("12345678123456781".getBytes());
System.out.println(new String(response));
socketClient.close();
Case 1 and case 3 work well. However, when I run case 2, on the Python server side, I get the following log:
DEBUG -- [handle()] Received data_length: b'\x00\x00\x00\t' # The '\t' shouldn't be here. A '\x09' should.
And then the server throws and exception and exits the connection. This happens with every string with 8 < length < 14. What am I doing wrong, and is there an easier way to achieve what I want?

I figured out why I was having problems with messages of 8 < length < 14.
I was getting the \t character when length was equal to 9. I noticed that if I changed the length to 10, it would become \n. And to 13, \r. I realized that there wasn't any \t magically appearing. Python was for some reason converting \x09 to \t, because the horizontal tab character \t has an ASCII code equal to 9!
And when I applied the strip() function in this line:
data_length: int = int.from_bytes(data_length_bytes.strip(), byteorder='big')
, Python deleted my \t, which was actually my \x09. My problem was logging the value before stripping it, and so I took a long time to figure out my mistake.
Therefore the solution was to simply not use strip(). I leave here my current working code (at least for my tests), for someone to use:
Python server handler:
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The request handler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def recv_all(self, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = self.request.recv(n - len(data))
if not packet:
return None
data += packet
return data
def handle(self):
while True:
data_length_bytes: bytes = self.recv_all(4)
# If recv read an empty request b'', then client has closed the connection
if not data_length_bytes:
break
# DON'T DO strip() ON THE DATA_LENGTH PACKET. It might delete what Python thinks is whitespace but
# it actually is a byte that makes part of the integer.
data_length: int = int.from_bytes(data_length_bytes, byteorder='big')
# Don't do strip() on data either (be sure to check if there is some error if you do use)
data: bytes = self.recv_all(data_length)
response: bytes = data.upper()
self.request.sendall(len(response).to_bytes(4, byteorder='big'))
self.request.sendall(response)
The Java client remained the same, but without that if(true) condition that I was using for debug reasons.

How to send bytes from Python to Java using sockets?

I've read a few posts on how to send a picture using sockets in Python, and how to send a picture using sockets in Java, I was wanting to combine the two and send a picture from Python to Java using sockets on both ends. Most of my code is taken from the posts I read but here is the python client:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("192.168.0.12",4141))
try:
file = open("subbed.jpg", 'rb')
bytes = file.read()
print "{0:b}".format(len(bytes))
size = len(bytes)
s.sendall(size)
answer = s.recv(4096)
print "Answer = %s" %answer
if answer == 'GOT SIZE':
s.sendall(bytes)
answer = s.recv(4096)
if answer == 'GOT IMAGE' :
s.sendall("byte")
file.close()
finally:
s.close()
the code for the Java server is:
public static void main(String[] args) {
while(true) {
try (
ServerSocket server = new ServerSocket(PORT_NUMBER);
Socket client = server.accept();
PrintWriter out = new PrintWriter(client.getOutputStream(), true);
InputStream in = client.getInputStream()) {
System.out.println("GOT CONNECTION FROM: " + client.getInetAddress().toString());
byte[] sizeAr = new byte[4];
in.read(sizeAr);
int size = ByteBuffer.wrap(sizeAr).asIntBuffer().get();
System.out.println(Integer.toBinaryString(size));
out.println("GOT SIZE");
byte[] imageAr = new byte[size];
in.read(imageAr);
BufferedImage image = ImageIO.read(new ByteArrayInputStream(imageAr));
ImageIO.write(image, "jpg", new File("C:\\myprivatelocation\\test.jpg"));
} catch (Exception ioe) {
ioe.printStackTrace();
}
}
}
The initial problem comes from sending the size I think. I'm no python expert, nor am I a Java expert, but I think what's happening is Python is sending the size as a string and Java is receiving it as a byte array and converting it to an integer, and there are some differences in the way they are stored in the two languages. Can anyone offer any assistance with this issue?

Although I would approach your problem slightly differently, the following code works:
Python Sender
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect(("127.0.0.1", 8888))
with open("C:\\temp\\test-input.jpg", 'rb') as f:
content = f.read()
size = len(content)
print("File bytes:", size)
s.sendall(size.to_bytes(4, byteorder='big'))
buff = s.recv(4)
resp = int.from_bytes(buff, byteorder='big')
print("Response:", resp)
if size == resp:
s.sendall(content)
buff = s.recv(2)
print(buff)
print("Complete.")
Java Receiver
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.ByteBuffer;
import javax.imageio.ImageIO;
class Server{
public static void main(String[] args) {
int PORT_NUMBER = 8888;
try (
ServerSocket server = new ServerSocket(PORT_NUMBER);
Socket client = server.accept();
OutputStream sout = client.getOutputStream();
InputStream sin = client.getInputStream();
){
System.out.println("GOT CONNECTION FROM: " + client.getInetAddress().toString());
// Get length
byte[] size_buff = new byte[4];
sin.read(size_buff);
int size = ByteBuffer.wrap(size_buff).asIntBuffer().get();
System.out.format("Expecting %d bytes\n", size);
// Send it back (?)
sout.write(size_buff);
// Create Buffers
byte[] msg_buff = new byte[1024];
byte[] img_buff = new byte[size];
int img_offset = 0;
while(true) {
int bytes_read = sin.read(msg_buff, 0, msg_buff.length);
if(bytes_read == -1) { break; }
// Copy bytes into img_buff
System.arraycopy(msg_buff, 0, img_buff, img_offset, bytes_read);
img_offset += bytes_read;
System.out.format("Read %d / %d bytes...\n", img_offset, size);
if(img_offset >= size) { break; }
}
BufferedImage image = ImageIO.read(new ByteArrayInputStream(img_buff));
ImageIO.write(image, "jpg", new File("C:\\temp\\test-output.jpg"));
// Send "OK"
byte[] OK = new byte[] {0x4F, 0x4B};
sout.write(OK);
}
catch (IOException ioe) { ioe.printStackTrace(); }
}
}
The sender opens a socket, reads the file, and sends the receiver the length. The receiver gets the length, parses the bytes and sends it back. Upon receipt of the "confirmation", the sender then sends the file contents. The receiver will then repeatedly read 1024 byte chunks from the socket input stream, inserting the bytes into img_data. When there are no more bytes expected (or the socket is closed), the receiver will send "OK" to the sender (unconditionally) and exit. The sender will just print that "OK" (in bytes), and exit.
Some of this could be cleaned up with a ByteArrayOutputStream, but I wanted to get as close to the functionality of your code as possible.

Something is off - you should be getting some kind of error when trying to send some integer in the socket:
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(('localhost', 7777))
>>> s.sendall(len(b'some bytes'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: a bytes-like object is required, not 'int'
Sockets can only send bytes, you have to convert the int object containing the size to bytes somehow, python won't do it automatically for you. Your code should fail with the error above.
In the java code you are using asIntBuffer() and Integer.toBinaryString to convert your values, in the python code you just try to send the data without converting, you should get error.
Now, to convert the int to bytes, you can use the struct module; It will convert to a binary representation used by the C language - I think that's what your java code expect
size_in_bytes = struct.pack('I', len(data_to_send))
In the same way, you should use struct.unpack to convert the bytes back to a integer object. See the documentation for more details and a table of possible conversions.

UDP Reading stream of bytes

I want to read a stream of bytes when sending a command through UDP. I have the following code:
import java.io.*;
import java.net.*;
public class udp_client
{
private static File fout = new File("data2.txt");
public static void main(String args[])
{
DatagramSocket sock = null;
int port = 10001;
String s;
BufferedReader cin = new BufferedReader(new InputStreamReader(System.in));
try
{
sock = new DatagramSocket();
InetAddress host = InetAddress.getByName("192.168.0.2");
byte[] b = {(byte)0xf3,(byte)0xf2,(byte)0x02,(byte)0x01,(byte)0x00,(byte)0x00,(byte)0xfe,(byte)0x0e };
DatagramPacket dp = new DatagramPacket(b , b.length , host , port);
sock.send(dp);
//now receive reply
//buffer to receive incoming data
byte[] buffer = new byte[65536];
DatagramPacket reply = new DatagramPacket(buffer, buffer.length);
sock.receive(reply);
byte[] data = reply.getData();
s = new String(data, 0, reply.getLength());
//echo the details of incoming data - client ip : client port - client message
echo(reply.getAddress().getHostAddress() + " : " + reply.getPort() + " - " + s);
FileOutputStream fos = new FileOutputStream(fout);
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fos));
for (int i=6;i<3078;i=i+3) {
//System.out.println(Integer.toHexString((data[k])));
System.out.println("hex="+Integer.valueOf(String.valueOf(data[k]), 16));
int r = (data[i] & 0xFF) | ((data[i+1] & 0xFF) << 8) | ((data[i+2] & 0x0F) << 16);
bw.write(Integer.toString(r));
bw.newLine();
//System.out.println("r["+i+"]="+r);
System.out.println(r);
//System.out.println(String.valueOf(data[i] & 0x00FF));
//JOptionPane.showMessageDialog(null, data[k]);
}
bw.close();
}
catch(IOException e)
{
System.err.println("IOException " + e);
}
}
//simple function to echo data to terminal
public static void echo(String msg)
{
System.out.println(msg);
}
}
the problem is that size of the replyis 520 and the data should be around 5000 bytes. How can I read the next data?

If the sender is sending packets that are 5000 bytes long, then the receive(packet) should receive and return them in their entirety provided that the buffer in the packet is large enough. (If the packet buffer is too small, the message will be truncated. Also, see caveats below about packet loss and fragmentation.)
In your case, the receive buffer is large enough to hold a 5000 byte message, so that implies that the sender is not sending UDP messages of that size.
Note: UDP is not recommended for sending large amounts of data. UDP messages get lost, and the receiver gets no notification if this occurs. The larger the message you send, the more likely it is that this will occur.
When you send a UDP message that won't fit into a single network packet, it will be broken up into fragments. The fragments are reassembled into the complete receiver in the OS protocol stack. However, if any of the fragments gets lost, the UDP message cannot be reassembled, and will be discarded by the OS (without notification).
Recommendations:
If you need reliability, use TCP rather than UDP.
Avoid sending large UDP messages.

Sending file to unknown client address through socket?

Essentially what I'm trying to do is have a client (with an unknown IP) connect via socket to a server, and have the server send a BufferedImage back to the client every x seconds.
I understand how this can be done with a known client IP, but not with an unknown. A simple example would be awesome, thanks.
Example with known IP:
BufferedImage scr = getImage();
Socket sock = new Socket(ip, 123456); //unknown IP
byte[] mybytearray = new byte[1024];
InputStream is = sock.getInputStream();
FileOutputStream fos = new FileOutputStream(scr);
BufferedOutputStream bos = new BufferedOutputStream(fos);
int bytesRead = is.read(mybytearray, 0, mybytearray.length);
bos.write(mybytearray, 0, bytesRead);
bos.close();
sock.close();
Also, if anyone could show me where I could loop it to keep sending the file, that would be awesome.

I whipped up some sample server client connection. Basically you just define your server to use your localhost then port forward or open the server port depending on what your network configuration is. Thers lots of tutorials about that online. On your client you need to either know your external or internal IP address depending on where your connecting from.
This sample just uses your localhost and sends a file from your harddrive, but I wrote it specifically with the intended use of adding in any InputStream or OutputStream so you can adapt this to reading or writing an image. Most servers you will just bind your ip address to 127.0.0.1. When connection to the server outside your local network you will need to find your external IP address. You can find this on websites like whatsmyip.org.
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.UnknownHostException;
public class SendFileExample {
public static void main(String [] args) throws IOException {
System.out.print("Type 's' for server or 'c' for client: ");
char c = (char) System.in.read();
if(Character.toLowerCase(c) == 's') {
createServer();
} else if(Character.toLowerCase(c) == 'c') {
createClient();
}
}
public static void createServer() throws IOException {
// create a server to listen on port 12345
ServerSocket socket = new ServerSocket(12345, 0, InetAddress.getByName("127.0.0.1"));
System.out.println("Server started on " + socket.getInetAddress().getHostAddress() + ":" + socket.getLocalPort() + ",\nWaiting for client to connect.");
Socket clientConnection = socket.accept();
System.out.println("Client accepted from "+clientConnection.getInetAddress().getHostAddress()+", sending file");
pipeStreams(new FileInputStream(new File("c:\\from.txt")), clientConnection.getOutputStream(), 1024);
System.out.println("File sent, closing out connection");
clientConnection.close();
socket.close();
}
public static void createClient() throws IOException {
System.out.println("Connecting to server.");
Socket socket = new Socket();
// connect to an address, this is the server address (which you have to know)
socket.connect(new InetSocketAddress(InetAddress.getByName("127.0.0.1"), 12345));
// read all bytes from the socket
System.out.println("Success, retreiving file.");
pipeStreams(socket.getInputStream(), new FileOutputStream(new File("c:\\to.txt")), 1024);
System.out.println("Done, file sent. Closing connection");
socket.close();
}
/**
* writes all bytes from inputStream to outputStream
* #param source
* #param out
* #throws IOException
*/
public static void pipeStreams(java.io.InputStream source, java.io.OutputStream destination, int bufferSize) throws IOException {
// 16kb buffer
byte [] buffer = new byte[bufferSize];
int read = 0;
while((read=source.read(buffer)) != -1) {
destination.write(buffer, 0, read);
}
destination.flush();
destination.close();
source.close();
}
}

I think the cleanest way to achieve this would be to have the client connect every x seconds to the server, and pull the image from the socket's stream.
If you want the server to have the initiative, have the client connect to the server, and then keep the socket open to send images every x second. This means that the client must be ready to read the images as they come. It also means that the length of an image must be sent before it's content, since the image is not terminated by the end of the stream.

InputStream.available() reports no bytes when listening to Bluetooth RFCOMM server that constantly streams data

I am trying to write a simple script in Java to read data from a bluetooth device that spits out a constant stream of data. I know the device is working, because I can solve my problem using Python, but I want to use Java eventually.
I have some sample code, but it hangs on the read command.
// Ref http://homepages.ius.edu/rwisman/C490/html/JavaandBluetooth.htm
import java.io.*;
import javax.microedition.io.*;
//import javax.bluetooth.*;
public class RFCOMMClient {
public static void main(String args[]) {
try {
StreamConnection conn = (StreamConnection) Connector.open(
"btspp://00078093523B:2", Connector.READ, true);
InputStream is = conn.openInputStream();
byte buffer[] = new byte[8];
try {
int bytes_read = is.read(buffer, 0, 8);
String received = new String(buffer, 0, bytes_read);
System.out.println("received: " + received);
} catch (IOException e) {
System.out.println(" FAIL");
System.err.print(e.toString());
}
conn.close();
} catch (IOException e) {
System.err.print(e.toString());
}
}
}
Note that the problem seems to be that the read() call believes there is no data available. However, the bluetooth device constantly spits out data (it is a sensor). Here is my Python code which does work:
In [1]: import bluetooth
In [2]: address = "00:07:80:93:52:3B"
In [3]: s = bluetooth.BluetoothSocket(bluetooth.RFCOMM)
In [4]: s.connect((address,1))
In [5]: s.recv(1024)
Out[5]: '<CONFIDENTIALDATAREMOVED>'
Please help,
Thanks!

read will block and wait for data; a better idiom would be to use available():
int bytesToRead = is.available();
if(bytesToRead > 0)
is.read(buffer, 0, bytesToRead);
else
// wait for data to become available