Develop Reference

How to create IPC through sockets with a Java client and a Python server?

I am trying to make two processes communicate through local sockets: a Python server and a Java client. The data I want to pass between both consists of the bytes of a Protobuf object, with variable size. I want the connection to remain open and be used until the end of the program, because I'm passing a lot of objects that need to be processed.
Because Protobuf objects have variable size, I am sending the size of the message/response before sending the true message/response containing the object.
Currently, I am using a TCPServer from the socketserver library on the Python side. I have the following handler implemented:
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The request handler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def recv_all(self, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = self.request.recv(n - len(data))
if not packet:
return None
data += packet
return data
def handle(self):
logger.debug("Beginning of handle cycle for client: {}.".format(self.client_address))
while True:
if True: # please disregard this if condition
# Receive 4 bytes (1 int) denoting the size of the message
data_length_bytes: bytes = self.recv_all(4)
logger.debug('Received data_length: {}'.format(data_length_bytes))
# If recv read an empty request b'', then client has closed the connection
if not data_length_bytes:
break
data_length: int = int.from_bytes(data_length_bytes.strip(), byteorder='big')
data: bytes = self.recv_all(data_length).strip()
response: bytes = data.upper()
# Send length of response first
self.request.sendall(len(response).to_bytes(4, byteorder='big'))
# Send response
self.request.sendall(response)
logger.debug(
'Sent response to: {}. Size of response: {} bytes. Response: {}.'.format(self.client_address,
len(response),
response))
logger.debug("End of handle cycle for client: {}.".format(self.client_address))
And the following client:
class SocketClient
{
private static Socket socket;
private int port;
private DataOutputStream out;
private DataInputStream in;
SocketClient(int port)
{
this.port = port;
this.createSocket();
}
private void createSocket() {
InetAddress address;
try {
address = InetAddress.getByName("localhost");
socket = new Socket(address, port);
this.out = new DataOutputStream(socket.getOutputStream());
this.in = new DataInputStream(socket.getInputStream());
} catch (IOException e) {
e.printStackTrace();
}
}
byte[] sendMessageAndReceiveResponse(byte[] messageToSend){
try {
if(true) { // again, please disregard this condition
//Send the size of the message to the server
this.out.writeInt(messageToSend.length);
out.flush();
this.out.write(messageToSend);
out.flush();
//Get the response message from the server
int length = in.readInt(); // read length of incoming message
byte[] buffer = null;
if(length>=0) {
buffer = new byte[length];
in.readFully(buffer, 0, buffer.length); // read the message
}
return buffer;
}
}
catch (ConnectException exception) {
System.out.println("ATTENTION! Could not connect to socket. Nothing was retrieved from the Python module.");
exception.printStackTrace();
return null;
}
catch (Exception exception)
{
exception.printStackTrace();
return null;
}
}
void close(){
//Closing the socket
try
{
in.close();
out.close();
socket.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
I run the following experiment after starting the Python server:
SocketClient socketClient = new SocketClient(5000);
byte[] response;
// Case 1
response = socketClient.sendMessageAndReceiveResponse("12345678".getBytes());
System.out.println(new String(response));
// Case 2
response = socketClient.sendMessageAndReceiveResponse("123456781".getBytes());
System.out.println(new String(response));
// Case 3
response = socketClient.sendMessageAndReceiveResponse("12345678123456781".getBytes());
System.out.println(new String(response));
socketClient.close();
Case 1 and case 3 work well. However, when I run case 2, on the Python server side, I get the following log:
DEBUG -- [handle()] Received data_length: b'\x00\x00\x00\t' # The '\t' shouldn't be here. A '\x09' should.
And then the server throws and exception and exits the connection. This happens with every string with 8 < length < 14. What am I doing wrong, and is there an easier way to achieve what I want?

I figured out why I was having problems with messages of 8 < length < 14.
I was getting the \t character when length was equal to 9. I noticed that if I changed the length to 10, it would become \n. And to 13, \r. I realized that there wasn't any \t magically appearing. Python was for some reason converting \x09 to \t, because the horizontal tab character \t has an ASCII code equal to 9!
And when I applied the strip() function in this line:
data_length: int = int.from_bytes(data_length_bytes.strip(), byteorder='big')
, Python deleted my \t, which was actually my \x09. My problem was logging the value before stripping it, and so I took a long time to figure out my mistake.
Therefore the solution was to simply not use strip(). I leave here my current working code (at least for my tests), for someone to use:
Python server handler:
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The request handler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def recv_all(self, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = self.request.recv(n - len(data))
if not packet:
return None
data += packet
return data
def handle(self):
while True:
data_length_bytes: bytes = self.recv_all(4)
# If recv read an empty request b'', then client has closed the connection
if not data_length_bytes:
break
# DON'T DO strip() ON THE DATA_LENGTH PACKET. It might delete what Python thinks is whitespace but
# it actually is a byte that makes part of the integer.
data_length: int = int.from_bytes(data_length_bytes, byteorder='big')
# Don't do strip() on data either (be sure to check if there is some error if you do use)
data: bytes = self.recv_all(data_length)
response: bytes = data.upper()
self.request.sendall(len(response).to_bytes(4, byteorder='big'))
self.request.sendall(response)
The Java client remained the same, but without that if(true) condition that I was using for debug reasons.

Java and Python socket working localhost but not working on any other IP

This is client code in Java
public static void main(String[] args) {
Socket rpiSocket = null;
DataInputStream in = null;
PrintStream out = null;
String str="Akif";
try {
rpiSocket = new Socket("localhost",5560);
out = new PrintStream(rpiSocket.getOutputStream());
in = new DataInputStream(new BufferedInputStream(rpiSocket.getInputStream()));
} catch (UnknownHostException e) {
System.err.println("Don't know about host: hostname");
} catch (IOException e) {
System.err.println("Couldn't get I/O for the connection to: hostname");
}
try {
if (rpiSocket != null && out != null && in != null) {
while(true) {
System.out.println("Please input your command ");
Scanner scanner = new Scanner(System.in);
String command = scanner.nextLine();
if(command.equals("KILL")) {
break;
}
System.out.println("Sending command to client: " + command);
out.print(command);
byte[] bytes = new byte[1024];
in.read(bytes);
String reply = new String(bytes, "UTF-8");
System.out.println("Reply from server: " + reply.trim());
}
}
rpiSocket.close();
System.out.println("Connections closed successfully");
}
catch (IOException e) {
System.err.println("IOException: " + e);
}
}
This is server code in Python
import socket
from sys import getsizeof
host = ''
#host = '192.168.2.181'
port = 5560
storedValue = "Yo, what's up?"
def setupServer():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print("Socket created.")
try:
s.bind((host, port))
except socket.error as msg:
print(msg)
print("Socket bind comPlete.")
return s
def setupConnection():
s.listen(1) # Allows one connection at a time.
print("Waiting for client")
conn, address = s.accept()
return conn
def GET():
reply = storedValue
return reply
def REPEAT(dataMessage):
reply = dataMessage[1]
return reply
def dataTransfer(conn, s):
# A big loop that sends/receives data until told not to.
while True:
# Receive the data
data = conn.recv(1028) # receive the data
data = data.decode('utf-8')
data = data.strip()
print("data value from client: " + data)
# Split the data such that you separate the command
# from the rest of the data.
command = str(data)
print("data length from client: " + command)
reply = ""
if command == "GET":
reply = GET()
print (command)
print (reply)
elif command == 'REPEAT':
reply = REPEAT('akif')
elif command == 'EXIT':
print("Our client has left us :(")
break
elif command == 'KILL':
print("Our server is shutting down.")
s.close()
break
else:
reply = 'Unknown Command'
# Send the reply back to the client
conn.sendall(bytes(reply, 'utf-8'))
print("Data has been sent!")
conn.close()
s = setupServer()
while True:
try:
conn = setupConnection()
dataTransfer(conn, s)
except:
break
Server and client working perfectly on localhost on my computer(win 10 pro). But if i run client on Win 10 PC and run server on Rpi PC, server working perfectly but client working until this line "in.read(bytes);" and it staying here.
P.S. = I tried turn of Windows firewall and Windows Defender. I used
wifi connection and I tried turned of modem firewall
What can i do to solve this. Thanks.

I solved the problem. The problem is Python version that i used. The problem occur pyton2 versions, but on python3 versions it Works correctly.

Python server and Java client socket

Server in python
import socket
from sys import getsizeof
host = ''
port = 5560
storedValue = "Yo, what's up?"
def setupServer():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print("Socket created.")
try:
s.bind((host, port))
except socket.error as msg:
print(msg)
print("Socket bind comPlete.")
return s
def setupConnection():
s.listen(1) # Allows one connection at a time.
print("Waiting for client")
conn, address = s.accept()
return conn
def GET():
reply = storedValue
return reply
def REPEAT(dataMessage):
reply = dataMessage[1]
return reply
def dataTransfer(conn, s):
# A big loop that sends/receives data until told not to.
while True:
# Receive the data
data = conn.recv(1028) # receive the data
data = data.decode('utf-8')
data = data.strip()
print("data value from client: " + data)
# Split the data such that you separate the command
# from the rest of the data.
command = str(data)
print("data length from client: " + command)
reply = ""
if command == "GET":
reply = GET()
print (command)
print (reply)
elif command == 'REPEAT':
reply = REPEAT(dataMessage)
elif command == 'EXIT':
print("Our client has left us :(")
break
elif command == 'KILL':
print("Our server is shutting down.")
s.close()
break
else:
reply = 'Unknown Command'
# Send the reply back to the client
conn.sendall(bytes(reply, 'utf-8'))
print("Data has been sent!")
conn.close()
s = setupServer()
while True:
try:
conn = setupConnection()
dataTransfer(conn, s)
except:
break
Client in java
import java.io.*;
import java.net.*;
import java.util.*;
public class pcClient {
public static void main(String[] args) {
Socket rpiSocket = null;
DataInputStream in = null;
PrintStream out = null;
try {
rpiSocket = new Socket("localhost",5560);
out = new PrintStream(rpiSocket.getOutputStream());
in = new DataInputStream(new BufferedInputStream(rpiSocket.getInputStream()));
} catch (UnknownHostException e) {
System.err.println("Don't know about host: hostname");
} catch (IOException e) {
System.err.println("Couldn't get I/O for the connection to: hostname");
}
try {
if (rpiSocket != null && out != null && in != null) {
while(true) {
System.out.println("Please input your command ");
Scanner scanner = new Scanner(System.in);
String command = scanner.nextLine();
if(command.equals("KILL")) {
break;
}
System.out.println("Sending command to client: " + command);
out.println(command);
byte[] bytes = new byte[1024];
in.read(bytes);
String reply = new String(bytes, "UTF-8");
System.out.println("Reply from server: " + reply.trim());
}
}
rpiSocket.close();
System.out.println("Connections closed successfully");
}
catch (IOException e) {
System.err.println("IOException: " + e);
}
}
}
I have the python server above and a java client. The java client takes input from user and sends it to the python. All this is working well. However, the python server is receiving the string from the java and an additional empty string. For example, from the java client, when i send the word "GET", the python server is able to receive this GET and print "Yo, what's up?". However, it goes back to the "While True" and immediately also additionally receives an empty string right after and starts checking the condition with that empty string. I tried to trim the strings received from the java client. How can i solve this? Thanks.

The problem is occurring because of out.println(command) you are using from the java program. It is sending the command and also some newline characters over the socket. If you replace it with out.print(command), the problem will be solved.
println may be sending the main string and the newline characters internally in two calls, I am not sure. But ideally, every "command" from the client side should first include the length of the command string, and then the real command. Server side may need buffering or splitting of the input data if recv returns more than or less than the required number of bytes.
There is a small python library datachunkpy which may help to split and process the commands (https://pypi.python.org/pypi/datachunkpy/1.0.0)

conn.recv() returns as much data as immediately available (e.g. one packet). The Java runtime library on the other hand is free to subdivide the sent data into multiple packets. I guess that it sends one packet with the "GET" and another packet with the final newline.
A solution would be for the server to wait and collect input from stream until a newline is found to finish the command.

Send a tcp request from PHP Website to JAVA program

I want to send a TCP Request from my website to my java application. So my java application should be able to receive a JSON array and print it.
I searched around for a few hours, but I could not find a solution.
Here is, what I have in PHP:
<?php
$array = array(
0 => "test",
1 => "test1"
);
json_encode($array);
$host = "tcp://localhost";
$port = 8123;
$data = json_encode($array);
$errstr = '';
$errno = '';
if ( ($fp = fsockopen($host, $port, $errno, $errstr, 3) ) === FALSE)
echo "$errstr ($errno)";
else {
print 'SUCCESS!<br />';
fwrite($fp, $data);
while (! feof($fp)) {
echo fgets($fp, 4096);
}
fclose($fp);
}
My Java code:
public class tcp {
public static void main(String argv[]) throws Exception {
String clientSentence;
String capitalizedSentence;
ServerSocket welcomeSocket = new ServerSocket(8123);
while (true) {
Socket connectionSocket = welcomeSocket.accept();
BufferedReader inFromClient =
new BufferedReader(new InputStreamReader(connectionSocket.getInputStream()));
DataOutputStream outToClient = new DataOutputStream(connectionSocket.getOutputStream());
clientSentence = inFromClient.readLine();
System.out.println("Received: " + clientSentence);
capitalizedSentence = clientSentence.toUpperCase() + '\n';
outToClient.writeBytes(capitalizedSentence);
connectionSocket.close();
}
}
}
So as you should see, I never have done something like this before.
My questions:
1) Do I have to open port "8123", also when the website and my application will run on localhost (ubuntu / debian)? -> how should I open them correctly?
2) When I start my app, I think I have to create the "tcp" java object. -> tcp tcp = new tcp - is this enough or do I have to call a method other something similar?
3) What do I have to change in my code? The Application does just nothing when I send a request...
So I hope you guys can help me with my problem :)
Greets
EDIT:
When I try to run my PHP script, I git following error:
Warning: fsockopen(): unable to connect to tcp://localhost:8123 (Connection refused) in /PATH_TO_PHP/TCPSEND/index.php on line 16
Connection refused (111)

Java client and a C++ server send and receive via TCP Socket

I have a C++ server and two clients (ruby and java).
Everything is running on a 64-bit linux-machine (java 1.7.0_17)
The ruby client is fully working, but the java version makes problems.
In Java I tried to send a String from the client to the server. Actually the Server received the entire String, but the server thinks there is still something more to receive.
The ruby client looks a little bit like this:
socket = TCPSocket.open(#options[:host],#options[:port])
test = "Hello, World"
socket.puts test
socket.shutdown 1
response = socket.gets
Everything here is working fine. The ruby client sends a string. The server receives that string and sends a reply.
The Java Version looks like:
String ip = "127.0.0.1";
int port = 6686;
java.net.Socket socket = new java.net.Socket(ip,port);
OutputStreamWriter out = new OutputStreamWriter(socket.getOutputStream());
InputStreamReader in = new InputStreamReader(socket.getInputStream());
String msg = "Hello, world!";
//send
PrintWriter pw = new PrintWriter(out, true);
pw.print(msg);
pw.flush();
// I also tried: out.write(msg); out.flush(); nothing changed
//receive the reply
BufferedReader br = new BufferedReader(in);
char[] buffer = new char[300];
int count = br.read(buffer, 0, 300);
String reply = new String(buffer, 0, count);
System.out.println(reply);
socket.close();
On the other side there is a C++ Server:
string receive(int SocketFD) {
char buffer[SOCKET_BUFFER_SIZE];
int recv_count;
// empty messagestring
string message = "";
// empty buffer
memset(buffer, 0, sizeof(buffer));
while ((recv_count = recv(SocketFD, buffer, sizeof(buffer) - 1, 0)) > 0) {
/*if (recv_count == -1) {
cout << "failed." << endl;
break;
}*/
cout << recv_count << endl;
if (ECHO_SOCKETS) cout << "received: " << buffer << endl;
message.append(buffer);
memset(buffer, 0, sizeof(buffer));
if (ECHO_SOCKETS) cout << "message is now: " << message << endl;
}
return message;
}
The server output from the Java-message is:
13
received: Hello, world!
message is now: Hello, world!
and then nothing happens.
The problem is that:
recv(SocketFD, buffer, sizeof(buffer) - 1, 0)
is catched in an endless loop (or something like that).
If I kill the Java-client process or I type something like:
pw.print(msg);
out.close();
the output on the server side is:
_sending reply: "Request unrecognized/invalid" request="Hello, world!"
send reply success
now close connection
This output is right (except "send reply success"), but in case of adding:
out.close();
the client can't receive the reply of the server. Because the Socket is closed.
java.net.SocketException: Socket is closed
at java.net.Socket.getInputStream(Socket.java:864)
at MyServer.writeMessage(MyServer.java:56)
at MyServer.test(MyServer.java:42)
at MyServer.main(MyServer.java:30)
Edit
I tried to call pw.flush(); and different delimiters like "\n", "\r", "\r\n" and "\n\r" but the server still thinks there is still something to read. I also tried to use DatagramSockets:
java.net.DatagramSocket dSocket = new java.net.DatagramSocket();
InetAddress address = InetAddress.getByName("localhost");
String msg = "Hello, world!";
byte[] buf = msg.getBytes();
java.net.DatagramPacket packet = new DatagramPacket(buf, buf.length, address, 6686);
But the server can't accept the packet.
Solution
The ruby-client does something like a socket.shutdownOutput(); (ruby: socket.shutdown 1) after the call of puts. I changed the java-client-code:
out.write(msg);
socket.shutdownOutput();
and it works!
As #Charly said: I have to define a "protocol". In my case I'm not allowed to change any communication related code (in the server and the ruby-client) because this functionality is used by a another group of researchers. So I've to modify my java-client in that way, that it does the exact same things at the exact same time as the ruby-client (something like a protocol).

PrintWriter buffer (when autoflush is true) is only flushed by calling println or printf. Calling print may not flush the buffer (Javadoc). Try calling println or use a OutputStreamWriter directly and flush().
Be aware of using the right charset (You can set it up in OutputStreamWriter constructor).

Close the stream respectively flush it in a way like this:
DataOutputStream dataOut = new DataOutputStream(socket.getOutputStream());
dataOut.writeUTF(s);
dataOut.flush();

while ((recv_count = recv(SocketFD, buffer, sizeof(buffer) - 1, 0)) > 0) {
if (recv_count == -1) {
I don't know what your problem is but this code is certainly nonsense. It is impossible for the inner test ever to succeed.