Develop Reference

Getting Garbage value while reading from serial port in java

I am reading data from serialport using jSerialComm package. I am getting following data
A total of 22 bytes are being received and I am getting these three garbage bytes too. The readable data is correct but what these garbage characters are happening?
Following is my code.
public static void main(String[] args) {
SerialPort serialPort = SerialPort.getCommPort("/dev/ttyUSB0");
if(serialPort.openPort())
{
System.out.println("Port Opened Successfully...");
}
else
{
System.out.println("Unable to open port....");
return;
}
serialPort.setComPortParameters(1200, 8, 1, 0);
try
{
while(true)
{
while(serialPort.bytesAvailable() != 0)
{
Thread.sleep(1000);
byte[] readBuffer = new byte[serialPort.bytesAvailable()];
int numRead = serialPort.readBytes(readBuffer, readBuffer.length);
String data = new String(readBuffer);
System.out.println("Read "+numRead+ " bytes." + readBuffer);
System.out.println(data);
}
}
}
catch(Exception e)
{
e.printStackTrace();
}
serialPort.closePort();
System.out.println("done...");
}

SerialPort.readBytes seems to be not thread safe, thus yielding "garbage" when called during character reception.
I suggest to use the snippet from the author's example which worked fine for me:
// Get a new instance of SerialPort by opening a port.
SerialPort port = SerialPort.open("COM2");
// Configure the connection
port.setTimeout(100);
port.setConfig(BaudRate.B115200, Parity.NONE, StopBits.ONE, DataBits.B8);
// You have the choice, you can either use the Java NIO channels
// or classic Input/Ouput streams to read and write data.
//DEL SerialChannel channel = port.getChannel();
InputStream istream = port.getInputStream();
// Read some data using a stream
byte[] byteBuffer = new byte[4096];
// Will timeout after 100ms, returning 0 if no bytes were available.
int n = istream.read(byteBuffer);
// *** Use n bytes of byteBuffer ***
//DEL ...
port.close();
//DEL : removed from the original code for clarity

You did not explain the protocol but I suggest to look it up. Presumably these are control characters or like a comment suggests binary data. You create a String from a byte buffer without an encoding so this also depends on your environment/ JVM's default encoding.
Try treating the first and the last two bytes as specified in the protocol in use for your project. It might also be related to jSerialComm not removing serial signalling, e.g. handshake, EOT and such.
If you're reverse-engineering the protocol instead maybe also try another library like RxTx to see if the bytes stay the same.
To inspect the bytes safely use for example BigInteger to print out a Hex-String instead:
BigInteger bigInteger = new BigInteger(1, bytes);
System.out.printf("%0" + (bytes.length << 1) + "x", bigInteger);

Use this code block and it will run perfectly;
serialPort.addDataListener(new SerialPortDataListener() {
#Override
public int getListeningEvents() {
return SerialPort.LISTENING_EVENT_DATA_RECEIVED;
}
#Override
public void serialEvent(SerialPortEvent serialPortEvent) {
if (serialPortEvent.getEventType() != SerialPort.LISTENING_EVENT_DATA_RECEIVED) {
return;
}
byte[] newData = serialPortEvent.getReceivedData();
String data = new String(newData);
System.out.println(data);
}
});

How to create IPC through sockets with a Java client and a Python server?

I am trying to make two processes communicate through local sockets: a Python server and a Java client. The data I want to pass between both consists of the bytes of a Protobuf object, with variable size. I want the connection to remain open and be used until the end of the program, because I'm passing a lot of objects that need to be processed.
Because Protobuf objects have variable size, I am sending the size of the message/response before sending the true message/response containing the object.
Currently, I am using a TCPServer from the socketserver library on the Python side. I have the following handler implemented:
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The request handler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def recv_all(self, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = self.request.recv(n - len(data))
if not packet:
return None
data += packet
return data
def handle(self):
logger.debug("Beginning of handle cycle for client: {}.".format(self.client_address))
while True:
if True: # please disregard this if condition
# Receive 4 bytes (1 int) denoting the size of the message
data_length_bytes: bytes = self.recv_all(4)
logger.debug('Received data_length: {}'.format(data_length_bytes))
# If recv read an empty request b'', then client has closed the connection
if not data_length_bytes:
break
data_length: int = int.from_bytes(data_length_bytes.strip(), byteorder='big')
data: bytes = self.recv_all(data_length).strip()
response: bytes = data.upper()
# Send length of response first
self.request.sendall(len(response).to_bytes(4, byteorder='big'))
# Send response
self.request.sendall(response)
logger.debug(
'Sent response to: {}. Size of response: {} bytes. Response: {}.'.format(self.client_address,
len(response),
response))
logger.debug("End of handle cycle for client: {}.".format(self.client_address))
And the following client:
class SocketClient
{
private static Socket socket;
private int port;
private DataOutputStream out;
private DataInputStream in;
SocketClient(int port)
{
this.port = port;
this.createSocket();
}
private void createSocket() {
InetAddress address;
try {
address = InetAddress.getByName("localhost");
socket = new Socket(address, port);
this.out = new DataOutputStream(socket.getOutputStream());
this.in = new DataInputStream(socket.getInputStream());
} catch (IOException e) {
e.printStackTrace();
}
}
byte[] sendMessageAndReceiveResponse(byte[] messageToSend){
try {
if(true) { // again, please disregard this condition
//Send the size of the message to the server
this.out.writeInt(messageToSend.length);
out.flush();
this.out.write(messageToSend);
out.flush();
//Get the response message from the server
int length = in.readInt(); // read length of incoming message
byte[] buffer = null;
if(length>=0) {
buffer = new byte[length];
in.readFully(buffer, 0, buffer.length); // read the message
}
return buffer;
}
}
catch (ConnectException exception) {
System.out.println("ATTENTION! Could not connect to socket. Nothing was retrieved from the Python module.");
exception.printStackTrace();
return null;
}
catch (Exception exception)
{
exception.printStackTrace();
return null;
}
}
void close(){
//Closing the socket
try
{
in.close();
out.close();
socket.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
I run the following experiment after starting the Python server:
SocketClient socketClient = new SocketClient(5000);
byte[] response;
// Case 1
response = socketClient.sendMessageAndReceiveResponse("12345678".getBytes());
System.out.println(new String(response));
// Case 2
response = socketClient.sendMessageAndReceiveResponse("123456781".getBytes());
System.out.println(new String(response));
// Case 3
response = socketClient.sendMessageAndReceiveResponse("12345678123456781".getBytes());
System.out.println(new String(response));
socketClient.close();
Case 1 and case 3 work well. However, when I run case 2, on the Python server side, I get the following log:
DEBUG -- [handle()] Received data_length: b'\x00\x00\x00\t' # The '\t' shouldn't be here. A '\x09' should.
And then the server throws and exception and exits the connection. This happens with every string with 8 < length < 14. What am I doing wrong, and is there an easier way to achieve what I want?

I figured out why I was having problems with messages of 8 < length < 14.
I was getting the \t character when length was equal to 9. I noticed that if I changed the length to 10, it would become \n. And to 13, \r. I realized that there wasn't any \t magically appearing. Python was for some reason converting \x09 to \t, because the horizontal tab character \t has an ASCII code equal to 9!
And when I applied the strip() function in this line:
data_length: int = int.from_bytes(data_length_bytes.strip(), byteorder='big')
, Python deleted my \t, which was actually my \x09. My problem was logging the value before stripping it, and so I took a long time to figure out my mistake.
Therefore the solution was to simply not use strip(). I leave here my current working code (at least for my tests), for someone to use:
Python server handler:
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The request handler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def recv_all(self, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = self.request.recv(n - len(data))
if not packet:
return None
data += packet
return data
def handle(self):
while True:
data_length_bytes: bytes = self.recv_all(4)
# If recv read an empty request b'', then client has closed the connection
if not data_length_bytes:
break
# DON'T DO strip() ON THE DATA_LENGTH PACKET. It might delete what Python thinks is whitespace but
# it actually is a byte that makes part of the integer.
data_length: int = int.from_bytes(data_length_bytes, byteorder='big')
# Don't do strip() on data either (be sure to check if there is some error if you do use)
data: bytes = self.recv_all(data_length)
response: bytes = data.upper()
self.request.sendall(len(response).to_bytes(4, byteorder='big'))
self.request.sendall(response)
The Java client remained the same, but without that if(true) condition that I was using for debug reasons.

Sockets -- Different bytes from python and java

I have been working on a networking project recently to revive a dead mmo game for personal learning, I have a python implementation which works to decode the game data using blowfish(pypi / pycryptodome) and would like to transfer this 'server' into a java project.
Initially using blowfish decryption in java (BouncyCastle and Cipher -- default) I was getting entirely different results between java and python. Through some research I found that java (along with most things) actually use blowfish-compat big endian.
This one python library seems to be the only one which correctly decodes the data. Next I decided to use a python asyncio server as a middle relay simply for encryption and decryption. The network flow now looks like this:
GameClient -> Java SocketServer -> Python server (decryption) -> Java SocketServer.
The original Python implementation results in these bytes in hex format:
32004a815f49367cc3691be26d7b668132506dc972d5a6bbad38299640c6e222c6e55096f50ff33711250675431633ca9ede
The Java implementation produces these results in hex format(using apache commons Hex.encodeHexString())
32004a815f49367cc3691be26d7b668132506dc972d5a6bbad38299640c6e222c6e5c65830d65f9b4d60eb26730685f486d7
Both of these hex representations are pre-blowfish decryption in Python they are just the raw bytes being sent from the game client.
My question is why do these bytes start off the same and then it seems java trails off? The python results are the correct results they are tested and work. I have tried wrapping the bytes in java in a buffer and then calling flip() however this did not produce the correct results either. Using another stackoverflow post (I do not have link I am sorry) I tried casting this byte[] into a BigInteger and that also did not produce the correct results.
Any help is greatly appreciated.
Python Implementation
#!/usr/bin/env python3
import asyncio
import binascii
import blowfish
import ipaddress
import os
import struct
import sys
AUTH_BLOWFISHKEY = b"[;'.]94-31==-%&#!^+]\000"
bf = blowfish.Cipher(AUTH_BLOWFISHKEY, byte_order="little")
class EncryptionRelay(asyncio.Protocol):
def connection_made(self, transport):
self.transport = transport
self.client = (transport.get_extra_info('peername')[0] + ":" # IP
+ str(transport.get_extra_info('peername')[1])) # port
print("Connection from: " + self.client)
def connection_lost(self, exc):
print("Connection closed: " + self.client)
def data_received(self, data):
print(data.hex()) #python output above
pt = b''.join(bf.decrypt_ecb(data[2:]))
self.transport.write(pt)
def closeSocket(self, reason):
print(reason)
self.transport.close()
def main():
loop = asyncio.get_event_loop()
coroutine = loop.create_server(EncryptionRelay, host=None, port=54556)
server = loop.run_until_complete(coroutine)
for socket in server.sockets:
print("Listening on: " + socket.getsockname()[0] + ":" +
str(socket.getsockname()[1]))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
if __name__ == "__main__":
main()
Java Implementation
public AuthServer(int port) {
serverGUI = new AuthServerGUI(port);
try {
serverSocket = new ServerSocket(port);
relay = new PythonEncryptionRelay(this);
new Thread(relay).start();
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public void run() {
while(true) {
try {
Socket socket = serverSocket.accept();
onConnection(socket); //sends an init packet to client -- irrelevant to question
byte[] incomingData = new byte[0];
byte[] temp = new byte[1024];
int k = -1;
while((k = socket.getInputStream().read(temp, 0, temp.length)) > -1) {
byte[] tbuff = new byte[incomingData.length + k];
System.arraycopy(incomingData, 0, tbuff, 0, incomingData.length);
System.arraycopy(temp, 0, tbuff, incomingData.length, k);
incomingData = tbuff;
receiveData(socket, incomingData);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
public void receiveData(Socket socket, byte[] data) {
int lenLo = (int) (data[0]);
int lenHi = (int) (data[1]);
int length = lenHi * 256 + lenLo;
if(lenHi < 0) {
System.out.println("Invalid Packet Length");
}
if(data.length != length) {
System.out.println("Incomplete Packet Received");
}
serverGUI.serverDebug("DATA RECEIVED");
serverGUI.serverDebug(Hex.encodeHexString(data)); //this is the java ouput above serverGUI is simply a jframe i built no data manipulation
serverGUI.serverDebug("DATA_RECEIVED DONE");
this.relay.sendData(data); //this function sends the data from socket server to the python asyncio server
}
public void receiveDataFromPythonRelay(Socket socket, byte[] data) {
serverGUI.debugPythonRelay("DATA RECEIVED");
serverGUI.debugPythonRelay(Hex.encodeHexString(data)); //this will be the output from the python script aka data decrypted.
//The data byte[] is created in the exact same way the incomingData array is built in the AuthServer run function
serverGUI.debugPythonRelay("DATA_RECEIVED DONE");
}
Additionally the way I am importing the data byte[] from sockets is programmed as such because the client does not send endl therefore readLine will not work from the streams.

A byte has 8 bits which means you can have maximum 0xff as value.
But Java uses signed byte which means msb is reserved for signed bit. This leaves you with only 7 bits for your value, so you can have maximum as 0x7f stored in byte type of variable. Any number greater than 0x07f will result in overflow.
Try using an int array. As int uses 4 bytes (32 bits) hence there will always be space for 8 bits.
use byte[] to read from the stream and then copy the contents in int[] , Use int intArr[i] = byteArr[i] & 0xFF; for ith value from byte[] to avoid negative numbers due to byte overflow

How to send bytes from Python to Java using sockets?

I've read a few posts on how to send a picture using sockets in Python, and how to send a picture using sockets in Java, I was wanting to combine the two and send a picture from Python to Java using sockets on both ends. Most of my code is taken from the posts I read but here is the python client:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("192.168.0.12",4141))
try:
file = open("subbed.jpg", 'rb')
bytes = file.read()
print "{0:b}".format(len(bytes))
size = len(bytes)
s.sendall(size)
answer = s.recv(4096)
print "Answer = %s" %answer
if answer == 'GOT SIZE':
s.sendall(bytes)
answer = s.recv(4096)
if answer == 'GOT IMAGE' :
s.sendall("byte")
file.close()
finally:
s.close()
the code for the Java server is:
public static void main(String[] args) {
while(true) {
try (
ServerSocket server = new ServerSocket(PORT_NUMBER);
Socket client = server.accept();
PrintWriter out = new PrintWriter(client.getOutputStream(), true);
InputStream in = client.getInputStream()) {
System.out.println("GOT CONNECTION FROM: " + client.getInetAddress().toString());
byte[] sizeAr = new byte[4];
in.read(sizeAr);
int size = ByteBuffer.wrap(sizeAr).asIntBuffer().get();
System.out.println(Integer.toBinaryString(size));
out.println("GOT SIZE");
byte[] imageAr = new byte[size];
in.read(imageAr);
BufferedImage image = ImageIO.read(new ByteArrayInputStream(imageAr));
ImageIO.write(image, "jpg", new File("C:\\myprivatelocation\\test.jpg"));
} catch (Exception ioe) {
ioe.printStackTrace();
}
}
}
The initial problem comes from sending the size I think. I'm no python expert, nor am I a Java expert, but I think what's happening is Python is sending the size as a string and Java is receiving it as a byte array and converting it to an integer, and there are some differences in the way they are stored in the two languages. Can anyone offer any assistance with this issue?

Although I would approach your problem slightly differently, the following code works:
Python Sender
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect(("127.0.0.1", 8888))
with open("C:\\temp\\test-input.jpg", 'rb') as f:
content = f.read()
size = len(content)
print("File bytes:", size)
s.sendall(size.to_bytes(4, byteorder='big'))
buff = s.recv(4)
resp = int.from_bytes(buff, byteorder='big')
print("Response:", resp)
if size == resp:
s.sendall(content)
buff = s.recv(2)
print(buff)
print("Complete.")
Java Receiver
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.ByteBuffer;
import javax.imageio.ImageIO;
class Server{
public static void main(String[] args) {
int PORT_NUMBER = 8888;
try (
ServerSocket server = new ServerSocket(PORT_NUMBER);
Socket client = server.accept();
OutputStream sout = client.getOutputStream();
InputStream sin = client.getInputStream();
){
System.out.println("GOT CONNECTION FROM: " + client.getInetAddress().toString());
// Get length
byte[] size_buff = new byte[4];
sin.read(size_buff);
int size = ByteBuffer.wrap(size_buff).asIntBuffer().get();
System.out.format("Expecting %d bytes\n", size);
// Send it back (?)
sout.write(size_buff);
// Create Buffers
byte[] msg_buff = new byte[1024];
byte[] img_buff = new byte[size];
int img_offset = 0;
while(true) {
int bytes_read = sin.read(msg_buff, 0, msg_buff.length);
if(bytes_read == -1) { break; }
// Copy bytes into img_buff
System.arraycopy(msg_buff, 0, img_buff, img_offset, bytes_read);
img_offset += bytes_read;
System.out.format("Read %d / %d bytes...\n", img_offset, size);
if(img_offset >= size) { break; }
}
BufferedImage image = ImageIO.read(new ByteArrayInputStream(img_buff));
ImageIO.write(image, "jpg", new File("C:\\temp\\test-output.jpg"));
// Send "OK"
byte[] OK = new byte[] {0x4F, 0x4B};
sout.write(OK);
}
catch (IOException ioe) { ioe.printStackTrace(); }
}
}
The sender opens a socket, reads the file, and sends the receiver the length. The receiver gets the length, parses the bytes and sends it back. Upon receipt of the "confirmation", the sender then sends the file contents. The receiver will then repeatedly read 1024 byte chunks from the socket input stream, inserting the bytes into img_data. When there are no more bytes expected (or the socket is closed), the receiver will send "OK" to the sender (unconditionally) and exit. The sender will just print that "OK" (in bytes), and exit.
Some of this could be cleaned up with a ByteArrayOutputStream, but I wanted to get as close to the functionality of your code as possible.

Something is off - you should be getting some kind of error when trying to send some integer in the socket:
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(('localhost', 7777))
>>> s.sendall(len(b'some bytes'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: a bytes-like object is required, not 'int'
Sockets can only send bytes, you have to convert the int object containing the size to bytes somehow, python won't do it automatically for you. Your code should fail with the error above.
In the java code you are using asIntBuffer() and Integer.toBinaryString to convert your values, in the python code you just try to send the data without converting, you should get error.
Now, to convert the int to bytes, you can use the struct module; It will convert to a binary representation used by the C language - I think that's what your java code expect
size_in_bytes = struct.pack('I', len(data_to_send))
In the same way, you should use struct.unpack to convert the bytes back to a integer object. See the documentation for more details and a table of possible conversions.

BinaryWriter to MemoryStream in Java

I have C# program that makes a tcp connection with another c# program. In the c# program to send an message I did this :
private TcpClient client;
private void SendPulse()
{
byte[] send_Buffer;
port = 11000;
while (true)
{
lock (locked)
{
try
{
BlockID = 1003;
using (MemoryStream ms = new MemoryStream())
{
using (BinaryWriter w = new BinaryWriter(ms))
{
NetworkStream stream = client.GetStream();
BlockID = 1003;
LengthMessage = 84;
// Header :
w.Write(BeginMessage);
w.Write(BlockID);
w.Write(LengthMessage);
w.Write(RadarID);
w.Write(Time);
w.Write(ModeSystem);
w.Write(Icd_primary_var);
w.Write(Icd_secondary_ver);
// Data :
w.Write(StatusSystem);
send_Buffer = ms.ToArray();
stream.Write(send_Buffer, 0, send_Buffer.Length);
Thread.Sleep(3000); // Send pulse every 3 seconds.
}
}
}
catch
{
}
}
}
}
The idea is to write in binarywriter and than convert the memory we wrote on to byte array and to send it.
Now I have Java programming. I want to do it too, I have connection with C# but I dont know how to send it. I did DataOutputStream but it send every parameter alone, I want all in 1 array of bytes exactly like in the c# code.
Thanks for helpers.

If you want to use DataOutputStream, you can wrap it around a BufferedOutputStream and flush() it when you are done.
Or you can use an NIO ByteBuffer and write it to the socket.
To make the message easier to decode I would add the length to the start, unless you know it every message will be that length.