Develop Reference

Getting Garbage value while reading from serial port in java

I am reading data from serialport using jSerialComm package. I am getting following data
A total of 22 bytes are being received and I am getting these three garbage bytes too. The readable data is correct but what these garbage characters are happening?
Following is my code.
public static void main(String[] args) {
SerialPort serialPort = SerialPort.getCommPort("/dev/ttyUSB0");
if(serialPort.openPort())
{
System.out.println("Port Opened Successfully...");
}
else
{
System.out.println("Unable to open port....");
return;
}
serialPort.setComPortParameters(1200, 8, 1, 0);
try
{
while(true)
{
while(serialPort.bytesAvailable() != 0)
{
Thread.sleep(1000);
byte[] readBuffer = new byte[serialPort.bytesAvailable()];
int numRead = serialPort.readBytes(readBuffer, readBuffer.length);
String data = new String(readBuffer);
System.out.println("Read "+numRead+ " bytes." + readBuffer);
System.out.println(data);
}
}
}
catch(Exception e)
{
e.printStackTrace();
}
serialPort.closePort();
System.out.println("done...");
}

SerialPort.readBytes seems to be not thread safe, thus yielding "garbage" when called during character reception.
I suggest to use the snippet from the author's example which worked fine for me:
// Get a new instance of SerialPort by opening a port.
SerialPort port = SerialPort.open("COM2");
// Configure the connection
port.setTimeout(100);
port.setConfig(BaudRate.B115200, Parity.NONE, StopBits.ONE, DataBits.B8);
// You have the choice, you can either use the Java NIO channels
// or classic Input/Ouput streams to read and write data.
//DEL SerialChannel channel = port.getChannel();
InputStream istream = port.getInputStream();
// Read some data using a stream
byte[] byteBuffer = new byte[4096];
// Will timeout after 100ms, returning 0 if no bytes were available.
int n = istream.read(byteBuffer);
// *** Use n bytes of byteBuffer ***
//DEL ...
port.close();
//DEL : removed from the original code for clarity

You did not explain the protocol but I suggest to look it up. Presumably these are control characters or like a comment suggests binary data. You create a String from a byte buffer without an encoding so this also depends on your environment/ JVM's default encoding.
Try treating the first and the last two bytes as specified in the protocol in use for your project. It might also be related to jSerialComm not removing serial signalling, e.g. handshake, EOT and such.
If you're reverse-engineering the protocol instead maybe also try another library like RxTx to see if the bytes stay the same.
To inspect the bytes safely use for example BigInteger to print out a Hex-String instead:
BigInteger bigInteger = new BigInteger(1, bytes);
System.out.printf("%0" + (bytes.length << 1) + "x", bigInteger);

Use this code block and it will run perfectly;
serialPort.addDataListener(new SerialPortDataListener() {
#Override
public int getListeningEvents() {
return SerialPort.LISTENING_EVENT_DATA_RECEIVED;
}
#Override
public void serialEvent(SerialPortEvent serialPortEvent) {
if (serialPortEvent.getEventType() != SerialPort.LISTENING_EVENT_DATA_RECEIVED) {
return;
}
byte[] newData = serialPortEvent.getReceivedData();
String data = new String(newData);
System.out.println(data);
}
});

Python network bytes different than java slightly

I have been working on a networking project recently to revive a dead mmo game for personal learning, I have a python implementation which works to decode the game data using blowfish(pypi / pycryptodome) and would like to transfer this 'server' into a java project.
Initially using blowfish decryption in java (BouncyCastle and Cipher -- default) I was getting entirely different results between java and python. Through some research I found that java (along with most things) actually use blowfish-compat big endian.
This one python library seems to be the only one which correctly decodes the data. Next I decided to use a python asyncio server as a middle relay simply for encryption and decryption. The network flow now looks like this:
GameClient -> Java SocketServer -> Python server (decryption) -> Java SocketServer.
The original Python implementation results in these bytes in hex format:
32004a815f49367cc3691be26d7b668132506dc972d5a6bbad38299640c6e222c6e55096f50ff33711250675431633ca9ede
The Java implementation produces these results in hex format(using apache commons Hex.encodeHexString())
32004a815f49367cc3691be26d7b668132506dc972d5a6bbad38299640c6e222c6e5c65830d65f9b4d60eb26730685f486d7
Both of these hex representations are pre-blowfish decryption in Python they are just the raw bytes being sent from the game client.
My question is why do these bytes start off the same and then it seems java trails off? The python results are the correct results they are tested and work. I have tried wrapping the bytes in java in a buffer and then calling flip() however this did not produce the correct results either.
The code I am using is spread over multiple classes and files due to the server needing to be multithreaded to implement the python server as a middleman, but if I need to post my code to get a response I will happily edit and post what is needed. Any help is greatly appreciated
EDIT: CODE POST
Python Implementation
#!/usr/bin/env python3
import asyncio
import binascii
import blowfish
import ipaddress
import os
import struct
import sys
AUTH_BLOWFISHKEY = b"[;'.]94-31==-%&#!^+]\000"
bf = blowfish.Cipher(AUTH_BLOWFISHKEY, byte_order="little")
class EncryptionRelay(asyncio.Protocol):
def connection_made(self, transport):
self.transport = transport
self.client = (transport.get_extra_info('peername')[0] + ":" # IP
+ str(transport.get_extra_info('peername')[1])) # port
print("Connection from: " + self.client)
def connection_lost(self, exc):
print("Connection closed: " + self.client)
def data_received(self, data):
print(data.hex()) #python output above
pt = b''.join(bf.decrypt_ecb(data[2:]))
self.transport.write(pt)
def closeSocket(self, reason):
print(reason)
self.transport.close()
def main():
loop = asyncio.get_event_loop()
coroutine = loop.create_server(EncryptionRelay, host=None, port=54556)
server = loop.run_until_complete(coroutine)
for socket in server.sockets:
print("Listening on: " + socket.getsockname()[0] + ":" +
str(socket.getsockname()[1]))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
if __name__ == "__main__":
main()
Java Implementation
public AuthServer(int port) {
serverGUI = new AuthServerGUI(port);
try {
serverSocket = new ServerSocket(port);
relay = new PythonEncryptionRelay(this);
new Thread(relay).start();
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public void run() {
while(true) {
try {
Socket socket = serverSocket.accept();
onConnection(socket); //sends an init packet to client -- irrelevant to question
byte[] incomingData = new byte[0];
byte[] temp = new byte[1024];
int k = -1;
while((k = socket.getInputStream().read(temp, 0, temp.length)) > -1) {
byte[] tbuff = new byte[incomingData.length + k];
System.arraycopy(incomingData, 0, tbuff, 0, incomingData.length);
System.arraycopy(temp, 0, tbuff, incomingData.length, k);
incomingData = tbuff;
receiveData(socket, incomingData);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
public void receiveData(Socket socket, byte[] data) {
int lenLo = (int) (data[0]);
int lenHi = (int) (data[1]);
int length = lenHi * 256 + lenLo;
if(lenHi < 0) {
System.out.println("Invalid Packet Length");
}
if(data.length != length) {
System.out.println("Incomplete Packet Received");
}
serverGUI.serverDebug("DATA RECEIVED");
serverGUI.serverDebug(Hex.encodeHexString(data)); //this is the java ouput above serverGUI is simply a jframe i built no data manipulation
serverGUI.serverDebug("DATA_RECEIVED DONE");
this.relay.sendData(data); //this function sends the data from socket server to the python asyncio server
}
public void receiveDataFromPythonRelay(Socket socket, byte[] data) {
serverGUI.debugPythonRelay("DATA RECEIVED");
serverGUI.debugPythonRelay(Hex.encodeHexString(data)); //this will be the output from the python script aka data decrypted.
//The data byte[] is created in the exact same way the incomingData array is built in the AuthServer run function
serverGUI.debugPythonRelay("DATA_RECEIVED DONE");
}
Additionally the way I am importing the data byte[] from sockets is programmed as such because the client does not send endl therefore readLine will not work from the streams.

How to send bytes from Python to Java using sockets?

I've read a few posts on how to send a picture using sockets in Python, and how to send a picture using sockets in Java, I was wanting to combine the two and send a picture from Python to Java using sockets on both ends. Most of my code is taken from the posts I read but here is the python client:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("192.168.0.12",4141))
try:
file = open("subbed.jpg", 'rb')
bytes = file.read()
print "{0:b}".format(len(bytes))
size = len(bytes)
s.sendall(size)
answer = s.recv(4096)
print "Answer = %s" %answer
if answer == 'GOT SIZE':
s.sendall(bytes)
answer = s.recv(4096)
if answer == 'GOT IMAGE' :
s.sendall("byte")
file.close()
finally:
s.close()
the code for the Java server is:
public static void main(String[] args) {
while(true) {
try (
ServerSocket server = new ServerSocket(PORT_NUMBER);
Socket client = server.accept();
PrintWriter out = new PrintWriter(client.getOutputStream(), true);
InputStream in = client.getInputStream()) {
System.out.println("GOT CONNECTION FROM: " + client.getInetAddress().toString());
byte[] sizeAr = new byte[4];
in.read(sizeAr);
int size = ByteBuffer.wrap(sizeAr).asIntBuffer().get();
System.out.println(Integer.toBinaryString(size));
out.println("GOT SIZE");
byte[] imageAr = new byte[size];
in.read(imageAr);
BufferedImage image = ImageIO.read(new ByteArrayInputStream(imageAr));
ImageIO.write(image, "jpg", new File("C:\\myprivatelocation\\test.jpg"));
} catch (Exception ioe) {
ioe.printStackTrace();
}
}
}
The initial problem comes from sending the size I think. I'm no python expert, nor am I a Java expert, but I think what's happening is Python is sending the size as a string and Java is receiving it as a byte array and converting it to an integer, and there are some differences in the way they are stored in the two languages. Can anyone offer any assistance with this issue?

Although I would approach your problem slightly differently, the following code works:
Python Sender
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect(("127.0.0.1", 8888))
with open("C:\\temp\\test-input.jpg", 'rb') as f:
content = f.read()
size = len(content)
print("File bytes:", size)
s.sendall(size.to_bytes(4, byteorder='big'))
buff = s.recv(4)
resp = int.from_bytes(buff, byteorder='big')
print("Response:", resp)
if size == resp:
s.sendall(content)
buff = s.recv(2)
print(buff)
print("Complete.")
Java Receiver
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.ByteBuffer;
import javax.imageio.ImageIO;
class Server{
public static void main(String[] args) {
int PORT_NUMBER = 8888;
try (
ServerSocket server = new ServerSocket(PORT_NUMBER);
Socket client = server.accept();
OutputStream sout = client.getOutputStream();
InputStream sin = client.getInputStream();
){
System.out.println("GOT CONNECTION FROM: " + client.getInetAddress().toString());
// Get length
byte[] size_buff = new byte[4];
sin.read(size_buff);
int size = ByteBuffer.wrap(size_buff).asIntBuffer().get();
System.out.format("Expecting %d bytes\n", size);
// Send it back (?)
sout.write(size_buff);
// Create Buffers
byte[] msg_buff = new byte[1024];
byte[] img_buff = new byte[size];
int img_offset = 0;
while(true) {
int bytes_read = sin.read(msg_buff, 0, msg_buff.length);
if(bytes_read == -1) { break; }
// Copy bytes into img_buff
System.arraycopy(msg_buff, 0, img_buff, img_offset, bytes_read);
img_offset += bytes_read;
System.out.format("Read %d / %d bytes...\n", img_offset, size);
if(img_offset >= size) { break; }
}
BufferedImage image = ImageIO.read(new ByteArrayInputStream(img_buff));
ImageIO.write(image, "jpg", new File("C:\\temp\\test-output.jpg"));
// Send "OK"
byte[] OK = new byte[] {0x4F, 0x4B};
sout.write(OK);
}
catch (IOException ioe) { ioe.printStackTrace(); }
}
}
The sender opens a socket, reads the file, and sends the receiver the length. The receiver gets the length, parses the bytes and sends it back. Upon receipt of the "confirmation", the sender then sends the file contents. The receiver will then repeatedly read 1024 byte chunks from the socket input stream, inserting the bytes into img_data. When there are no more bytes expected (or the socket is closed), the receiver will send "OK" to the sender (unconditionally) and exit. The sender will just print that "OK" (in bytes), and exit.
Some of this could be cleaned up with a ByteArrayOutputStream, but I wanted to get as close to the functionality of your code as possible.

Something is off - you should be getting some kind of error when trying to send some integer in the socket:
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(('localhost', 7777))
>>> s.sendall(len(b'some bytes'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: a bytes-like object is required, not 'int'
Sockets can only send bytes, you have to convert the int object containing the size to bytes somehow, python won't do it automatically for you. Your code should fail with the error above.
In the java code you are using asIntBuffer() and Integer.toBinaryString to convert your values, in the python code you just try to send the data without converting, you should get error.
Now, to convert the int to bytes, you can use the struct module; It will convert to a binary representation used by the C language - I think that's what your java code expect
size_in_bytes = struct.pack('I', len(data_to_send))
In the same way, you should use struct.unpack to convert the bytes back to a integer object. See the documentation for more details and a table of possible conversions.

UDP Packet received okay in Java but corrupted in Python

I am trying to record audio from an Android tablet and send it to a python server. At the start of the byte packet, I include some relevant information about the state of the Android app (A byte array called "actives" -- but considering it's receiving fine by a Java server, this should not be relevant). The android code is as follows:
int read = recorder.read(buffer, 0, buffer.length);
for (int a = 0; a < actives.length; a++) {
outBuffer[a+1] = (byte)actives[a];
logger = logger + Byte.toString(actives[a]) + ",";
}
int furthest=0;
for(int a =0; a < buffer.length; a++){
outBuffer[actives.length+1+a]=buffer[a];
if(buffer[a]!=0)furthest=a;
}
packet = new DatagramPacket(outBuffer, read,
serverAddress, PORT);
Log.d("writing", logger+Byte.toString(outBuffer[7])+".length"+Integer.toString(1+furthest+actives.length+1));
Log.d("streamer","Packet length "+outBuffer.length);
try {
socket.send(packet);
}catch (IOException e){
Log.e("streamer", "Exception: " + e);
}
Log.d("streamer","packetSent");
I receive a clean signal on the other end using a Java server.
Image of received java output: !(http://i.imgur.com/31UWzya.png)
This is my Java server:
DatagramSocket serverSocket = new DatagramSocket(3001);
int byteSize=970;
byte[] receiveData = new byte[byteSize];
DatagramPacket receivePacket = new DatagramPacket(receiveData,
receiveData.length);
while(true){ // recieve data until timeout
try {
serverSocket.receive(receivePacket);
String rcvd = "rcvd from " + receivePacket.getAddress();
System.out.println("receiver"+"Received a packet!" +rcvd);
break;
}
catch (Exception e) {
// timeout exception.
System.out.println("Timeout reached without packet!!! " + e);
timeoutReached=true;
break;
}
}
if(timeoutReached)continue;
currTime = System.currentTimeMillis();
data = receivePacket.getData();
Here is my Python server's output:
!(http://i.imgur.com/RYkcCCE.png)
And here is the code:
import socket
ip="192.ip.address"
port=3001;
sock=socket.socket(socket.AF_INET,socket.SOCK_DGRAM);
sock.bind(('',port));
while(True):
data,addr=sock.recvfrom(970);
print("address",addr);
print("received a data!");
print(data);
In the last line of the python script, I have tried to change "print(data)" to "print(data.decode())", in which case I get this error:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)
I am not running these servers at the same time
My guess is that it has to do something with Java using unsigned ints and python not doing that. Is there a way in Python that I can convert this data, because data.decode() is not working? Alternatively I should be able to convert the data in Java somehow? None of the answers on stackoverflow that I have tried have worked.

Decoding is the right approach. In your android app explicitly mention the character encoding. UTF-8 is the standard Charset that is used.
Your log is pretty clear. You are trying to decode the data packet as ASCII (which is the default encoding of the decode() function) but I'm guessing its ISO_8859_1 or UTF-8 (more likely).
Next try data.decode('utf8', 'ignore') in your android app. Note: 'ignore' is an optional argument and to be used only in case of debugging as it will ignore malformed(corrupted) data and try to convert individual characters. If you want to use decode() in production use 'strict' or no second argument ('strict' is the default).
In place of 'utf8' try other options from other Python Encodings.

This was pretty brutal to attack head-on. I tried specifying the encoding in Java (before sending) like another SO post suggested, but that didn't help. So I side-stepped the problem by converting my Android byte array into a comma-separated string, then converting the string back into UTF-8 bytes.
sendString="";
for(int a =0; a < buffer.length; a++){
sendString=sendString+Byte.toString(buffer[a])+",";
}
byte[] outBuffer = sendString.getBytes("UTF-8");
Make sure you reset your string to null ("") each time you go through the while loop, or your ish will get very slow af.
Then in Python,right after receiving:
data=data.decode("utf8");
Although I am stringifying 980 characters, it does not appear to add much to the processing time... although I do wish that I could send the raw bytes, as speed is very important to me here. I'll leave the question open in case someone can come up with a better solution.

Converting Python Networking Code To Java

Given the following function definitions in Python:
class NetworkClient:
def __init__(self, ip):
self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP)
self.s.connect((str(ip), 7331))
def readmem(self, address, length):
if length == 0: raise BaseException, "Why are you giving me no length" #Please don't do this to me
if not self.ValidMemory().validrange(address, length): return
if not self.ValidMemory().validaccess(address, length, "read"): return
self.s.send("\x04") #cmd_readmem
request = struct.pack(">II", address, address + length)
self.s.send(request)
status = self.s.recv(1)
if status == "\xbd": #Non-zero memory was found
response = self.s.recv(length)
elif status == "\xb0": #All zeroes
response = "\x00" * length
else: return #Something went terribly wrong
return response
I now want to do the same in Java. As example I want to read the memory at address 0x10000003 with length 0x1 e.g. 1 byte.
So far I wrote the following code:
import java.io.*;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
public class NetworkClient
{
public static void main(String[] arguments) throws IOException, InterruptedException
{
try (Socket clientSocket = new Socket("192.168.178.35", 7331);
PrintWriter outputWriter =
new PrintWriter(clientSocket.getOutputStream(), true);
BufferedReader inputReader =
new BufferedReader(
new InputStreamReader(clientSocket.getInputStream())))
{
System.out.println("Connected");
byte readMemoryCommand = (byte) 4;
int memoryAddress = 0x10000003;
int length = 0x1;
outputWriter.print(readMemoryCommand);
outputWriter.print(memoryAddress);
outputWriter.println(memoryAddress + length);
System.out.println("Receiving");
System.out.println(inputReader.read());
System.out.println("Received");
}
}
public static byte[] toByteArray(String letters)
{
return letters.getBytes(StandardCharsets.US_ASCII);
}
}
Establishing the connection works but then after sending the data, there is no response. The server does not send anything back and my Java program is stuck on readLine(). The output is as follows:
Connected
Receiving
Replacing readLine() with read() does not help. Using a DataOutputStream with write and writeInt calls does not work either.
The difficulty I'm having here seems to be to prepare and send the request data correctly. For your information, the Python function struct.pack produces the following result in an interactive Python session:
>>> import struct
>>> struct.pack(">II", 10000003, 10000004)
b'\x00\x98\x96\x83\x00\x98\x96\x84'
>>>
Both integers are converted to hexadecimal and "packed" into a byte array.
The readmem command String is converted to an ASCII byte array I believe:
>>> print("\x04")
*Box with questionmark in it*
>>> s = "04"
>>> ":".join("{:02x}".format(ord(c)) for c in s)
'30:34'
>>>
The latter code for printing the bytes is taken from here.
So what is wrong with the Java code?

You should be using DataOutputStream instead of PrintWriter. Pay close attention to the documentation—the write and writeByte methods send one byte, while writeInt sends four bytes. Similarly, reading should be done with a DataInputStream, specifically the readByte and readFully methods, rather than BufferedReader.
Writers and Readers deal with text, not bytes. PrintWriter.print sends an ASCII representation of a number, rather than the byte itself. BufferedReader (and most Readers) will try to decode the incoming bytes into text, instead of reading the bytes directly.