Develop Reference

Sockets -- Different bytes from python and java

I have been working on a networking project recently to revive a dead mmo game for personal learning, I have a python implementation which works to decode the game data using blowfish(pypi / pycryptodome) and would like to transfer this 'server' into a java project.
Initially using blowfish decryption in java (BouncyCastle and Cipher -- default) I was getting entirely different results between java and python. Through some research I found that java (along with most things) actually use blowfish-compat big endian.
This one python library seems to be the only one which correctly decodes the data. Next I decided to use a python asyncio server as a middle relay simply for encryption and decryption. The network flow now looks like this:
GameClient -> Java SocketServer -> Python server (decryption) -> Java SocketServer.
The original Python implementation results in these bytes in hex format:
32004a815f49367cc3691be26d7b668132506dc972d5a6bbad38299640c6e222c6e55096f50ff33711250675431633ca9ede
The Java implementation produces these results in hex format(using apache commons Hex.encodeHexString())
32004a815f49367cc3691be26d7b668132506dc972d5a6bbad38299640c6e222c6e5c65830d65f9b4d60eb26730685f486d7
Both of these hex representations are pre-blowfish decryption in Python they are just the raw bytes being sent from the game client.
My question is why do these bytes start off the same and then it seems java trails off? The python results are the correct results they are tested and work. I have tried wrapping the bytes in java in a buffer and then calling flip() however this did not produce the correct results either. Using another stackoverflow post (I do not have link I am sorry) I tried casting this byte[] into a BigInteger and that also did not produce the correct results.
Any help is greatly appreciated.
Python Implementation
#!/usr/bin/env python3
import asyncio
import binascii
import blowfish
import ipaddress
import os
import struct
import sys
AUTH_BLOWFISHKEY = b"[;'.]94-31==-%&#!^+]\000"
bf = blowfish.Cipher(AUTH_BLOWFISHKEY, byte_order="little")
class EncryptionRelay(asyncio.Protocol):
def connection_made(self, transport):
self.transport = transport
self.client = (transport.get_extra_info('peername')[0] + ":" # IP
+ str(transport.get_extra_info('peername')[1])) # port
print("Connection from: " + self.client)
def connection_lost(self, exc):
print("Connection closed: " + self.client)
def data_received(self, data):
print(data.hex()) #python output above
pt = b''.join(bf.decrypt_ecb(data[2:]))
self.transport.write(pt)
def closeSocket(self, reason):
print(reason)
self.transport.close()
def main():
loop = asyncio.get_event_loop()
coroutine = loop.create_server(EncryptionRelay, host=None, port=54556)
server = loop.run_until_complete(coroutine)
for socket in server.sockets:
print("Listening on: " + socket.getsockname()[0] + ":" +
str(socket.getsockname()[1]))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
if __name__ == "__main__":
main()
Java Implementation
public AuthServer(int port) {
serverGUI = new AuthServerGUI(port);
try {
serverSocket = new ServerSocket(port);
relay = new PythonEncryptionRelay(this);
new Thread(relay).start();
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public void run() {
while(true) {
try {
Socket socket = serverSocket.accept();
onConnection(socket); //sends an init packet to client -- irrelevant to question
byte[] incomingData = new byte[0];
byte[] temp = new byte[1024];
int k = -1;
while((k = socket.getInputStream().read(temp, 0, temp.length)) > -1) {
byte[] tbuff = new byte[incomingData.length + k];
System.arraycopy(incomingData, 0, tbuff, 0, incomingData.length);
System.arraycopy(temp, 0, tbuff, incomingData.length, k);
incomingData = tbuff;
receiveData(socket, incomingData);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
public void receiveData(Socket socket, byte[] data) {
int lenLo = (int) (data[0]);
int lenHi = (int) (data[1]);
int length = lenHi * 256 + lenLo;
if(lenHi < 0) {
System.out.println("Invalid Packet Length");
}
if(data.length != length) {
System.out.println("Incomplete Packet Received");
}
serverGUI.serverDebug("DATA RECEIVED");
serverGUI.serverDebug(Hex.encodeHexString(data)); //this is the java ouput above serverGUI is simply a jframe i built no data manipulation
serverGUI.serverDebug("DATA_RECEIVED DONE");
this.relay.sendData(data); //this function sends the data from socket server to the python asyncio server
}
public void receiveDataFromPythonRelay(Socket socket, byte[] data) {
serverGUI.debugPythonRelay("DATA RECEIVED");
serverGUI.debugPythonRelay(Hex.encodeHexString(data)); //this will be the output from the python script aka data decrypted.
//The data byte[] is created in the exact same way the incomingData array is built in the AuthServer run function
serverGUI.debugPythonRelay("DATA_RECEIVED DONE");
}
Additionally the way I am importing the data byte[] from sockets is programmed as such because the client does not send endl therefore readLine will not work from the streams.

A byte has 8 bits which means you can have maximum 0xff as value.
But Java uses signed byte which means msb is reserved for signed bit. This leaves you with only 7 bits for your value, so you can have maximum as 0x7f stored in byte type of variable. Any number greater than 0x07f will result in overflow.
Try using an int array. As int uses 4 bytes (32 bits) hence there will always be space for 8 bits.
use byte[] to read from the stream and then copy the contents in int[] , Use int intArr[i] = byteArr[i] & 0xFF; for ith value from byte[] to avoid negative numbers due to byte overflow

How to send bytes from Python to Java using sockets?

I've read a few posts on how to send a picture using sockets in Python, and how to send a picture using sockets in Java, I was wanting to combine the two and send a picture from Python to Java using sockets on both ends. Most of my code is taken from the posts I read but here is the python client:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("192.168.0.12",4141))
try:
file = open("subbed.jpg", 'rb')
bytes = file.read()
print "{0:b}".format(len(bytes))
size = len(bytes)
s.sendall(size)
answer = s.recv(4096)
print "Answer = %s" %answer
if answer == 'GOT SIZE':
s.sendall(bytes)
answer = s.recv(4096)
if answer == 'GOT IMAGE' :
s.sendall("byte")
file.close()
finally:
s.close()
the code for the Java server is:
public static void main(String[] args) {
while(true) {
try (
ServerSocket server = new ServerSocket(PORT_NUMBER);
Socket client = server.accept();
PrintWriter out = new PrintWriter(client.getOutputStream(), true);
InputStream in = client.getInputStream()) {
System.out.println("GOT CONNECTION FROM: " + client.getInetAddress().toString());
byte[] sizeAr = new byte[4];
in.read(sizeAr);
int size = ByteBuffer.wrap(sizeAr).asIntBuffer().get();
System.out.println(Integer.toBinaryString(size));
out.println("GOT SIZE");
byte[] imageAr = new byte[size];
in.read(imageAr);
BufferedImage image = ImageIO.read(new ByteArrayInputStream(imageAr));
ImageIO.write(image, "jpg", new File("C:\\myprivatelocation\\test.jpg"));
} catch (Exception ioe) {
ioe.printStackTrace();
}
}
}
The initial problem comes from sending the size I think. I'm no python expert, nor am I a Java expert, but I think what's happening is Python is sending the size as a string and Java is receiving it as a byte array and converting it to an integer, and there are some differences in the way they are stored in the two languages. Can anyone offer any assistance with this issue?

Although I would approach your problem slightly differently, the following code works:
Python Sender
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect(("127.0.0.1", 8888))
with open("C:\\temp\\test-input.jpg", 'rb') as f:
content = f.read()
size = len(content)
print("File bytes:", size)
s.sendall(size.to_bytes(4, byteorder='big'))
buff = s.recv(4)
resp = int.from_bytes(buff, byteorder='big')
print("Response:", resp)
if size == resp:
s.sendall(content)
buff = s.recv(2)
print(buff)
print("Complete.")
Java Receiver
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.ByteBuffer;
import javax.imageio.ImageIO;
class Server{
public static void main(String[] args) {
int PORT_NUMBER = 8888;
try (
ServerSocket server = new ServerSocket(PORT_NUMBER);
Socket client = server.accept();
OutputStream sout = client.getOutputStream();
InputStream sin = client.getInputStream();
){
System.out.println("GOT CONNECTION FROM: " + client.getInetAddress().toString());
// Get length
byte[] size_buff = new byte[4];
sin.read(size_buff);
int size = ByteBuffer.wrap(size_buff).asIntBuffer().get();
System.out.format("Expecting %d bytes\n", size);
// Send it back (?)
sout.write(size_buff);
// Create Buffers
byte[] msg_buff = new byte[1024];
byte[] img_buff = new byte[size];
int img_offset = 0;
while(true) {
int bytes_read = sin.read(msg_buff, 0, msg_buff.length);
if(bytes_read == -1) { break; }
// Copy bytes into img_buff
System.arraycopy(msg_buff, 0, img_buff, img_offset, bytes_read);
img_offset += bytes_read;
System.out.format("Read %d / %d bytes...\n", img_offset, size);
if(img_offset >= size) { break; }
}
BufferedImage image = ImageIO.read(new ByteArrayInputStream(img_buff));
ImageIO.write(image, "jpg", new File("C:\\temp\\test-output.jpg"));
// Send "OK"
byte[] OK = new byte[] {0x4F, 0x4B};
sout.write(OK);
}
catch (IOException ioe) { ioe.printStackTrace(); }
}
}
The sender opens a socket, reads the file, and sends the receiver the length. The receiver gets the length, parses the bytes and sends it back. Upon receipt of the "confirmation", the sender then sends the file contents. The receiver will then repeatedly read 1024 byte chunks from the socket input stream, inserting the bytes into img_data. When there are no more bytes expected (or the socket is closed), the receiver will send "OK" to the sender (unconditionally) and exit. The sender will just print that "OK" (in bytes), and exit.
Some of this could be cleaned up with a ByteArrayOutputStream, but I wanted to get as close to the functionality of your code as possible.

Something is off - you should be getting some kind of error when trying to send some integer in the socket:
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(('localhost', 7777))
>>> s.sendall(len(b'some bytes'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: a bytes-like object is required, not 'int'
Sockets can only send bytes, you have to convert the int object containing the size to bytes somehow, python won't do it automatically for you. Your code should fail with the error above.
In the java code you are using asIntBuffer() and Integer.toBinaryString to convert your values, in the python code you just try to send the data without converting, you should get error.
Now, to convert the int to bytes, you can use the struct module; It will convert to a binary representation used by the C language - I think that's what your java code expect
size_in_bytes = struct.pack('I', len(data_to_send))
In the same way, you should use struct.unpack to convert the bytes back to a integer object. See the documentation for more details and a table of possible conversions.

Java DataOutputStream doesnt write all messages ( TCP Socket )

in my Client-Server application I found a strange error. I got the following Methods :
sendLoginResponse();
sendPlayerList();
sendCurrentLevelState();
Each Methods sends a byte array to the Client-Side
If I only call 2 of them, everything works fine and the Client-Side gets all the sent byte arrays. But if I call all 3 of them only the first and second one arrive the Client, the order of the following methods doesnt matter. but the Server says all of them were send. To write to the Client iam using the
write(byte[] b, int off, int len); method
all The lenghts within the packages make sense too.
Here comes the strange point:
if I add a Thread.sleep(1000); after the second Method, the third one does now arrive the Client after the sleep. I Have also tried to flush the DataOutputStream after every write call, but this didnt help.
EDIT:
So let's say I'd send 3 Login-Response
The Method's that gives me the byte[]:
public byte[] getLoginResponse(int playerID){
byte[] msg = new byte[4];
short shortMsgLength = 4;
byte[] msgLength = shortToBytes(shortMsgLength);
msg[0] = 2;
msg[1] = msgLength[0];
msg[2] = msgLength[1];
msg[3] = (byte) playerID;
return msg;
}
private byte[] shortToBytes(short value) {
byte[] returnByteArray = new byte[2];
returnByteArray[0] = (byte) (value & 0xff);
returnByteArray[1] = (byte) ((value >>> 8) & 0xff);
return returnByteArray;
}
And the Send Method:
private void sendLoginResponse() {
try{
byte[] msg = rfcObject.getLoginResponse(playerID);
out.write(msg,0,msg.length);
}catch(Exception e){
System.err.println(e.getMessage());
System.exit(0);
}
}
So if I call the sendLoginResponse(); three times in a row, the client only recieves 2 byte-arrays, but the server says it has been sent 3 times. If i add a
Thread.sleep(1000); `after the second Method-Call, everything works fine..`
The Client that reads the message runs in a Thread:
public void run(){
while(true){
try {
byte[] data = new byte[MAX_DATA_SIZE]; // MAX_DATA = 255
byteCount = in.read(data);
} catch (IOException ex) {
handleExceptionError(ex);
}
}
}
thank you!

if I call the sendLoginResponse(); three times in a row, the client only recieves 2 byte-arrays, but the server says it has been sent 3 times.
This is because TCP is a stream-oriented protocol. Meaning it doesn't know or care how your messages are delimited. There's no concept of individual messages in TCP, just a stream of bytes, with the guarantee that the order of bytes is preserved.
So when the sender calls three write, the three byte arrays are simply concatenated over the connection and arrives at the receiver in the same order, but the receiver doesn't necessarily need three read to get all the bytes, and even if it does take three read, the read doesn't necessarily gives you the same byte array passed to each corresponding write.
Your message already have the necessary information to get the individual message back from the byte stream:
// Client code for reading individual messages from a TCP connection
byte type = din.readByte();
// Read the message length, little-endian.
// We cannot use din.readShort because it's big-endian
int lenLo = din.read();
int lenHi = din.read();
short len = (short)(lenLo | (lenHi << 8));
byte [] body = new byte[len];
din.readFully(body);

DataOutputStream and TCP don't lose data.
As almost invariable seen in questions of this nature, the problem is at the receiving end. You are probably assuming that `read()' fills the buffer, and ignoring the count that it returns.
Based on your protocol description in comments, you should be using DataInputStream.readFully() in this circumstance:
byte type = din,readByte();
int length = din.readShort();
byte[] data = new byte[length];
din.readFully(data);

Reading all string from a server socket which has not been closed

I have a problem with socket programming in Java.
There is a server which has been written in python like this which I shouldn't not change.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send('from server\nnewline\0')
data = s.recv(BUFFER_SIZE)
s.close()
Now I want to write a code in Java which read the string from a server. Something like this:
public static String readStr(Socket client) throws IOException {
InputStreamReader inStream = new InputStreamReader(
client.getInputStream());
BufferedReader inBuff = new BufferedReader(inStream);
String answer = new String();
String str = inBuff.readLine();
while (str!=null) {
answer = answer.concat(str + "\n");
str = inBuff.readLine();
}
answer = answer.substring(0, answer.length() - 1);
System.out.println("answer:\n "+answer);
return answer;
}
But it seems that it blocks at line str = inBuff.readLine(); at the last line of the message. I tried the read() method but it was blocked too.

When designing a protocol over tcp, the best way is to include some kind of framer. This is done in the current protocol by the usage of a NUL byte.
When reading the data from the socket, you should first divide it into blocks/frames by some operations, before parsing the individual blocks.
A crude way to divide the packets into blocks is reading until you find a NUL byte, then returning that block as a byte array. (This is not the most efficient implementation)
public byte[] readPacket(InputStream in) throws IOException {
ByteArrayOutputStream tempStr = new ByteArrayOutputStream();
int read;
read=in.read();
while(read > 0){
tempStr.write(read);
read=in.read();
}
if(read == -1)
throw new EOFException();
return tempStr.toByteArray();
}
Because you now have proper frames for your data, you can now easily read the data:
byte[] frame = readPacket(in);
String[] lines = new String(frame, StandardCharsets.UTF8).split("\n");
// Do something with the lines

This is probably because the last line sent by the server does not end and readLine() method only returns when it reaches end of the line. Since you change the server's code. I recommend you use another method for reading from the stream. You may also use InputStreamReader class.

Apart from already mentioned inconsistent message/line ending - once with \n, second with \0, at the server there is no detection of end of the message. So the server will loop as long as the socket is not closed (or shut down for writing) at the client side. And as you have this line before closing the socket:
data = s.recv(BUFFER_SIZE)
in other words the client is waiting for some response from the server. But the server is stuck forever reading the message from the client in a loop.
So you need to either close the socket prior to that recv call or send some message (like empty line) and detect in on the server and eventually exit the loop.

how to send an array of bytes over a TCP connection (java programming)

Can somebody demonstrate how to send an array of bytes over a TCP connection from a sender program to a receiver program in Java.
byte[] myByteArray
(I'm new to Java programming, and can't seem to find an example of how to do this that shows both ends of the connection (sender and receiver.) If you know of an existing example, maybe you could post the link. (No need to reinvent the wheel.) P.S. This is NOT homework! :-)

The InputStream and OutputStream classes in Java natively handle byte arrays. The one thing you may want to add is the length at the beginning of the message so that the receiver knows how many bytes to expect. I typically like to offer a method that allows controlling which bytes in the byte array to send, much like the standard API.
Something like this:
private Socket socket;
public void sendBytes(byte[] myByteArray) throws IOException {
sendBytes(myByteArray, 0, myByteArray.length);
}
public void sendBytes(byte[] myByteArray, int start, int len) throws IOException {
if (len < 0)
throw new IllegalArgumentException("Negative length not allowed");
if (start < 0 || start >= myByteArray.length)
throw new IndexOutOfBoundsException("Out of bounds: " + start);
// Other checks if needed.
// May be better to save the streams in the support class;
// just like the socket variable.
OutputStream out = socket.getOutputStream();
DataOutputStream dos = new DataOutputStream(out);
dos.writeInt(len);
if (len > 0) {
dos.write(myByteArray, start, len);
}
}
EDIT: To add the receiving side:
public byte[] readBytes() throws IOException {
// Again, probably better to store these objects references in the support class
InputStream in = socket.getInputStream();
DataInputStream dis = new DataInputStream(in);
int len = dis.readInt();
byte[] data = new byte[len];
if (len > 0) {
dis.readFully(data);
}
return data;
}

Just start with this example from the Really Big Index. Notice though, that it's designed to transmit and receive characters, not bytes. This isn't a big deal, though - you can just deal with the raw InputStream and OutputStream objects that the Socket class provides. See the API for more info about the different types of readers, writers and streams. Methods you'll be interested in are OutputStream.write(byte[]) and InputStream.read(byte[]).

The Oracle Socket Communications Tutorial would seem to be the appropriate launch point.
Note that it's going to extra trouble to turn characters into bytes. If you want to work at the byte level, just peel that off.

This Sun Sockets tutorial should give you a good starting point

What you need to use is the write method of an java.io.OutputStream, and the read method of an java.io.InputStream, both of which you can retrieve from the Socket you open.

I would ask you to use ObjectOutputStream and ObjectInputStream. These send everything as an object and receive as the same.
ObjectOutputStream os = new ObjectOutputStream(socket.getOutputStream());
os.flush();
ObjectInputStream is = new ObjectInputStream(socket.getInputStream());
os.writeObject(byte_array_that_you_want_to_send);
byte[] temp = (byte[]) is.readObject();
Also remember first create the output stream, flush it and then go ahead with the input stream because if something left out in the stream the input stream wont be created.

I'm guessing that the question is worded incorrectly. I found this when searching for an answer to why my use of InputStream and OutputStream seemed to be setting the entire array to 0 upon encountering a byte of value 0. Do these assume that the bytes contain valid ASCII and not binary. Since the question doesn't come right out and ask this, and nobody else seems to have caught it as a possibility, I guess I'll have to satisfy my quest elsewhere.
What I was trying to do was write a TransparentSocket class that can instantiate either a TCP (Socket/ServerSocket) or a UDP (DatagramSocket) to use the DatagramPacket transparently. It works for UDP, but not (yet) for TCP.
Follow-up: I seem to have verified that these streams are themselves useless for binary transfers, but that they can be passed to a more programmer-friendly instantiation, e.g.,
new DataOutputStream(socket.getOutputStream()).writeInt(5);
^ So much for that idea. It writes data in a "portable" way, i.e., probably ASCII, which is no help at all, especially when emulating software over which I have no control!

import java.io.*;
import java.net.*;
public class ByteSocketClient
{
public static void main(String[] args) throws UnknownHostException, IOException
{
Socket s=new Socket("",6000);
DataOutputStream dout=new DataOutputStream(new BufferedOutputStream(s.getOutputStream()));
byte[] a = {(byte)0xC0,(byte)0xA8,(byte)0x01,(byte)0x02,(byte)0x53,(byte)0x4D,(byte)0x41,(byte)0x52,(byte)0x54};
dout.write(a);
dout.close();
s.close();
}
}

Here is an example that streams 100 byte wav file frames at a time.
private Socket socket;
public void streamWav(byte[] myByteArray, int start, int len) throws IOException {
Path path = Paths.get("path/to/file.wav");
byte[] data = Files.readAllBytes(path);
OutputStream out = socket.getOutputStream();
DataOutputStream os = new DataOutputStream(out);
os.writeInt(len);
if (len > 0) {
os.write(data, start, len);
}
}
public void readWav() throws IOException {
InputStream in = socket.getInputStream();
int frameLength = 100; // use useful number of bytes
int input;
boolean active = true;
while(active) {
byte[] frame = new byte[frameLength];
for(int i=0; i<frameLength; i++) {
input = in.read();
if(input < 0) {
active = false;
break;
} else frame[i] = (byte) input;
}
// playWavPiece(frame);
// streamed frame byte array is full
// use it here ;-)
}
}