Develop Reference

UDP client does not receive bytes

This question has been asked a lot, but so far, none of the solutions that I applied from previous answers have helped me.
Main goal
I am trying to learn UDP conexions and this is my attempt. I want to have a client ask for a picture at a server via UDP and the server will send it. Then the client will create a file with that information given.
Explanation
My main idea is to ask the server for an image using a "GET" command (not the HTTP, just GET) followed by the name of the image(extension included). Then the client awaits an answer which is the image requested.
Problems
The client waits and answer which does no come
Research
From another similar question it was a problem that I was using the same PORT for both receive and connect, so I added two ports, receivingPORT and sendingPORT, no results from the Client.
From other similar questions, It was a Firewall problem. So, on a Win10 machine, I created a new rule for UDP in the Firewall for the ports that I am using for this application, and nothing was received by the Client...
I have checked that the image is loaded into byte[] and the image is sent. But on the Client, nothing is received and stays there waiting for a connection to come through
CODE from Server
public class UDPserver {
static DatagramSocket serverUDP;
static DatagramPacket packet;
static InetAddress address;
static byte[] buffer = new byte[65507];//65507
final static int receivingPORT = 6668;
final static int sendingPORT = 6669;
public static void main(String[] args) throws SocketException, IOException, InterruptedException{
boolean serverActive = true;
String order = "";
String file = "";
//Instantiate server
serverUDP = new DatagramSocket(receivingPORT);
while(serverActive){
//Kind of packet we want to receive
packet = new DatagramPacket(buffer, buffer.length);
System.out.println("Server awaiting connection...");
//Receive it
serverUDP.receive(packet);
System.out.println("Received packet from: " + packet.getAddress() + "/" + packet.getPort());
//What does the packet contain?
String msg = new String(packet.getData());
address = packet.getAddress();
System.out.println("Order from: " + address + "/" + receivingPORT + " says: " + msg);
try{
order = msg.split(" ")[0].trim();
file = msg.split(" ")[1].trim();
} catch (Exception e){
}
switch(order){
case("GET"):{
System.out.println("Sending back an image...");
buffer = loadImageFromServer(file);
packet = new DatagramPacket(buffer, buffer.length, address, sendingPORT);
Thread.sleep(5000);
serverUDP.send(packet);
System.out.println("Client served");
break;
}
case("DISCONNECT"):{
buffer = "Server is disconnecting...".getBytes();
packet = new DatagramPacket(buffer, buffer.length, address, sendingPORT);
serverUDP.send(packet);
serverActive = false;
serverUDP.close();
break;
}
}
}
}
static byte[] loadImageFromServer(String path) {
try {
System.out.println("Loading path: " + path);
//Instantiate a buffer from the image for it
BufferedImage img = ImageIO.read(UDPserver.class.getResource(path));
//Create a byte[] stream object to handle the data
ByteArrayOutputStream baos = new ByteArrayOutputStream();
//Write the image data into those above with jpg format
ImageIO.write(img, "png", baos);
//Flush the information
baos.flush();
byte[] buffer = baos.toByteArray(); //Write it out on a byte string and return it
return buffer;
} catch (IOException ex) {
Logger.getLogger(UDPserver.class.getName()).log(Level.SEVERE, null, ex.fillInStackTrace());
System.exit(-1);
}
return null;
}
}
CODE client
public class Client {
static DatagramSocket clientUDP;
static InetAddress address;
static DatagramPacket packetSend;
static DatagramPacket packetReceive;
static int SIZE = 65507;
final static int receivingPORT = 6669;
final static int sendingPORT = 6668;
static byte[] buffer = new byte[SIZE];
static Scanner scan = new Scanner(System.in);
public static void main(String[] args) throws SocketException, UnknownHostException, IOException{
boolean clientLoop = true;
//Get address
address = InetAddress.getByName("localhost");
//Instantiate Client -> UDP
clientUDP = new DatagramSocket();
while(clientLoop){
System.out.print("Enter any key and press enter");
scan.next(); //Just to stop the loop
//Load the buffer
buffer = "GET imagenServidor.png".getBytes();
//buffer = "DISCONNECT".getBytes();
System.out.println("Buffer is ready");
//Arm the packet
packetSend = new DatagramPacket(buffer, buffer.length, address, sendingPORT);
System.out.println("Packet is armed!");
//Send the packet to the server
clientUDP.send(packetSend);
System.out.println("Order sent to server");
System.out.println("Waiting an answer");
packetReceive = new DatagramPacket(buffer, buffer.length, address, receivingPORT);
clientUDP.receive(packetReceive);
System.out.println("Server answered!");
ByteArrayInputStream bais = new ByteArrayInputStream(packetReceive.getData());
BufferedImage image = ImageIO.read(bais);
System.out.println(image);
}
clientUDP.close();
}
}
NOTES
This is a UDP exercise

The Reason
MTU!
You are sending packets with long buffe through UDP directly, which may not work in most network circumstances.
A packet sent through UDP should not be longer than the network MTU, otherwise it would be dropped. The network MTU may not be more than 1500 on most net nods(routers/switchs/hosts...), and even smaller sometimes. Though some nods may do sigmentation for ip packets, but you should not count on it when you are using UDP.
Suggestions
Use TCP instead in this application, as for:
You are sending data which expected to be complete (otherwise it would be useless).
You do not care about congestion control algorithms.
So just go with TCP.
Edit Based on The Update of The Question
So, as this is an excercise, in which you have to use UDP only.
As a file might be useless unless it is complete, you have to make sure:
All packets are possible to pass the path. Which means network should be connected both physically and virtually, and packet size should always be smaller than the MTU.
If any packets are lost, both the receiver and the sender should be able to know.
If any apckets come out of order, the receiver should be able to know.
Sender should be able to cache and resend the packets which are not confirmed by the receiver yet.
Make sure your have a good network connection. Split the image buffer into buffer array with each buffer item length less than 1000bytes(should be safe).
Then let's design an amature but simple protocol for this:
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| type | sequence number |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| payload ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
For types, we may need:
hello: 0x01
bye: 0x02
ack: 0x03
nack: 0x04
data: 0x05
feedback: 0x06
...
Sequence should be mono-increasing. e.g. 1, 2, 3, 4.... (Not necessory to start from 1 but OK)
It works like following:
Sender->Receiver: hello(seq=i)
Receiver->Sender: ack(seq=i)
# Sender->Receiver: hello(seq=i)
# if timeout and got no ack for seq=i
Sender->Receiver: data(seq=i+1)
Receiver->Sender: ack(seq=i+1)
# Sender->Receiver: hello(seq=i+1)
# if timeout and got no ack for seq=i+1
Sender->Receiver: data(seq=i+2)
Sender->Receiver: data(seq=i+3)
Receiver->Sender: ack(seq=i+2)
Receiver->Sender: ack(seq=i+3)
# Sender->Receiver: hello(seq=i+2)
# if timeout and got no ack for seq=i+2 or got nack for seq=i+2
Sender->Receiver: bye(seq=n)
Receiver->Sender: ack(seq=n)
# bye is not necessory

Firstly, I think you need to learn how to use wirshark or tcmpdump to analysis network streams when debugging, that will help you find out the problem and solve it.
As for your program, there are several problems the user207421 has mensioned. I think it's better to use TCP, but if you want to learn UDP by this way, the thing you need is to do a slim reliable UDP by yourself.
For example, you may need the following models
Build a send buffer and recive buffer, check every time if the buffer is empty, if not, send/receive and process it.（Cause UDP has MTU）
Add some extra format of information in the head of each datagram, which includes the size of the whole message, the sequence of the datagram, the left size, etc.(Cause you need to cut your message into many parts)
Build a controller, which need to have some function like retransmission, rebuild the message, etc.(Cause UDP is unreliable, you need to check the completeness of all parts)
Hope that can help you.

How to create IPC through sockets with a Java client and a Python server?

I am trying to make two processes communicate through local sockets: a Python server and a Java client. The data I want to pass between both consists of the bytes of a Protobuf object, with variable size. I want the connection to remain open and be used until the end of the program, because I'm passing a lot of objects that need to be processed.
Because Protobuf objects have variable size, I am sending the size of the message/response before sending the true message/response containing the object.
Currently, I am using a TCPServer from the socketserver library on the Python side. I have the following handler implemented:
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The request handler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def recv_all(self, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = self.request.recv(n - len(data))
if not packet:
return None
data += packet
return data
def handle(self):
logger.debug("Beginning of handle cycle for client: {}.".format(self.client_address))
while True:
if True: # please disregard this if condition
# Receive 4 bytes (1 int) denoting the size of the message
data_length_bytes: bytes = self.recv_all(4)
logger.debug('Received data_length: {}'.format(data_length_bytes))
# If recv read an empty request b'', then client has closed the connection
if not data_length_bytes:
break
data_length: int = int.from_bytes(data_length_bytes.strip(), byteorder='big')
data: bytes = self.recv_all(data_length).strip()
response: bytes = data.upper()
# Send length of response first
self.request.sendall(len(response).to_bytes(4, byteorder='big'))
# Send response
self.request.sendall(response)
logger.debug(
'Sent response to: {}. Size of response: {} bytes. Response: {}.'.format(self.client_address,
len(response),
response))
logger.debug("End of handle cycle for client: {}.".format(self.client_address))
And the following client:
class SocketClient
{
private static Socket socket;
private int port;
private DataOutputStream out;
private DataInputStream in;
SocketClient(int port)
{
this.port = port;
this.createSocket();
}
private void createSocket() {
InetAddress address;
try {
address = InetAddress.getByName("localhost");
socket = new Socket(address, port);
this.out = new DataOutputStream(socket.getOutputStream());
this.in = new DataInputStream(socket.getInputStream());
} catch (IOException e) {
e.printStackTrace();
}
}
byte[] sendMessageAndReceiveResponse(byte[] messageToSend){
try {
if(true) { // again, please disregard this condition
//Send the size of the message to the server
this.out.writeInt(messageToSend.length);
out.flush();
this.out.write(messageToSend);
out.flush();
//Get the response message from the server
int length = in.readInt(); // read length of incoming message
byte[] buffer = null;
if(length>=0) {
buffer = new byte[length];
in.readFully(buffer, 0, buffer.length); // read the message
}
return buffer;
}
}
catch (ConnectException exception) {
System.out.println("ATTENTION! Could not connect to socket. Nothing was retrieved from the Python module.");
exception.printStackTrace();
return null;
}
catch (Exception exception)
{
exception.printStackTrace();
return null;
}
}
void close(){
//Closing the socket
try
{
in.close();
out.close();
socket.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
I run the following experiment after starting the Python server:
SocketClient socketClient = new SocketClient(5000);
byte[] response;
// Case 1
response = socketClient.sendMessageAndReceiveResponse("12345678".getBytes());
System.out.println(new String(response));
// Case 2
response = socketClient.sendMessageAndReceiveResponse("123456781".getBytes());
System.out.println(new String(response));
// Case 3
response = socketClient.sendMessageAndReceiveResponse("12345678123456781".getBytes());
System.out.println(new String(response));
socketClient.close();
Case 1 and case 3 work well. However, when I run case 2, on the Python server side, I get the following log:
DEBUG -- [handle()] Received data_length: b'\x00\x00\x00\t' # The '\t' shouldn't be here. A '\x09' should.
And then the server throws and exception and exits the connection. This happens with every string with 8 < length < 14. What am I doing wrong, and is there an easier way to achieve what I want?

I figured out why I was having problems with messages of 8 < length < 14.
I was getting the \t character when length was equal to 9. I noticed that if I changed the length to 10, it would become \n. And to 13, \r. I realized that there wasn't any \t magically appearing. Python was for some reason converting \x09 to \t, because the horizontal tab character \t has an ASCII code equal to 9!
And when I applied the strip() function in this line:
data_length: int = int.from_bytes(data_length_bytes.strip(), byteorder='big')
, Python deleted my \t, which was actually my \x09. My problem was logging the value before stripping it, and so I took a long time to figure out my mistake.
Therefore the solution was to simply not use strip(). I leave here my current working code (at least for my tests), for someone to use:
Python server handler:
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The request handler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def recv_all(self, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = self.request.recv(n - len(data))
if not packet:
return None
data += packet
return data
def handle(self):
while True:
data_length_bytes: bytes = self.recv_all(4)
# If recv read an empty request b'', then client has closed the connection
if not data_length_bytes:
break
# DON'T DO strip() ON THE DATA_LENGTH PACKET. It might delete what Python thinks is whitespace but
# it actually is a byte that makes part of the integer.
data_length: int = int.from_bytes(data_length_bytes, byteorder='big')
# Don't do strip() on data either (be sure to check if there is some error if you do use)
data: bytes = self.recv_all(data_length)
response: bytes = data.upper()
self.request.sendall(len(response).to_bytes(4, byteorder='big'))
self.request.sendall(response)
The Java client remained the same, but without that if(true) condition that I was using for debug reasons.

Socket Java to C++ - Messages are always 8192 characters long

I am implementing a TCP socket connection from a java to a C++ programm. Currently it's one way but should become two-way someday. My messages are pretty long (~100.000 characters). Somehow my application only sends 8192 characters/bytes at once. How can that be? Is there any tool that can help debugging? Both, client and server, run on a local windows machine. I am not familar with network programming, so any help is appreciated! Thanks alot in advance!
Here is my code so far:
JAVA:
make a connection:
try {
serverSocket = new ServerSocket(socketPort);
System.out.println("waiting for client ...");
while (true) {
clientSocket = serverSocket.accept();
System.out.println("client connected.");
if (clientSocket!=null) break;
}
}
catch (IOException e) {
System.out.println(e);
}
send stuff:
OutputStream out = clientSocket.getOutputStream();
BufferedReader in = new BufferedReader( new InputStreamReader(clientSocket.getInputStream()));
//send the new data to client
PrintWriter pw = new PrintWriter(out, true);
String outString = "VERY LONG TEXT ENDING WITH SPECIAL LETTER LIKE $";
pw.println(outString);
C++:
make a connection
bool connectToHost(int PortNo, char* IPAddress)
{
//Start up Winsock…
WSADATA wsadata;
int error = WSAStartup(0x0202, &wsadata);
//Did something happen?
if (error)
return false;
//Did we get the right Winsock version?
if (wsadata.wVersion != 0x0202)
{
WSACleanup(); //Clean up Winsock
return false;
}
//Fill out the information needed to initialize a socket…
SOCKADDR_IN target; //Socket address information
target.sin_family = AF_INET; // address family Internet
target.sin_port = htons (PortNo); //Port to connect on
target.sin_addr.s_addr = inet_addr (IPAddress); //Target IP
mSocket = socket (AF_INET, SOCK_STREAM, IPPROTO_TCP); //Create socket
if (mSocket == INVALID_SOCKET)
{
return false; //Couldn't create the socket
}
//Try connecting...
if (connect(mSocket, (SOCKADDR *)&target, sizeof(target)) == SOCKET_ERROR)
{
return false; //Couldn't connect
}
else
return true; //Success
}
receive stuff:
if (connectToHost(3141, "127.0.0.1"))
{
int iResult;
char recvbuf[DEFAULT_BUFLEN]; // DEFAULT_BUFLEN = 1000000
// Receive until the peer closes the connection
do {
iResult = recv(mSocket, recvbuf, DEFAULT_BUFLEN, 0); // DEFAULT_BUFLEN = 1000000
if ( iResult > 0 )
{
std::cout<<"recvbuf: "<< recvbuf[strlen(recvbuf)-1]<<""<< std::endl; //not the last character that I sent, but supposed to be
std::cout<<"recvbuf size: "<< iResult <<""<< std::endl; //fist are 8192 and then add up until sent size
}
else if ( iResult == 0 )
printf("Connection closed\n");
else
printf("recv failed with error: %d\n", WSAGetLastError());
} while( iResult > 0 );
} else
{
printf("connect failed\n");
}

TCP sockets work on a byte stream concept. The TCP socket ensures your data arrives without error and in order as a byte stream. The sender adds bytes to the TCP byte stream, and the socket takes care of sending them to the destination. The socket does not separate your logical messages; it is your responsibility to insert separators for any logical messages that are embedded in the byte stream. The TCP socket does not necessarily send a packet on the socket every time you write bytes to the socket. This is to increase the efficiency, as measured by the number of data bytes versus the total bytes sent (data + overhead). You can read about Nagle's algorithm for TCP.
When reading from a socket, you are again consuming the byte stream. The number of times you need to call receive may not match the number of times send was called. But you know the correct bytes will be delivered in order, and the number of these bytes will be the same as those sent.
The size 8192 is probably the buffer size that triggers sending a packet.
If you send only 1 byte, then flush the socket, you should see only the one byte on the receiving end. You can also disable Nagle's algorithm by setting TCP_NODELAY in the java socket options.

You can't receive more at a time than was in the socket receive buffer. Raise the socket receive buffer size from 8192, with setsockopt(), and do the same at the sending end for the socket send buffer.

datainputstream missing bytes in.read(b)

I'm trying to write a program which acts as a server that will read bytes from a client that is written in PHP - sends request via socket (which i cannot recode due to policy) Here is the server code:
The server runs in: Red Hat Enterprise Linux Server release 6.2 (Santiago)
public void run() {
try {
serverSocket = new ServerSocket(port);
serverSocket.setSoTimeout(0);
while(!isInterrupted) {
try {
Socket server = serverSocket.accept();
LOG.info("Request received from : " + server.getRemoteSocketAddress());
DataInputStream in = new DataInputStream(server.getInputStream());
// DataInputStream in = new DataInputStream(
// new BufferedInputStream(server.getInputStream(), 10000));
byte[] bytes = new byte[10000];
int byteDupLength = in.read(t_bytes);
// in.readFully(bytes); // I tried this but to no avail
// int byteDupLength = bytes.length;
LOG.info(byteDupLength);
byte[] byteDup = new byte[byteDupLength];
System.arraycopy(bytes, 4, byteDup, 0, byteDupLength);
// FOR INFORMATION ONLY
/*for (byte b : byteDup){
LOG.info(b);
}*/
ByteBuffer buffer = ByteBuffer.wrap(byteDup);
LOG.info(buffer);
forwardRequest(byteDup);
server.close();
}
catch(SocketTimeoutException s) {
LOG.error("Socket timed out!", s);
break;
}
catch(IOException e)
{
LOG.error("IOException:", e);
break;
}
}
}
catch (IOException ex) {
LOG.error("Server socket is null", ex);
}
LOG.fatal("ReceiverEngine interrupted!");
}
I encountered a problem when the client sends request consisting of 4948 bytes. The only bytes the server can read is 2090.
Another thing that seems a mystery to me is that, when I run the server via Netbeans in my local (which is a Windows 7 Pro), it works as expected. I dont know what is wrong. Please help.. :)
Thanks!

TCP is a byte stream protocol.
The read() method isn't guaranteed to fill the buffer.
Therefore if you don't receive the expected number of bytes in a single read, you have to loop until you do receive them.
readFully() would have worked if the buffer size agreed with the size of what was sent. In your case you specified a buffer of 10,000 bytes, which weren't sent, so it would have blocked waiting for the other 10000-4948 bytes.

Beginner Java Socket Programming Errors

I'm starting to write my first Java networking program, and long story short I'm having difficulty making sure that I'm taking the right approach. Our professor has given us a server program to test against this UDP client, but I'm getting some errors I can't seem to squash. Specifically, I get IO exceptions, either "Connection Refused" or "No route to host" exceptions.
public class Lab2Client {
/**
* #param args[1] == server name, args[2] == server port, args[3] == myport
*/
public static void main(String[] args) {
//Serverport is set to 10085, our client is 10086
try {
Socket echoSocket = new Socket(args[0],Integer.parseInt(args[2]));
System.out.println("Server connection Completed\n");
DataOutputStream output = new DataOutputStream(echoSocket.getOutputStream());
byte[] toSend = new byte[5];
toSend[0] = 12; toSend[1] = 34;//Code Number
toSend[2] = 15;//GroupId
toSend[3] = 86;toSend[4] = 100;//Port number in Little Endian Order
output.write(toSend);
System.out.println("Sent Request. Waiting for reply...\n");
DataInputStream input = new DataInputStream(echoSocket.getInputStream());
byte[] toRecieve = new byte[]{0,0,0,0,0,0,0,0};
input.read(toRecieve);
checkMessage(toRecieve);
}
catch (UnknownHostException e) {
System.err.println("Servername Incorrect!");
System.exit(1);
}
catch (IOException e){
System.err.println("IO Exception. Exiting...");
System.err.println(e);
System.exit(1);
}
}
I also have some questions about my implementation regarding receiving messages in Java. I'll be getting a datagram that contains either:
a) 3 formatting bytes (unimportant to the question) along with an IP and port number
or
b) 3 formatting bytes and a port.
Is using a DataInputStream the correct way to do this? I know using an array with 9 elements is lazy instead of dynamically allocating one that's either 5 or 9, but right now I'm just trying to get this working. That being said, is there a different approach anyone would suggest for this?

You need not to wrap the stream returned by Socket.getOuputStream() with DataOutputStream - it is already the DataOutputStream
In this line:
Socket echoSocket = new Socket(args[0],Integer.parseInt(args[2]));
I suppose it should be args[1], not args[0].
Here you have to convert the integer value to its byte representation:
toSend[3] = 10086 & 0xFF;toSend[4] = 10086>>8; //Port number in Little Endian Order
Answer to your question: case b as you are not sending the IP

thought I'd leave this up for posterity. The problem is simple, and I'm a fool for not noticing it sooner.
The correct programs I was testing this against used the UDP protocol, and this program is written in TCP. The corrected code is:
public class Lab2Client {
/**
* #param args[0] == server name, args[1] == server port, args[2] == myport
*/
public static void main(String[] args) {
//Serverport is 10085, our client is 10086
try {
DatagramSocket clientSocket = new DatagramSocket();
InetAddress IPAddress = InetAddress.getByName(args[0]);
int portToSend = Integer.parseInt(args[2]);
System.out.println("Clent Socket Created");
byte[] toSend = new byte[5];
toSend[0] = 0x12; toSend[1] = 0x34;//Code Number
toSend[2] = 15;//GroupId, f in hex
toSend[3] = 0x27;toSend[4] = 0x66;
System.out.println("Byte Array Constructed");
DatagramPacket sendPacket = new DatagramPacket(toSend, toSend.length, IPAddress, Integer.parseInt(args[1]));
clientSocket.send(sendPacket);
System.out.println("Sent Request. Waiting for reply...\n");
DataInputStream input = new DataInputStream(echoSocket.getInputStream());
toRecieve can either be an error message, a return of what we sent,
or a byte stream full of IP info and port numbers.
the "heavy" byte stream is either 4 for IPv4 of 16 for IPv6, 2 bytes for port,
and the magic number (2 bytes) for a total of 9-20 bytes*/
byte[] toRecieve = new byte[9];
DatagramPacket receivePacket = new DatagramPacket(toRecieve, toRecieve.length);
clientSocket.receive(receivePacket);
checkMessage(toRecieve);
} //and so on and so forth...
Thanks to #Serge for the help, though nobody could have answered my question correctly with how I asked it. The byte shifting you suggested was important too.