So i am experimenting with socket buffer sizes and have created 2 test cases
Case 1[server sends data to client]
First we have a server which sends 100 bytes of data to a client
public static void main(String[] args)throws Exception
{
try(ServerSocket server=new ServerSocket())
{
server.bind(new InetSocketAddress(InetAddress.getLocalHost(),2500));
byte[] data=new byte[100];
for(int i=0;i<data.length;i++){data[i]=(byte)i;}
while(true)
{
try(Socket client=server.accept())
{
try(OutputStream output=client.getOutputStream())
{
output.write(data);
}
}
}
}
}
And next we have an Client which simply reads the 100 bytes received and prints it but i have set the Receive Buffer size to 25 bytes. Therefore i expect the server to send 4 packets each of size 25 bytes.
public static void main(String[] args)throws Exception
{
try(Socket client=new Socket())
{
//receive window as 25 bytes
client.setReceiveBufferSize(25);
client.setSoLinger(true,0);
client.bind(new InetSocketAddress("192.168.1.2",5000));
client.connect(new InetSocketAddress(InetAddress.getLocalHost(),2500),2000);
try(InputStream input=client.getInputStream())
{
int length;
byte[] data=new byte[100];
while((length=input.read(data))>0)
{
System.out.println(Arrays.toString(Arrays.copyOfRange(data,0,length)));
System.out.println("============");
}
}
}
}
And sure enough upon examining the packets sent from the server end using wire shark i see 4 packets[ignore malformed packets as i get no errors when receiving data on the client side and sometimes it shows malformed and sometimes no error] each with payload of size 25 bytes.
Further confirmation that each packet is 25 bytes is examining the ACK window packets sent from the client side
So case 1 is complete upon setting Receive Buffer Size from the client side server sends packets of that much size
Case 2[Client sends data to server]
The roles are reversed and this time we set the ServerSocket Receive Buffer size before binding as follows
public static void main(String[] args)throws Exception
{
try(ServerSocket server=new ServerSocket())
{
//server receive window
server.setReceiveBufferSize(10);
server.bind(new InetSocketAddress(InetAddress.getLocalHost(),2500));
int length;
byte[] data=new byte[100];
while(true)
{
try(Socket client=server.accept())
{
try(InputStream input=client.getInputStream())
{
while((length=input.read(data))>0)
{
System.out.println(Arrays.toString(Arrays.copyOfRange(data,0,length)));
System.out.println("============");
}
}
}
}
}
}
And the client sends data with no additional settings as follows
public static void main(String[] args)throws Exception
{
try(Socket client=new Socket())
{
client.setReuseAddress(true);
client.connect(new InetSocketAddress(InetAddress.getLocalHost(),2500),2000);
byte[] data=new byte[100];
for(int i=0;i<data.length;i++){data[i]=(byte)i;}
try(OutputStream output=client.getOutputStream()){output.write(data);}
}
}
Since the server has said it is willing to receive packets of only 10 bytes i expect the client to send 10 packets each of size 10 bytes. And sure enough upon examining the packets received on the server end using wire shark i get the expected output
So this completes understanding of the Receive Buffer on both the client and the server side. If one side broadcasts its receive buffer size the other side sends only that much data with each packet transmission which makes total sense.
Now comes the harder to understand part the Send Buffers. My understanding of an Send Buffer is that
Holds bytes sent by the socket and gets emptied out only after
receiving an ACK from the recipient . And if it gets full it blocks the
socket from sending any more data until an ACK is received.
Thus if the sender has an send_buffer=10 bytes and if the receiver has an receive_buffer=30 bytes. The sender should still send only 10 Bytes as it is capable of holding on to only that much data which must then be Acknowledged by the receiver and then it can send the next 10 bytes and so on. But despite all my combinations of setting send buffer on the server and client side as follows
1)Client sends data to server
server side=ServerSocket.setReceiveBufferSize(30);
client side=client.setSendBufferSize(10);
2)Server sends data to client
server side=serverSocket.accept().setSendBufferSize(10);
client side=client.setReceiveBufferSize(30);
The packets received on the recipient side using wire shark is always the same. i.e sender always sends packets of size 30 bytes. i.e sender is always dominated by the receiver's settings
a) Where have i gone wrong in my understanding ?
b) An very simply test case as presented above where Send Buffer actually makes an difference in both client & server side would be appreciated
I filed an incident report a few days ago and it has been confirmed as a bug
Related
This question has been asked a lot, but so far, none of the solutions that I applied from previous answers have helped me.
Main goal
I am trying to learn UDP conexions and this is my attempt. I want to have a client ask for a picture at a server via UDP and the server will send it. Then the client will create a file with that information given.
Explanation
My main idea is to ask the server for an image using a "GET" command (not the HTTP, just GET) followed by the name of the image(extension included). Then the client awaits an answer which is the image requested.
Problems
The client waits and answer which does no come
Research
From another similar question it was a problem that I was using the same PORT for both receive and connect, so I added two ports, receivingPORT and sendingPORT, no results from the Client.
From other similar questions, It was a Firewall problem. So, on a Win10 machine, I created a new rule for UDP in the Firewall for the ports that I am using for this application, and nothing was received by the Client...
I have checked that the image is loaded into byte[] and the image is sent. But on the Client, nothing is received and stays there waiting for a connection to come through
CODE from Server
public class UDPserver {
static DatagramSocket serverUDP;
static DatagramPacket packet;
static InetAddress address;
static byte[] buffer = new byte[65507];//65507
final static int receivingPORT = 6668;
final static int sendingPORT = 6669;
public static void main(String[] args) throws SocketException, IOException, InterruptedException{
boolean serverActive = true;
String order = "";
String file = "";
//Instantiate server
serverUDP = new DatagramSocket(receivingPORT);
while(serverActive){
//Kind of packet we want to receive
packet = new DatagramPacket(buffer, buffer.length);
System.out.println("Server awaiting connection...");
//Receive it
serverUDP.receive(packet);
System.out.println("Received packet from: " + packet.getAddress() + "/" + packet.getPort());
//What does the packet contain?
String msg = new String(packet.getData());
address = packet.getAddress();
System.out.println("Order from: " + address + "/" + receivingPORT + " says: " + msg);
try{
order = msg.split(" ")[0].trim();
file = msg.split(" ")[1].trim();
} catch (Exception e){
}
switch(order){
case("GET"):{
System.out.println("Sending back an image...");
buffer = loadImageFromServer(file);
packet = new DatagramPacket(buffer, buffer.length, address, sendingPORT);
Thread.sleep(5000);
serverUDP.send(packet);
System.out.println("Client served");
break;
}
case("DISCONNECT"):{
buffer = "Server is disconnecting...".getBytes();
packet = new DatagramPacket(buffer, buffer.length, address, sendingPORT);
serverUDP.send(packet);
serverActive = false;
serverUDP.close();
break;
}
}
}
}
static byte[] loadImageFromServer(String path) {
try {
System.out.println("Loading path: " + path);
//Instantiate a buffer from the image for it
BufferedImage img = ImageIO.read(UDPserver.class.getResource(path));
//Create a byte[] stream object to handle the data
ByteArrayOutputStream baos = new ByteArrayOutputStream();
//Write the image data into those above with jpg format
ImageIO.write(img, "png", baos);
//Flush the information
baos.flush();
byte[] buffer = baos.toByteArray(); //Write it out on a byte string and return it
return buffer;
} catch (IOException ex) {
Logger.getLogger(UDPserver.class.getName()).log(Level.SEVERE, null, ex.fillInStackTrace());
System.exit(-1);
}
return null;
}
}
CODE client
public class Client {
static DatagramSocket clientUDP;
static InetAddress address;
static DatagramPacket packetSend;
static DatagramPacket packetReceive;
static int SIZE = 65507;
final static int receivingPORT = 6669;
final static int sendingPORT = 6668;
static byte[] buffer = new byte[SIZE];
static Scanner scan = new Scanner(System.in);
public static void main(String[] args) throws SocketException, UnknownHostException, IOException{
boolean clientLoop = true;
//Get address
address = InetAddress.getByName("localhost");
//Instantiate Client -> UDP
clientUDP = new DatagramSocket();
while(clientLoop){
System.out.print("Enter any key and press enter");
scan.next(); //Just to stop the loop
//Load the buffer
buffer = "GET imagenServidor.png".getBytes();
//buffer = "DISCONNECT".getBytes();
System.out.println("Buffer is ready");
//Arm the packet
packetSend = new DatagramPacket(buffer, buffer.length, address, sendingPORT);
System.out.println("Packet is armed!");
//Send the packet to the server
clientUDP.send(packetSend);
System.out.println("Order sent to server");
System.out.println("Waiting an answer");
packetReceive = new DatagramPacket(buffer, buffer.length, address, receivingPORT);
clientUDP.receive(packetReceive);
System.out.println("Server answered!");
ByteArrayInputStream bais = new ByteArrayInputStream(packetReceive.getData());
BufferedImage image = ImageIO.read(bais);
System.out.println(image);
}
clientUDP.close();
}
}
NOTES
This is a UDP exercise
The Reason
MTU!
You are sending packets with long buffe through UDP directly, which may not work in most network circumstances.
A packet sent through UDP should not be longer than the network MTU, otherwise it would be dropped. The network MTU may not be more than 1500 on most net nods(routers/switchs/hosts...), and even smaller sometimes. Though some nods may do sigmentation for ip packets, but you should not count on it when you are using UDP.
Suggestions
Use TCP instead in this application, as for:
You are sending data which expected to be complete (otherwise it would be useless).
You do not care about congestion control algorithms.
So just go with TCP.
Edit Based on The Update of The Question
So, as this is an excercise, in which you have to use UDP only.
As a file might be useless unless it is complete, you have to make sure:
All packets are possible to pass the path. Which means network should be connected both physically and virtually, and packet size should always be smaller than the MTU.
If any packets are lost, both the receiver and the sender should be able to know.
If any apckets come out of order, the receiver should be able to know.
Sender should be able to cache and resend the packets which are not confirmed by the receiver yet.
Make sure your have a good network connection. Split the image buffer into buffer array with each buffer item length less than 1000bytes(should be safe).
Then let's design an amature but simple protocol for this:
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| type | sequence number |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| payload ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
For types, we may need:
hello: 0x01
bye: 0x02
ack: 0x03
nack: 0x04
data: 0x05
feedback: 0x06
...
Sequence should be mono-increasing. e.g. 1, 2, 3, 4.... (Not necessory to start from 1 but OK)
It works like following:
Sender->Receiver: hello(seq=i)
Receiver->Sender: ack(seq=i)
# Sender->Receiver: hello(seq=i)
# if timeout and got no ack for seq=i
Sender->Receiver: data(seq=i+1)
Receiver->Sender: ack(seq=i+1)
# Sender->Receiver: hello(seq=i+1)
# if timeout and got no ack for seq=i+1
Sender->Receiver: data(seq=i+2)
Sender->Receiver: data(seq=i+3)
Receiver->Sender: ack(seq=i+2)
Receiver->Sender: ack(seq=i+3)
# Sender->Receiver: hello(seq=i+2)
# if timeout and got no ack for seq=i+2 or got nack for seq=i+2
Sender->Receiver: bye(seq=n)
Receiver->Sender: ack(seq=n)
# bye is not necessory
Firstly, I think you need to learn how to use wirshark or tcmpdump to analysis network streams when debugging, that will help you find out the problem and solve it.
As for your program, there are several problems the user207421 has mensioned. I think it's better to use TCP, but if you want to learn UDP by this way, the thing you need is to do a slim reliable UDP by yourself.
For example, you may need the following models
Build a send buffer and recive buffer, check every time if the buffer is empty, if not, send/receive and process it.（Cause UDP has MTU）
Add some extra format of information in the head of each datagram, which includes the size of the whole message, the sequence of the datagram, the left size, etc.(Cause you need to cut your message into many parts)
Build a controller, which need to have some function like retransmission, rebuild the message, etc.(Cause UDP is unreliable, you need to check the completeness of all parts)
Hope that can help you.
I have developing a simple client in java (I use a Windows 7 machine) to communicate with a server. The problem was that the server never understood my request. So I have analysed the communication with Wireshark and have noticed that only one byte is send in a first TCP packet, and 40ms after the remaining bytes are send in a other packet.
In fact, we communicate with binary frames, so all the frames must begin with the total length of the frame on 2 bytes. So it is normal that the server will never understand me. All my frames never exceed 10 bytes, so it's a insignificant amount of data. I know that TCP packets can be segmented, but for me it has no sense to segment a tiny frame after only one byte.
After unsuccessful hours of research, I tried casually to send bytes in a other way using write method instead of writeBytes method. Now I send all the data in only one TCP packet and the communication works fine, but I have never find the explanation. If someone knows, I will be happy to learn it.
Here is my code :
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Socket;
public class Client {
public static void main(String argv[]) {
try {
Socket socket = new Socket();
socket.connect(new InetSocketAddress("10.2.1.1", 1003), 1000);
DataOutputStream outToServer = new DataOutputStream(socket.getOutputStream());
DataInputStream inFromServer = new DataInputStream(socket.getInputStream());
// Hexadecimal frame to send to server
String hexFrame = "0004FF9D3175";
// Build bytes frame
String bytesFrame = "";
for (int i=0; i<hexFrame.length()/2; i++) {
bytesFrame += (char) Integer.parseInt(hexFrame.substring(i*2, (i+1)*2), 16);
}
// This generates 2 TCP packets
// outToServer.writeBytes(bytesFrame);
// This generates only 1 TCP packet
outToServer.write(bytesFrame.getBytes());
// Read answer from server
hexFrame = "";
short frame_length = inFromServer.readShort();
for (int i=0; i<frame_length; i++) {
hexFrame += String.format("%2s", Integer.toHexString(inFromServer.readUnsignedByte())).replace(" ", "0");
}
System.out.println("Receive : " + hexFrame);
socket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Neither Java nor TCP makes any guarantees about this. TCP can segment the data any way it likes, and you have no business relying on any two bytes being delivered consecutively. The problem here is really at the reading end, that makes incorrect assumptions.
In fact, we communicate with binary frames
In fact you are communicating over a byte-stream protocol. No frames, no message boundaries, nothing.
However, if you want a little more control over this you should use a BufferedOutputStream between the DataOutputStream and the socket output stream, and similarly a BufferedInputStream at the receiving end. Flush the stream when you want the data to be sent, typically just before the next read.
I have a Java TCP Server Socket program that is expecting about 64 bytes of data from a piece of remote hardware. The Server code is:
public void run () throws Exception
{
//Open a socket on localhost at port 11111
ServerSocket welcomeSocket = new ServerSocket(11111);
while(true) {
//Open and Accept on Socket
Socket connectionSocket = welcomeSocket.accept();
DataInputStream dIn = new DataInputStream(connectionSocket.getInputStream());
int msgLen = dIn.readInt();
System.out.println("RX Reported Length: "+ msgLen);
byte[] msg = new byte[msgLen];
if(msgLen > 0 ) {
dIn.readFully(msg);
System.out.println("Message Length: "+ msg.length);
System.out.println("Recv[HEX]: " + StringTools.toHexString(msg));
}
}
}
This works correctly as I am able to test locally with a simple ACK program:
public class ACK_TEST {
public static void main (String[] args)
{
System.out.println("Byte Sender Running");
try
{
ACK_TEST obj = new ACK_TEST ();
obj.run();
}
catch (Exception e)
{
e.printStackTrace ();
}
}
public void run () throws Exception
{
Socket clientSocket = new Socket("localhost", 11111);
DataOutputStream dOut = new DataOutputStream(clientSocket.getOutputStream());
byte rtn[] = null;
rtn = new byte[1];
rtn[0] = 0x06; // ACK
dOut.writeInt(rtn.length); // write length of the message
dOut.write(rtn); // write the message
System.out.println("Byte Sent");
clientSocket.close();
}
}
And this correctly produces this output from the Server side:
However, when I deploy the same Server code on the Raspberry Pi and the hardware sends data to it, the data length is far greater and causes a heap memory issue (Even with the Heap pre-set at 512MB, which is definitely incorrect and unnecessary)
My presumption is I am reading the data wrong from the TCP socket as from the debug from the hardware, it's certainly not sending packets of this size.
Update: I have no access to the Client source code. I do however need to take the input TCP data stream, place it into a byte array, and then another function (Not shown) parses out some known HEX codes. That function expects a byte array input.
Update: I reviewed the packet documentation. It is a 10 byte header. The first Byte is a protocol identifier. The next 2 bytes is the Packet Length (Total number of bytes in the packet, including all the header bytes and checksum) and the last 7 are a Unique ID. Therefore, I need to read those 2 bytes and create a byte array that size.
Apparently the length from the header is about 1GB. Looks like the problem on the other end. Don't you mix low/big endian encoding?
Goodevening everyone.
I am trying to create an application using Eclipse(Kepler) where I send an array of DatagramPackets. My server side application is receiving all packets sent by the client. But when the server tries to respond back by sending the response packets back to client, my client is unable to receive all the packets.
I would be really obliged if someone helped me out.
here is my client side application code:
public static void main(String [] args) throws IOException
{
timing t=new timing(); // timing is the name of my class
InetAddress inet;
DatagramSocket ds;
int k=1024*60;
DatagramPacket[] p=new DatagramPacket[64];
DatagramPacket []recd=new DatagramPacket[64];
byte[] buf=new byte[1024*60];
if(args[0]==null)
{
args[0]="localhost";
}
inet=InetAddress.getByName(args[0]);
ds=new DatagramSocket(6443);
for(int i=0;i>64;i++)
{
p[i]=new DatagramPacket(sent.getBytes(),sent.length(),inet,7443);
recd[i]=new DatagramPacket(buf,buf.length);
}
ds.setSoTimeout(120000);
int buffer=ds.getReceiveBufferSize();
int j=ds.getSendBufferSize();
while(h<64)
{
p[h]=new DatagramPacket(sent.getBytes(),sent.length(),inet,7443);
ds.send(p[h]);
System.out.println("Client has sent packet:"+h);
h++;
}
System.out.println("Receiving.");
h=0;
while(h<64) // UNABLE TO RECEIVE ALL SERVER SIDE PACKETS . PROBLEM CODE
{
recd[h]=new DatagramPacket(buf,buf.length);
ds.receive(recd[h]);
System.out.println("Client has recd packet:"+h);
h++;
}
SERVER SIDE APPLICATION:
try{
byte[] buf=new byte[60*1024];
InetAddress add=InetAddress.getByName("localhost");
for(int i=0;i<64;i++)
dp[i]=new DatagramPacket(buf,buf.length,add,6443);
ds.setSoTimeout(120000);
System.out.println("SERVER READY AND LISTENING PORT 6443");
int h=0;
while(h<64)
{
dp[h]=new DatagramPacket(buf,buf.length,add,6443);
ds.receive(dp[h]);
System.out.println("Packet "+h+"recd.");
h++;
}
String x1=new String(dp[63].getData());
System.out.println("Server recd:"+x1);// correct. no problem here
InetAddress add2=dp[63].getAddress();
int port=dp[63].getPort();// this is fine. same data sent in all packets
h=0;
while(h<64)
{
dp[h]=new DatagramPacket(buf,buf.length,add2,port);
ds.send(dp[h]);
System.out.println("Server has sent packet:"+h);
h++;
}
kindly help me as when i send a single datagrampacket its being recd. but this array of packets isnt.
I do not know much about your code, but if I really want this 'for' loop:
for (int i=0;i>64;i++)
to be executed at least once I would change it to i<64.
You don't show what sent is in the client, or therefore how long it is, but I suspect it is some reasonable size that can actually be sent.
By contrast, in the server you are attempting to send 60k datagrams, which won't work unless the sender's socket send buffer and the receiver's socket receive buffer are at least that size.
I have a server running on a separate thread, and for some reason, it only runs when it receives packets! Why is it doing this? Shouldn't it be running continuously?
public void run() {
while (running) {
System.out.println(true);
byte[] data = new byte[1024];
DatagramPacket packet = new DatagramPacket(data, data.length);
try {
this.socket.receive(packet);
} catch (IOException e) {
e.printStackTrace();
}
parsePacket(packet.getData(), packet.getAddress(), packet.getPort());
}
}
And I start it like this:
public static void main(String[] args) {
GameServer server = new GameServer();
server.start();
}
The class extends Thread.
socket.receive is a blocking method. So the code is waiting on receive untill you receive any data.
From here
public void receive(DatagramPacket p)
throws IOException
Receives a datagram packet from this socket. When this method returns,
the DatagramPacket's buffer is filled with the data received. The
datagram packet also contains the sender's IP address, and the port
number on the sender's machine.
This method blocks until a datagram is received. The length field of
the datagram packet object contains the length of the received
message. If the message is longer than the packet's length, the
message is truncated.
It clearly says that method blocks and wait for Datagram.
Your Thread is running correctly. The method DatagramSocket.receive(DatagramPacket) blocks until a packet is received.
The default behaviour is to block infinitely until a packet is received. You can specfiy a timeout using DatagramSocket.setSoTimeout(int), if you want to periodically log whether a packet is received or not, or to check if your Thread is still running.