Related
This question has been asked a lot, but so far, none of the solutions that I applied from previous answers have helped me.
Main goal
I am trying to learn UDP conexions and this is my attempt. I want to have a client ask for a picture at a server via UDP and the server will send it. Then the client will create a file with that information given.
Explanation
My main idea is to ask the server for an image using a "GET" command (not the HTTP, just GET) followed by the name of the image(extension included). Then the client awaits an answer which is the image requested.
Problems
The client waits and answer which does no come
Research
From another similar question it was a problem that I was using the same PORT for both receive and connect, so I added two ports, receivingPORT and sendingPORT, no results from the Client.
From other similar questions, It was a Firewall problem. So, on a Win10 machine, I created a new rule for UDP in the Firewall for the ports that I am using for this application, and nothing was received by the Client...
I have checked that the image is loaded into byte[] and the image is sent. But on the Client, nothing is received and stays there waiting for a connection to come through
CODE from Server
public class UDPserver {
static DatagramSocket serverUDP;
static DatagramPacket packet;
static InetAddress address;
static byte[] buffer = new byte[65507];//65507
final static int receivingPORT = 6668;
final static int sendingPORT = 6669;
public static void main(String[] args) throws SocketException, IOException, InterruptedException{
boolean serverActive = true;
String order = "";
String file = "";
//Instantiate server
serverUDP = new DatagramSocket(receivingPORT);
while(serverActive){
//Kind of packet we want to receive
packet = new DatagramPacket(buffer, buffer.length);
System.out.println("Server awaiting connection...");
//Receive it
serverUDP.receive(packet);
System.out.println("Received packet from: " + packet.getAddress() + "/" + packet.getPort());
//What does the packet contain?
String msg = new String(packet.getData());
address = packet.getAddress();
System.out.println("Order from: " + address + "/" + receivingPORT + " says: " + msg);
try{
order = msg.split(" ")[0].trim();
file = msg.split(" ")[1].trim();
} catch (Exception e){
}
switch(order){
case("GET"):{
System.out.println("Sending back an image...");
buffer = loadImageFromServer(file);
packet = new DatagramPacket(buffer, buffer.length, address, sendingPORT);
Thread.sleep(5000);
serverUDP.send(packet);
System.out.println("Client served");
break;
}
case("DISCONNECT"):{
buffer = "Server is disconnecting...".getBytes();
packet = new DatagramPacket(buffer, buffer.length, address, sendingPORT);
serverUDP.send(packet);
serverActive = false;
serverUDP.close();
break;
}
}
}
}
static byte[] loadImageFromServer(String path) {
try {
System.out.println("Loading path: " + path);
//Instantiate a buffer from the image for it
BufferedImage img = ImageIO.read(UDPserver.class.getResource(path));
//Create a byte[] stream object to handle the data
ByteArrayOutputStream baos = new ByteArrayOutputStream();
//Write the image data into those above with jpg format
ImageIO.write(img, "png", baos);
//Flush the information
baos.flush();
byte[] buffer = baos.toByteArray(); //Write it out on a byte string and return it
return buffer;
} catch (IOException ex) {
Logger.getLogger(UDPserver.class.getName()).log(Level.SEVERE, null, ex.fillInStackTrace());
System.exit(-1);
}
return null;
}
}
CODE client
public class Client {
static DatagramSocket clientUDP;
static InetAddress address;
static DatagramPacket packetSend;
static DatagramPacket packetReceive;
static int SIZE = 65507;
final static int receivingPORT = 6669;
final static int sendingPORT = 6668;
static byte[] buffer = new byte[SIZE];
static Scanner scan = new Scanner(System.in);
public static void main(String[] args) throws SocketException, UnknownHostException, IOException{
boolean clientLoop = true;
//Get address
address = InetAddress.getByName("localhost");
//Instantiate Client -> UDP
clientUDP = new DatagramSocket();
while(clientLoop){
System.out.print("Enter any key and press enter");
scan.next(); //Just to stop the loop
//Load the buffer
buffer = "GET imagenServidor.png".getBytes();
//buffer = "DISCONNECT".getBytes();
System.out.println("Buffer is ready");
//Arm the packet
packetSend = new DatagramPacket(buffer, buffer.length, address, sendingPORT);
System.out.println("Packet is armed!");
//Send the packet to the server
clientUDP.send(packetSend);
System.out.println("Order sent to server");
System.out.println("Waiting an answer");
packetReceive = new DatagramPacket(buffer, buffer.length, address, receivingPORT);
clientUDP.receive(packetReceive);
System.out.println("Server answered!");
ByteArrayInputStream bais = new ByteArrayInputStream(packetReceive.getData());
BufferedImage image = ImageIO.read(bais);
System.out.println(image);
}
clientUDP.close();
}
}
NOTES
This is a UDP exercise
The Reason
MTU!
You are sending packets with long buffe through UDP directly, which may not work in most network circumstances.
A packet sent through UDP should not be longer than the network MTU, otherwise it would be dropped. The network MTU may not be more than 1500 on most net nods(routers/switchs/hosts...), and even smaller sometimes. Though some nods may do sigmentation for ip packets, but you should not count on it when you are using UDP.
Suggestions
Use TCP instead in this application, as for:
You are sending data which expected to be complete (otherwise it would be useless).
You do not care about congestion control algorithms.
So just go with TCP.
Edit Based on The Update of The Question
So, as this is an excercise, in which you have to use UDP only.
As a file might be useless unless it is complete, you have to make sure:
All packets are possible to pass the path. Which means network should be connected both physically and virtually, and packet size should always be smaller than the MTU.
If any packets are lost, both the receiver and the sender should be able to know.
If any apckets come out of order, the receiver should be able to know.
Sender should be able to cache and resend the packets which are not confirmed by the receiver yet.
Make sure your have a good network connection. Split the image buffer into buffer array with each buffer item length less than 1000bytes(should be safe).
Then let's design an amature but simple protocol for this:
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| type | sequence number |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| payload ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
For types, we may need:
hello: 0x01
bye: 0x02
ack: 0x03
nack: 0x04
data: 0x05
feedback: 0x06
...
Sequence should be mono-increasing. e.g. 1, 2, 3, 4.... (Not necessory to start from 1 but OK)
It works like following:
Sender->Receiver: hello(seq=i)
Receiver->Sender: ack(seq=i)
# Sender->Receiver: hello(seq=i)
# if timeout and got no ack for seq=i
Sender->Receiver: data(seq=i+1)
Receiver->Sender: ack(seq=i+1)
# Sender->Receiver: hello(seq=i+1)
# if timeout and got no ack for seq=i+1
Sender->Receiver: data(seq=i+2)
Sender->Receiver: data(seq=i+3)
Receiver->Sender: ack(seq=i+2)
Receiver->Sender: ack(seq=i+3)
# Sender->Receiver: hello(seq=i+2)
# if timeout and got no ack for seq=i+2 or got nack for seq=i+2
Sender->Receiver: bye(seq=n)
Receiver->Sender: ack(seq=n)
# bye is not necessory
Firstly, I think you need to learn how to use wirshark or tcmpdump to analysis network streams when debugging, that will help you find out the problem and solve it.
As for your program, there are several problems the user207421 has mensioned. I think it's better to use TCP, but if you want to learn UDP by this way, the thing you need is to do a slim reliable UDP by yourself.
For example, you may need the following models
Build a send buffer and recive buffer, check every time if the buffer is empty, if not, send/receive and process it.（Cause UDP has MTU）
Add some extra format of information in the head of each datagram, which includes the size of the whole message, the sequence of the datagram, the left size, etc.(Cause you need to cut your message into many parts)
Build a controller, which need to have some function like retransmission, rebuild the message, etc.(Cause UDP is unreliable, you need to check the completeness of all parts)
Hope that can help you.
In the application that I am developing in Android, I send bytes of number 5 using sockets tcp and udp. I would like to know if it is possible to get the amount of payload that was received until a SocketTimeoutException exception was thrown.I'm doing some moving tests with Wi-Fi Direct technology so, when sending, all may not be received because the peers are disconnected before.
Also for the case of UDP when packet loss occurs I would like to know the amount of information that I receive.
To read what I get, I use readFully and recieve. This reception is done in a single step or in a loop in which I receive large amounts of information. I could not receive the bytes one by one because it would be really slow.
TCP RX:
ServerSocket serverSocket = new ServerSocket(SERVERPORT);
Socket client = serverSocket.accept();
DataInputStream DIS = new DataInputStream(client.getInputStream());
int tamMensaje = (100 * 1000 * 1000);
byte[] payload = new byte[tamMensaje];
DIS.readFully(payload);
int failures = 0;
for (int i = 0; i < tamMensaje; i++) {
if (payload[i] != 5) {
failures = failures + 1;
}
}
int nPayLoad = payload.length- failures;
client.close();
serverSocket.close();
UDP RX:
DatagramSocket datagramSocket = new DatagramSocket(SERVERPORT);
byte[] buffer = new byte[1400];
DatagramPacket packet = new DatagramPacket(buffer, buffer.length);
int tamMensaje = 1400 *71428;
int iteration = 71428;
for (int i = 0; i < iteration; i++) {
datagramSocket.receive(packet);
msg_received = msg_received + new String(buffer, 0, packet.getLength());
}
byte[] payload = msg_received.getBytes();
int failures = 0;
for (int i = 0; i < tamMensaje; i++) {
if (payload[i] != 5) {
failures = failures + 1;
}
}
int nPayload = payload.length- failures;
datagramSocket.close();
How can I know the amount of information received if the communication is cut off when sending for TCP and UDP occurs as well as in case UDP does not receive everything it should?
Thanks
You can't if you use any of the compound readXXX() methods, as their APIs don't provide any means of retrieving it.
However as, for example, an int is normally sent in one go, e.g. by writeInt(), there is really little reason why half of it would arrive long before the other half, even if you got really unlucky and segmentation or packetization split it. It could happen, but you would have to be really be extraordinarily unlucky, and in any case half an int is no more use than none of it.
Similar considerations apply to readFully(): presumably you are reading something, all of which is supposed to be there, so half of it wouldn't be of much use; and conversely, if it would, don't use readFully().
If you use the basic read() method, the answer of course is zero: nothing arrived before the timeout.
I could not receive the bytes one by one because it would be really slow.
Of course, but that's not the only alternative. Use read(byte[]), with a buffer size of your choice, say 8192. Or a BufferedInputStream.
I tested the performance of transferring data from a program to another over socket on a single computer, and the speed is 120MBytes/s, is it normal?
My server and client programs are both extremely simple.
And my computer is AMD Athlon X2 4000+, 4G DDR2 667 ram, with windows xp sp3.
My friend said it was slow, and should be faster. But I don't know how can I improve them, or is there any other libraries I can try to get a better speed?
UPDATE
The server and client programs were both on my own computer, a single computer. The network card will limit the speed or not?
Server.java
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
public class SimpleServer {
public static void main(String[] args) throws Exception {
ServerSocket server = new ServerSocket(6666);
Socket socket = server.accept();
OutputStream output = socket.getOutputStream();
byte[] bytes = new byte[10 * 1024]; // 10K
for (int i = 0; i < bytes.length; i++) { bytes[i] = 12; } // fill the bytes
// send them again and again
while (true) {
output.write(bytes);
}
}
}
Client.java
public class SimpleClient {
public static void main(String[] args) throws Exception {
Socket socket = new Socket("127.0.0.1", 6666);
InputStream input = socket.getInputStream();
long total = 0;
long start = System.currentTimeMillis();
byte[] bytes = new byte[10240]; // 10K
// read the data again and again
while (true) {
int read = input.read(bytes);
total += read;
long cost = System.currentTimeMillis() - start;
if (cost > 0 && System.currentTimeMillis() % 1000 == 0) {
System.out.println("Read " + total + " bytes, speed: " + (total / (1024.0*1024)) / (cost / 1000.0) + " MB/s");
}
}
}
}
Can you give me the output of this program?
public class SimpleServer {
public static void main(String[] args) throws Exception {
ServerSocket server = new ServerSocket(6666);
Socket socket = server.accept();
OutputStream output = socket.getOutputStream();
byte[] bytes = new byte[32*1024]; // 32K
while (true) {
output.write(bytes);
}
}
}
public class SimpleClient {
public static void main(String[] args) throws Exception {
Socket socket = new Socket("127.0.0.1", 6666);
InputStream input = socket.getInputStream();
long total = 0;
long start = System.currentTimeMillis();
byte[] bytes = new byte[32*1024]; // 32K
for(int i=1;;i++) {
int read = input.read(bytes);
if (read < 0) break;
total += read;
if (i % 500000 == 0) {
long cost = System.currentTimeMillis() - start;
System.out.printf("Read %,d bytes, speed: %,d MB/s%n", total, total/cost/1000);
}
}
}
}
on my machine prints
Read 25,586,204,672 bytes, speed: 5,245 MB/s
Read 53,219,426,304 bytes, speed: 5,317 MB/s
Read 85,018,968,064 bytes, speed: 5,416 MB/s
Read 117,786,968,064 bytes, speed: 5,476 MB/s
Try sending 32K blocks many times (for at least 2 seconds) and you should get 400 MB/s or more. e.g. at least 10,000 times.
On a very fast machine you can get 1 GB/s on a single client. With multiple clients you might get 8 GB/s.
Here is an example
Making file transfer more efficient Java
If you have a 100 Mb card you can expect around 11 MB/s (bytes per second).
Similarly for 1 Gb ethernet youc an expect around 110 MB/s
For a 10 Gig-E ethernet you might get up to 1 GB/s however you might only get half this unles syour system is highly tuned.
You're only transferring 10k bytes to perform your test. There is overhead to a lot of what you're doing and such a small set of data may be getting skewed by this overhead. (e.g. creating sockets, allocating memory etc.) You should create and transfer a much larger set of data (dozens of MBs+) to get a more realistic idea of what's going on. Doing this will make the overhead a smaller, less significant part of the data transfer process.
You should also perform this test a number of times (> 10) and take the average because your computer will be under loads from services, network transfers etc. at different, random points in time and these loads will affect your transfer rate. With 10+ iterations you could also drop any run times that are unusually slow, such as > 2 standard deviations.
The network card will limit the speed or not?
You are using 127.0.0.1 - the local / loopback address. Traffic from / to that address doesn't pass through the network card.
This means that this is not a valid way to measure real network throughput. However, it is a reasonable measure of Java's ability to move data through the local network stack ... on your machine.
What would be interesting would be to compare the result you are getting with an equivalent client / server pair written in C or C++.
Testing bandwidth between two peers in the same host doesn't exercise the NIC or the network at all, it is all looped back. Your data is therefore basically meaningless. Try it between two computers.
WinXP doesn't support auto-window scaling and probably the send/receive buffer size of the socket is too low for throughput test.
More also FileInputStream (socket uses FileInputStream practically) has smaller char[] buffer on Windows than Linux in the native code. The read/write is done via auxiliary buffer on the stack and the large chunk you put is funneled through.
Using a large direct buffer (as large as the socket buffers) is your best bet for throughput.
Well, it is interesting, but this nio implementation was on par with the SimpleServer/SimpleClient provided above(and sometimes actually beat it) (Though this is only if run in single threaded mode which is more one to one anyways!!!)
Both SimpleServer/SimpleClient and the nio IntegTestLocalhostThroughput.java provided in this directory...
https://github.com/deanhiller/webpieces/tree/master/core/core-asyncserver/src/test/java/org/webpieces/nio/api/throughput
I just thought that was quite interesting that nio is basically just as fast but scales better than the old io stuff.
Im trying to create a Speex Voip client and server. I have the basics down and its working OK on the local machine over UDP. I am using JSpeex for portability. Im looking for tips on creating the client and server. What are your thoughts?
The JSpeex library can only encode 320 bytes per call so the packets sent to the server are tiny (in my case ~244 bytes). Would it be better for the client to wait until about 1 or 2 KB of encoded data is ready before sending or let the server handle buffering the packets?
Also, any help on how to implement buffering the data would be nice.
Some of what I have that works on the local machine.
Client:
public void run() {
int nBytesToRead = (m_inputAudioFormat.getFrameSize() * 160);
int nAvailable = 0;
byte[] abPCMData = new byte[nBytesToRead];
byte[] abSpeexData = null;
UserSpeexPacket userSpeexPacket = new UserSpeexPacket("Xiphias3", "TheLounge", null, 0);
while (m_captureThread != null) {
nAvailable = m_line.available();
if (nAvailable >= nBytesToRead) {
int nBytesRead = m_line.read(abPCMData, 0, nBytesToRead);
if (nBytesRead == -1) break;
if (nBytesRead < nBytesToRead)
Arrays.fill(abPCMData, nBytesRead, abPCMData.length, (byte) 0);
abSpeexData = createSpeexPacketFromPCM(abPCMData, 0, abPCMData.length);
//DatagramPacket packet = new DatagramPacket(abSpeexData, 0, abSpeexData.length, m_connection.getInetAddress(), m_nServerPort);
userSpeexPacket.setSpeexData(abSpeexData);
userSpeexPacket.incrementPacketNumber();
DatagramPacket packet = UserSpeexPacket.userSpeexPacketToDatagramPacket(m_connection.getInetAddress(), m_connection.getPort(), userSpeexPacket);
try {
m_connection.send(packet);
}
catch(IOException iox) {
System.out.println("Connection to server lost: " + iox.getMessage());
break;
}
}
}
closeLine();
disconnect();
}
public byte[] createSpeexPacketFromPCM(byte[] abPCMData, int nOffset, int nLength)
{
byte[] abEncodedData = null;
m_speexEncoder.processData(abPCMData, nOffset, nLength);
abEncodedData = new byte[m_speexEncoder.getProcessedDataByteSize()];
m_speexEncoder.getProcessedData(abEncodedData, 0);
return abEncodedData;
}
Server:
DatagramPacket packet = new DatagramPacket(new byte[2048], 0, 2048);
byte[] abPCMData = null;
long lPrevVolPrintTime = 0;
while (m_bServerRunning) {
try {
m_serverSocket.receive(packet);
//System.out.println("Packet size is " + packet.getData().length);
//System.out.println("Got packet from " + packet.getAddress().getHostAddress());
//abPCMData = decodeSpeexPacket(packet.getData(), 0, packet.getLength());
UserSpeexPacket usp = UserSpeexPacket.datagramPacketToUserSpeexPacket(packet);
abPCMData = decodeSpeexPacket(usp.getSpeexData(), 0, usp.getSpeexData().length);
m_srcDataLine.write(abPCMData, 0, abPCMData.length);
if (System.currentTimeMillis() >= (lPrevVolPrintTime + 500)) {
//System.out.println("Current volume: " + AudioUtil.getVolumeLevelForPCM22050Hz16Bit1Channel(abPCMData, 0, abPCMData.length));
lPrevVolPrintTime = System.currentTimeMillis();
}
}
catch (IOException iox) {
if (m_bServerRunning) {
System.out.println("Server socket broke: " + iox.getMessage());
stopServer();
}
}
}
I am working on a similar project. From everything I've read, and personal experience, your best option is to work with small bits of data and send them as soon as you can. You want any jitter buffering to be done on the side of the receiver.
It is typical for a VoIP application to send 50-100 packets per second. For uLaw encoding at 8000Hz, this would result in a packet size of 80-160 bytes. The reasoning for this is that some packets will inevitably be dropped, and you want the impact to the receiver to be as small as possible. So with 10ms or 20ms of audio data per packet, a dropped packet may result in a small hiccup, but not nearly as bad as losing 2k of audio data (~250ms).
Additionally, with a large packet size, you must accumulate all of the data at the sender before sending it. So given a typical network latency of 50ms, with 20ms of audio data per packet, the receiver is not going to hear what the sender says for a minimum of 70ms. Now imagine what happens when 250ms of audio is being sent at once. 270ms will elapse between the sender speaking and the receiver playing that audio.
Users seem to be more forgiving of packet loss here and there, which results in sub-par audio quality, because the audio quality of most telephones isn't that great to begin with. However, users are also used to very low latency on modern telephone circuits, so introducing a round-trip delay of even 250ms can be extremely frustrating.
Now, as far as implementing buffering, I have found a good strategy to use a Queue (whoops, using .NET here :)), and then wrap that in a class that tracks the desired minimum and maximum number of packets in the Queue. Use rigorous locking since you will most likely be accessing it from multiple threads. If the Queue "bottoms out" and has zero packets in it (buffer underrun), set a flag and return null until the packet count reaches your desired minimum. Your consumer will have to check for null being returned and not queue anything into the output buffer, however. Alternatively your consumer could keep track of the last packet and repeatedly enqueue it, which may cause looping audio, but in some cases that may "sound" better than silence. You will have to do this until the producer puts enough packets into the queue to reach the minimum. This will result in a longer period of silence for the user, but that is generally better accepted than short, frequent periods of silence (choppiness). If you get a burst of packets and the producer fills up the queue (reaching the desired maximum), you can either start ignoring the new packets, or drop enough packets out of the front of the queue to return to the minimum.
Picking those min/max values is tough though. You are trying to balance smooth audio (no underruns) with minimum latency between sender and receiver. VoIP is fun but it sure can be frustrating! Good luck!
I set up a server with a ServerSocket, connect to it with a client machine. They're directly networked through a switch and the ping time is <1ms.
Now, I try to push a "lot" of data from the client to the server through the socket's output stream. It takes 23 minutes to transfer 0.6Gb. I can push a much larger file in seconds via scp.
Any idea what I might be doing wrong? I'm basically just looping and calling writeInt on the socket. The speed issue doesn't matter where the data is coming from, even if I'm just sending a constant integer and not reading from disk.
I tried setting the send and receive buffer on both sides to 4Mb, no dice. I use a buffered stream for the reader and writer, no dice.
Am I missing something?
EDIT: code
Here's where I make the socket
System.out.println("Connecting to " + hostname);
serverAddr = InetAddress.getByName(hostname);
// connect and wait for port assignment
Socket initialSock = new Socket();
initialSock.connect(new InetSocketAddress(serverAddr, LDAMaster.LDA_MASTER_PORT));
int newPort = LDAHelper.readConnectionForwardPacket(new DataInputStream(initialSock.getInputStream()));
initialSock.close();
initialSock = null;
System.out.println("Forwarded to " + newPort);
// got my new port, connect to it
sock = new Socket();
sock.setReceiveBufferSize(RECEIVE_BUFFER_SIZE);
sock.setSendBufferSize(SEND_BUFFER_SIZE);
sock.connect(new InetSocketAddress(serverAddr, newPort));
System.out.println("Connected to " + hostname + ":" + newPort + " with buffers snd=" + sock.getSendBufferSize() + " rcv=" + sock.getReceiveBufferSize());
// get the MD5s
try {
byte[] dataMd5 = LDAHelper.md5File(dataFile),
indexMd5 = LDAHelper.md5File(indexFile);
long freeSpace = 90210; // ** TODO: actually set this **
output = new DataOutputStream(new BufferedOutputStream(sock.getOutputStream()));
input = new DataInputStream(new BufferedInputStream(sock.getInputStream()));
Here's where I do the server-side connection:
ServerSocket servSock = new ServerSocket();
servSock.setSoTimeout(SO_TIMEOUT);
servSock.setReuseAddress(true);
servSock.bind(new InetSocketAddress(LDA_MASTER_PORT));
int currPort = LDA_START_PORT;
while (true) {
try {
Socket conn = servSock.accept();
System.out.println("Got a connection. Sending them to port " + currPort);
clients.add(new MasterClientCommunicator(this, currPort));
clients.get(clients.size()-1).start();
Thread.sleep(500);
LDAHelper.sendConnectionForwardPacket(new DataOutputStream(conn.getOutputStream()), currPort);
currPort++;
} catch (SocketTimeoutException e) {
System.out.println("Done listening. Dispatching instructions.");
break;
}
catch (IOException e) {
e.printStackTrace();
}
catch (Exception e) {
e.printStackTrace();
}
}
Alright, here's where I'm shipping over ~0.6Gb of data.
public static void sendTermDeltaPacket(DataOutputStream out, TIntIntHashMap[] termDelta) throws IOException {
long bytesTransferred = 0, numZeros = 0;
long start = System.currentTimeMillis();
out.write(PACKET_TERM_DELTA); // header
out.flush();
for (int z=0; z < termDelta.length; z++) {
out.writeInt(termDelta[z].size()); // # of elements for each term
bytesTransferred += 4;
}
for (int z=0; z < termDelta.length; z++) {
for (int i=0; i < termDelta[z].size(); i++) {
out.writeInt(1);
out.writeInt(1);
}
}
It seems pretty straightforward so far...
You do not want to write single bytes when you are transferring large amounts of data.
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
public class Transfer {
public static void main(String[] args) {
final String largeFile = "/home/dr/test.dat"; // REPLACE
final int BUFFER_SIZE = 65536;
new Thread(new Runnable() {
public void run() {
try {
ServerSocket serverSocket = new ServerSocket(12345);
Socket clientSocket = serverSocket.accept();
long startTime = System.currentTimeMillis();
byte[] buffer = new byte[BUFFER_SIZE];
int read;
int totalRead = 0;
InputStream clientInputStream = clientSocket.getInputStream();
while ((read = clientInputStream.read(buffer)) != -1) {
totalRead += read;
}
long endTime = System.currentTimeMillis();
System.out.println(totalRead + " bytes read in " + (endTime - startTime) + " ms.");
} catch (IOException e) {
}
}
}).start();
new Thread(new Runnable() {
public void run() {
try {
Thread.sleep(1000);
Socket socket = new Socket("localhost", 12345);
FileInputStream fileInputStream = new FileInputStream(largeFile);
OutputStream socketOutputStream = socket.getOutputStream();
long startTime = System.currentTimeMillis();
byte[] buffer = new byte[BUFFER_SIZE];
int read;
int readTotal = 0;
while ((read = fileInputStream.read(buffer)) != -1) {
socketOutputStream.write(buffer, 0, read);
readTotal += read;
}
socketOutputStream.close();
fileInputStream.close();
socket.close();
long endTime = System.currentTimeMillis();
System.out.println(readTotal + " bytes written in " + (endTime - startTime) + " ms.");
} catch (Exception e) {
}
}
}).start();
}
}
This copies 1 GiB of data in short over 19 seconds on my machine. The key here is using the InputStream.read and OutputStream.write methods that accept a byte array as parameter. The size of the buffer is not really important, it just should be a bit larger than, say, 5. Experiment with BUFFER_SIZE above to see how it effects the speed but also keep in mind that it probably is different for every machine you are running this program on. 64 KiB seem to be a good compromise.
Hey, I figured I'd follow up for anyone that was interested.
Here's the bizarre moral of the story:
NEVER USE DataInputStream/DataOutputStream and sockets!!
If I wrap the socket in a BufferedOutputStream/BufferedInputStream, life is great. Writing to it raw is just fine.
But wrap the socket in a DataInputStream/DataOutputStream, or even have DataOutputStream(BufferedOutputStream(sock.getOutputStream())) is EXTREMELY SLOW.
An explanation for that would be really interesting to me. But after swapping everything in and out, this is what's up. Try it yourself if you don't believe me.
Thanks for all the quick help, though.
Maybe you should try sending ur data in chunks(frames) instead of writing each byte seperately. And align your frames with the TCP packet size for best performance.
Can you try doing this over loopback, it should then transfer the data in second.
If it takes minutes, there is something wrong with your application. If is only slow sending data over the internet it could be you network link which is slow.
My guess is that you have a 10 Mb/s network between your client and your server and this is why your transfer is going slowly. If this is the case, try using a DeflatoutOutputStream and an InflatorInputStream for your connection.
How are you implementing the receiving end? Please post your receiving code as well.
Since TCP is a reliable protocol, it will take steps to make sure the client is able to receive all of the data sent by the sender. This means that if your client cannot get the data out of the data receive buffer in time, then the sending side will simply stop sending more data until the client has a chance to read all the bytes in the receiving buffer.
If your receiving side is reading data one byte at a time, then your sender probably will spend a lot of time waiting for the receiving buffer to clear, hence the long transfer times. I'll suggest changing your receiving code to reading as many bytes as possible in each read operation . See if that will solve your problem.
Since I cannot yet comment on this site, I must write answer to #Erik here.
The problem is that DataOutputStream doesn't buffer. The whole Stream-thing in Java is based on decorators design pattern. So you could write
DataOutputStream out = new DataOutputStream(new BufferedOutputStream(socket.getOutputStream()));
It will wrap the original stream in a BufferedOutputStream which is more efficient, which is then wrapped into a DataOutputStream which offers additional nice features like writeInt(), writeLong() and so on.
#Erik: using DataXxxputStream is not the problem here. Problem is you were sending data in too small chunks. Using a buffer solved your problem because even you would write bit by bit the buffer would solve the problem.
Bombe's solution is much nicer, generic and faster.
You should download a good packet sniffer. I'm a huge fan of WireShark personally and I end up using it every time I do some socket programming. Just keep in mind you've got to have the client and server running on different systems in order to pick up any packets.
Things to try:
Is the CPU at 100% while the data is being sent? If so, use visualvm and do a CPU profiling to see where the time is spent
Use a SocketChannel from java.nio - these are generally faster since they can use native IO more easily - of course this only helps if your operation is CPU bound
If it's not CPU bound, there's something going wrong at the network level. Use a packet sniffer to analyze this.
I was using PrintWriter to send data. I removed that and sent data with BufferedOutputStream.send(String.getBytes()) and got about 10x faster sending.
How is your heap size set? I had a similar problem recently with the socket transfer of large amounts of data and just by looking at JConsole I realized that the application was spending most of its time doing full GCs.
Try -Xmx1g
USe Byte buffer for sending the data