everyone. I am a beginner in network programming. Currently, I want to do an experiment which sends a file(~2M bytes) from Android phone to Ubuntu server. How can I send it at the highest speed? I have tried something like using Bufferedreader in Java, reading each byte out of the file and sending that single byte to server through socket "outputstream write" function. This way seems to cost too much time. I notice that if in the same network condition, I send same file using some Immediate Messenger tools,like Skype;it is much quicker than I did. Does anybody know the API or implementation protocol beneath those Immediate Messenger software?
I probably need to call other efficient APIs other than socket? I am also trying to read the whole file into a byte array and then call "socket write" function to send the huge byte array to server for a just single time.Although when I receive it at server side to found that there are a lot of "padding zeros" distributed in my original data, the whole transfer seems to cost less time than the "single byte transfer" method. Anybody has any advice on this?Thanks a lot!
Thanks for everybody's answer. I think I just made a silly mistake. The real reason for my slow transmission using TCP socket is that every time I just read a single byte out of the file and call "void write(int b)" to send this single byte to server.This method is very time consuming. Now every time I try to read 256 bytes out of file and send these 256 bytes through "void write(byte[] b,int off,int len)";in this way, the transmission is pretty fast.Therefore, it is not the problem of TCP itself. It is my mistake of calling the wrong API. I have not tried UDP yet. But I think it is also a good choice.Thanks again,everybody.
Short of reinventing TCP/IP the fastest way to send will be via UDP, if your connection is good enough (few losses) than you on;y need to implement packet sequence so the sender prepends sequence number to the data packets, the receiver keeps track of missed packets and requests them again after all data is sent. After all data is received the receiver can reassemble the complete file.
This is simplistic implementation of TCP over UDP.
Related
I have recently tried my hands over Socket Programming by creating an SSL Socket used to stream data live from a server with no success of course. When I analyze the data packets through Wireshark, I realize the size of request data has been magnified n number of times in the packet and hence the request reaches the server in fragments where as the actual JSON request is a handful of bytes and should reach the server in a single shot.
Any help would be appreciated.
Wrap a BufferedOutputStream around the SSLSocket's output stream, and don't flush it until you really gotta, which is usually not until you're about to read the reply. Otherwise you can be sending one byte at a time to the SSLSocket, which becomes one SSL message per byte, which can expand the data by more than 40x.
However:
the request reaches the server in fragments
That can happen any time. The server has to be able to cope with receiving data as badly fragmented as one byte at a time.
where as the actual JSON request is a handful of bytes and should reach the server in a single shot.
There is no such guarantee in TCP.
TCP_NODELAY is an option to enable quick sending of TCP packets, regardless of their size. This is very useful option when speed matters, however, I'm curious what it will do to this:
Socket socket = [some socket];
socket.setTcpNoDelay(true);
OutputStream out = socket.getOutputStream();
out.write(byteArray1);
out.write(byteArray2);
out.write(byteArray3);
out.flush();
I tried to find what flush actually does on a SocketOutputStream, but as far as I know now, it doesn't do anything. I had hoped it would tell the socket "send all your buffered data NOW", but, unfortunately, no it doesn't.
My question is: are these 3 byte arrays sent in one packet? I know you don't have much control over how TCP constructs a network packet, but is there any way to tell the socket to (at least try to) pack these byte arrays, so network overhead is avoided? Could manually packing the byte arrays and sending them in one call to write help?
My question is: are these 3 byte arrays sent in one packet?
As you have disabled the Nagle algorithm, almost certainly not, but you can't be 100% sure.
I know you don't have much control over how TCP constructs a network packet, but is there any way to tell the socket to (at least try to) pack these byte arrays
Yes. Don't disable the Nagle algorithm.
so network overhead is avoided? Could manually packing the byte arrays and sending them in one call to write help?
Yes, or simpler still just wrap the socket output stream in a BufferedOutputStream and call flush() when you want the data to be sent, as per your present code. You are correct that flush() does nothing on a socket output stream, but it flushes a BufferedOutputStream.
Could manually packing the byte arrays and sending them in one call to write help?
Yes, send them all in a single call to write. This will maximize the chances that your bytes will be sent in a single packet.
Of course, you can never know, since there are too many variables involved - different OSs and lots of different networking gear between you and your peer, but if you give the OS the ability to pack everything together, it will generally try to.
If you disable nagle and make seperate system calls (remember, the OS controls the sockets, not your appliation or java), you're asking the OS to send them individually. The OS has no idea you're about to call write again with some more data.
I read this question about the error that I'm getting and I learned that UDP data payloads can't be more than 64k. The suggestions that I've read are to use TCP, but that is not an option in this particular case. I am interfacing with an external system that is transmitting data over UDP, but I don't have access to that external system at this time, so I'm simulating it.
I have data messages that are upwards of 1,400,000 bytes in some instances and it's a requirement that the UDP protocol is used. I am not able to change protocols (I would much rather use TCP or a reliable protocol build on UDP). Instead, I have to find a way to transmit large payloads over UDP from a test application into the system that I am building and to read those large payloads in the system that I'm building for processing. I don't have to worry about dropped packets, either - if I don't get the datagram, I don't care - just wait for the next payload to arrive. If it's incomplete or missing, just throw it all away and continue waiting. I also don't know the size of the datagram in advance (they range of a few hundred bytes to 1,400,000+ bytes.
I've already set my send and receive buffer sizes large enough, but that's not sufficient. What else can I do?
UDP packets have a 16 bit length field. It's nothing to do with Java. They cannot be bigger, period. If the server you are talking to is immutable, you are stuck with what you can fit into a packet.
If you can change the server and thus the protocol, you can more or less reimplement TCP for yourself. Since UDP is defined to be unreliable, you need the full retransmission mechanism to cope with packets that are dropped in the network somewhere. So, you have to split the 'message' into chunks, send the chunks, and have a protocol for requesting retransmission of lost chunks.
It's a requirement ...
The requirement should also therefore dictate the packetization technique. You need more information about the external system and its protocol. Note that the maximum IPv4 UDP payload Is 65535-28 bytes, and the maximum practical payload is < 1500 bytes once a router gets involved.
Hi I want to send a fixed amount of data (say 5MB) from server to Android client over TCP using the Java programming language. The data doesn't matter it will be dropped at the client I am only doing this to do performance measurements on the phone.
Can anyone recommend a good way of accomplishing this? How can I have the server continually sending a large amount of data? Without the intermittent behaviour of reading a line from a file, then sending those bytes... then reading another line and sending those ones.
Ideas? Cheers.
You don't need to send a file, you could just send a block of data.
// server which sends 5 MB on connection.
ServerSocket ss =
Socket s = ss.accept();
s.getOutputStream().write(new byte[5*1024*1024]);
s.close();
You could have a byte array of size 1024 and loop sending it out the socket over and over. The data doesn't need to be unique. You could use a larger array if you felt a KB isn't big enough for you.
Sounds like you were asking about the server there. Were you also wondering about the client?
Does anyone have any tips on how to calculate the bandwidth usage of a socket?
For example, as I send data over a socket to the server I am connected to, I want to show the Kb/s that is being sent.
Google search didn't reveal anything useful. Maybe I'm searching the wrong terms.
The best you're probably going to be able to easily do is to record when you start writing and then count bytes you've successfully sent to the Socket.getOutputStream.write() method. For a small amount of data, that will be very inaccurate as it's just filling up the OS's transmission buffer which will initially take bytes much faster than it actually sends them.
It should amortize to essentially the correct rate over a fairly large amount of data, however.