Ok, So I'm making a Java program that has a server and client and I'm sending a Zip file from server to client. I have sending the file down, almost. But recieving I've found some inconsistency. My code isn't always getting the full archive. I'm guessing it's terminating before the BufferedReader has the full thing. Here's the code for the client:
public void run(String[] args) {
try {
clientSocket = new Socket("jacob-custom-pc", 4444);
out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedInputStream(clientSocket.getInputStream());
BufferedReader inRead = new BufferedReader(new InputStreamReader(in));
int size = 0;
while(true) {
if(in.available() > 0) {
byte[] array = new byte[in.available()];
in.read(array);
System.out.println(array.length);
System.out.println("recieved file!");
FileOutputStream fileOut = new FileOutputStream("out.zip");
fileOut.write(array);
fileOut.close();
break;
}
}
}
} catch(IOException e) {
e.printStackTrace();
System.exit(-1);
}
}
So how can I be sure the full archive is there before it writes the file?
On the sending side write the file size before you start writing the file. On the reading side Read the file size so you know how many bytes to expect. Then call read until you have gotten everything you expect. With network sockets it may take more than one call to read to get everything that was sent. This is especially true as your data gets larger.
HTTP sends a content-length: x+\n in bytes. This is elegant, it might throw a TimeoutException if the conn is broken.
You are using a TCP socket. The ZIP file is probably larger than the network MTU, so it will be split up into multiple packets and reassembled at the other side. Still, something like this might happen:
client connects
server starts sending. The ZIP file is bigger than the MTU and therefore split up into multiple packets.
client busy-waits in the while (true) until it gets the first packets.
client notices that data has arrived (in.available() > 0)
client reads all available data, writes it to the file and exits
the last packets arrive
So as you can see: Unless the client machine is crazily slow and the network is crazily fast and has a huge MTU, your code simply won't receive the entire file by design. That's how you built it.
A different approach: Prefix the data with the length.
Socket clientSocket = new Socket("jacob-custom-pc", 4444);
DataInputStream dataReader = new DataInputStream(clientSocket.getInputStream());
FileOutputStream out = new FileOutputStream("out.zip");
long size = dataReader.readLong();
long chunks = size / 1024;
int lastChunk = (int)(size - (chunks * 1024));
byte[] buf = new byte[1024];
for (long i = 0; i < chunks; i++) {
dataReader.read(buf);
out.write(buf);
}
dataReader.read(buf, 0, lastChunk);
out.write(buf, 0, lastChunk);
And the server uses DataOutputStream to send the size of the file before the actual file. I didn't test this, but it should work.
How can I make sure I received whole file through socket stream?
By fixing your code. You are using InputStream.available() as a test for end of stream. That's not what it's for. Change your copy loop to this, which is also a whole lot simpler:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Use with any buffer size greater than zero, typically 8192.
In.available() just tells you that there is no data to be consumed by in.read() without blocking (waiting) at the moment but it does not mean the end of stream. But, they may arrive into your PC at any time, with TCP/IP packet. Normally, you never use in.available(). In.read() suffices everything for the reading the stream entirely. The pattern for reading the input streams is
byte[] buf;
int size;
while ((size = in.read(buf)) != -1)
process(buf, size);
// end of stream has reached
This way you will read the stream entirely, until its end.
update If you want to read multiple files, then chunk you stream into "packets" and prefix every one with an integer size. You then read until size bytes is received instead of in.read = -1.
update2 Anyway, never use in.available for demarking between the chunks of data. If you do that, you imply that there is a time delay between incoming data pieces. You can do this only in the real-time systems. But Windows, Java and TCP/IP are all these layers incompatible with real-time.
Related
I've been trying to get this working for a few days now, but I've had no success.
I want to send a file over socket client/server. The only difference is: I want to send an object that contains the file bytes.
So the client loads a file, reads chunks of 1024 bytes, store them in a object, and send the object to the server. Since the file can be larger than 1024 bytes, I want to send the object repeatedly but with different bytes stored in them (as the buffer reads it). On the server, I want to compose the array of bytes and save it as a file.
The reason I'm using 1024 is because I want to avoid any sort of out of memory error, if the file is, let's say, 4 GB in size.
I tried doing the following on the client:
File file = new File("C:\\test\\test.txt");
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
ObjectOutputStream oos = new ObjectOutputStream(socket.getOutputStream());
byte[] bytes = new byte[1024];
FileTest ft = new FileTest();
ft.setName("Testing");
int counttest = 1;
while (bis.read(bytes) > 0) {
ft.setCounttest(counttest);
ft.setBytes(bytes);
oos.writeObject(ft);
counttest += 1;
}
On the server:
int bufferSize = socket.getReceiveBufferSize();
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream("C:\\test\\test2.txt"));
ObjectInputStream ois = new ObjectInputStream(socket.getInputStream());
byte[] bytes = new byte[bufferSize];
while (true) {
FileTest ft = (FileTest) ois.readObject();
if (ft != null) {
System.out.println(ft.getName());
bos.write(ft.getBytes());
}
}
So I tested sending a txt file with a sequence of numbers and the test2.txt file produced by the server came out only with the first 1024 chunk of bytes repeated twice. Also, the counttest integer never increases when received in the server.
Any idea how to accomplish this?
Thanks in advance.
You are running into the effects of the ObjectOutputStream's attempt to preserve object identity. Repeatedly writing the same object instance will result on the same instance on the receiver's end. This is generally a good thing, but confusing if you are modifying the object on the sender's end and expecting those modifications to show up on the receiver's end.
Thus, you have two issues:
In order to send the data each time, you either need to create a new FileTest instance each time, or use the writeUnshared() method.
Due to this identity preserving behavior, you will need to periodically reset() the ObjectOutputStream in order to keep all these instances from being held forever (and potentially leading to an OOME on the client or server).
When I try to send a large file from server by splitting it, some of the packages don't arrive at the client... as you can see in the console output
http://s7.postimg.org/94yjfame3/error.png
the client receive only 19799.. bytes , and the server sent 62800.. bytes.
the code is too long to past here... but here are the basics:
// server side -> send data
BufferedOutputStream out = new BufferedOutputStream(socket.getOutputStream());
byte[] somePackageInfo= new byte[500];
byte[] streamOut = new byte[20000];
while(getDataFromLargeFile(somePackageInfo,streamOut) != 0) {
out.write(somePackageInfo,0,500);
out.write(streamOut);
out.flush();
}
out.write(0);
out.flush();
// client side -> get data
BufferedInputStream in = new BufferedInputStream(socket.getInputStream());
byte[] somePackageInfo= new byte[500];
byte[] streamIn= new byte[20000];
while(true) {
if(in.read(somePackageInfo,0,500) == 0) break;
in.read(streamIn);
saveDataToLargeFile(somePackageInfo,streamIn);
}
I tried to slow down the transfer (sleep(500)) but only most of the packages arrived.
tried to remove the flush() but still only most of the packages arrived.
what causes this problem and how can i fix it?
Your copy code is wrong. You are ignoring the count returned by read, and assuming that it fills the buffer. It isn't required to do that. See the Javadoc.
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Use with any buffer size greater than zero, typically 8192. Use at both ends.
Adding sleeps is literally a waste of time.
This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 7 years ago.
Presently My Server program can able to receive the file from client socket and able to save that received file in server machine.
But I am need to receive many files from client socket to server socket without closing and opening the socket connection every time.
I have written the code, this is working fine. But in this I am closing and opening the server and client socket connection in every iteration. But I need to do this without connecting and disconnecting both the sockets every time.
Please guide me seniors...
My Server code:
int img_count=1;
int bytesRead;
int current = 0;
byte [] mybytearray = new byte [100000];
InputStream is = null;
FileOutputStream fos = null;
BufferedOutputStream bos = null;
Socket sock=null;
// create socket
ServerSocket servsock = new ServerSocket(6668);
System.out.println("Waiting... for client req");
int i=0;
for ( i=0; i<9; i++)
{
sock = servsock.accept(); // Waiting for Client
String fname = "Image000"+(img_count++)+".JPG";
String fpath = "C:/RX_images/"+fname; // Image saving path
File myFile = new File (fpath);
is = sock.getInputStream();
fos = new FileOutputStream(myFile);
bos = new BufferedOutputStream(fos);
bytesRead = is.read(mybytearray,0,mybytearray.length);
current = bytesRead;
do {
bytesRead =
is.read(mybytearray, current, (mybytearray.length-current));
if(bytesRead >= 0) current += bytesRead;
} while(bytesRead > -1);
bos.write(mybytearray, 0 , current);
bos.flush();
fos.flush();
fos.close();
bos.close();
is.close();
sock.close();
} // End of for loop
servsock.close();
System.out.println("Received : "+ (i++)+ " Images");
My Client Code:
int i=0;
int img_count=1;
FileInputStream fis=null;
BufferedInputStream bis=null;
OutputStream os=null;
Socket client=null;
System.out.println("Sending...");
for ( i=0; i<9; i++)
{
client = new Socket("192.168.1.54",6668);
String fname = "Image000"+(img_count++)+".JPG";
String fpath = "C:/Tx_Images/"+fname; // Image path
File myFile = new File (fpath);
byte [] mybytearray = new byte [(int)myFile.length()];
fis = new FileInputStream(myFile);
bis = new BufferedInputStream(fis);
bis.read(mybytearray,0,mybytearray.length);
os = client.getOutputStream();
os.write(mybytearray,0,mybytearray.length);
bis.close();
fis.close();
os.flush();
os.close();
client.close();
Thread.sleep(2000);
} // End of for loop
System.out.println("\n Sent : "+(i++)+" Images");
I am very new to java,
Help me please....
Since the socket is just a stream of bytes, in order to handle more than one file you are going to have to construct a simple protocol of some sort. In other words, the sender will have to send bytes that differentiate between the bytes in one file and the bytes in another. Since you are sending binary data, there is no series of bytes you can send to "mark" the beginning and/or/ending -- for example if you send 4 zero bytes at the end, that might be data and so the receiver cannot be sure if it's a marker or data. Two ways to handle it come to mind offhand -- break your file up into sections that are a maximum of N bytes, and send the sections one at a time. You will have to have a count of the bytes in each section, since at least one section will not have the same number of bytes as all other sections. Alternately,y you could count the bytes in the file and start with bytes that give that count, so the receiver knows how many bytes to expect. While you are giving the count, you could also give information such as the name and the type of file, if you wanted. Good luck.
This question really depends on whether you need the client to keep the connection open, or not. Typically you just need to keep the server side listening, and it's ok for the client to reconnect each time it needs to send a file.
Use an ExecutorService to keep the server side going and handle multiple connections with separate threads. Then just have the client connect and send what it needs to send and disconnect. See this question for a quick example: Multithreading Socket communication Client/Server
Also, look at how they close resources (finally) and stop the server in that example too. That is not related to your question, but you'll want to make your I/O and error handling more robust as well.
If you really do require that the server and client stay connected and send multiple files (or whatever data) then you'll need to implement some sort of a protocol as rcook notes, and you'll need to go deeper into networking and have a heartbeat and such. And, even if you do that, the client still needs to be smart enough to try to reconnect if the socket is closed, etc.
Just make simple protocol like:
File Name\r\n
File Size\r\n
File Data\r\n
File Name\r\n
File Size\r\n
File Data\r\n
....
I hope you will understand this. You can send file information initially then server will parse this file information, and make your server to read number bytes as you specified in file information. These will enable you to see file end marker and when to begin new file. BUT you must know file size before.
This will not work for data streams which have unknown length.
Make your server to read number of bytes you will be specifying, so server can know when to end file writing and begin new file or whether file is fully received before socket closes...
I would like to send image file from java server to android app using this code:
Server(Java):
File file = new File("./clique.jpg");
FileInputStream stream = new FileInputStream(file);
DataOutputStream writer = new DataOutputStream(socket.getOutputStream());
byte[] contextB = new byte[4096];
int n;
int i = 0;
while ( (n=stream.read(contextB))!=-1 ){
writer.write(contextB, 0, n);
writer.flush();
System.out.println(n);
i+=n;
}
writer.flush();
stream.close();
android app:
DataInputStream reader = new DataInputStream(socket.getInputStream());
byte[] buffer = new byte[4096];
ByteArrayOutputStream content = new ByteArrayOutputStream();
int n;
int i = 0;
reader = new DataInputStream(socket.getInputStream());
while ( (n=reader.read(buffer)) != null){
content.write(buffer, 0, n);
content.flush();
}
Utility.CreateImageFile(content.toByteArray());
What I noticed is that in the android app n which is the number of read bytes is not 4096 while I am sending from server byte blocks of 4096 size,also I can not get n=-1 which is the end of stream,it blocks until I close the app then I get n=-1.
Regarding the number of bytes you read at a time has nothing to do with the number of bytes you write -it very much depends on the network conditions and will be variable with every chunk (basically as many bytes managed to be transmitted in short period of time between your reads as many you will get in the read chunk.
Regarding the end of stream - in your server code you have forgotten to close the output stream (you only close the stream which is input stream - you should also close the writer which in turn will close the underlying output stream.
Two comments:
1) I would really recommend to use Buffered Readers/Writers wrapping the writes/readers - the code you will get will be nicer and you will not have to create/manage buffers yourself.
2) Use try {} finally and close your streams in finally clauses - this is the best practice that will make sure that you will close the streams and free resources even in case of problems while reading/writing.
You got a problem in your android code:
while ( (n=reader.read(buffer)) != null) {
n can not be null.
Use writer.close() instead of writer.flush() after your loop on the server.
I´m having a problem, in my server, after I send a file with X bytes, I send a string saying this file is over and another file is coming, like
FILE: a SIZE: Y\r\n
send Y bytes
FILE a FINISHED\r\n
FILE b SIZE: Z\r\n
send Z byes
FILE b FINISHED\r\n
FILES FINISHED\r\n
In my client it does not recive properly.
I use readline() to get the command lines after reading Y or Z bytes from the socket.
With one file it works fine, with multiple files it rarely works (yeah, I dont know how it worked once or twice)
Here are some code I use to transfer binary
public static void readInputStreamToFile(InputStream is, FileOutputStream fout,
long size, int bufferSize) throws Exception
{
byte[] buffer = new byte[bufferSize];
long curRead = 0;
long totalRead = 0;
long sizeToRead = size;
while(totalRead < sizeToRead)
{
if(totalRead + buffer.length <= sizeToRead)
{
curRead = is.read(buffer);
}
else
{
curRead = is.read(buffer, 0, (int)(sizeToRead - totalRead));
}
totalRead = totalRead + curRead;
fout.write(buffer, 0, (int) curRead);
}
}
public static void writeFileInputStreamToOutputStream(FileInputStream in, OutputStream out, int bufferSize) throws Exception
{
byte[] buffer = new byte[bufferSize];
int count = 0;
while((count = in.read(buffer)) != -1)
{
out.write(buffer, 0, count);
}
}
just for note I could solve replacing readline to this code:
ByteArrayOutputStream ba = new ByteArrayOutputStream();
int ch;
while(true)
{
ch = is.read();
if(ch == -1)
throw new IOException("Conecção finalizada");
if(ch == 13)
{
ch = is.read();
if(ch == 10)
return new String(ba.toByteArray(), "ISO-8859-1");
else
ba.write(13);
}
ba.write(ch);
}
PS: "is" is my input stream from socket: socket.getInputStream();
still I dont know if its the best implementation to do, im tryinf to figure out
There's no readLine() calls in the code here, but to answer your question; Yes, calling BufferedReader.readLine() might very well leave stuff around in its internal buffer. It's buffering the input.
If you wrap one of your InputStream in a BufferedReader, you can't really get much sane behavior if you read from the BufferedReader and then later on read from the InputStream.
You could read bytes from your InputStream and parse out a text line from that by looking for a pair of \r\n bytes. When you got a line saying "FILE: a SIZE: Y\r\n" , you go on as usual, except the buffer you used to parse lines might contain the first few bytes of your file, so write those bytes out first.
Or you use the idea of FTP and use one TCP stream for commands and one TCP stream for the actual transfer, reading from the command stream with a BufferedReader.readLine(), and reading the data as you already do with an InputStream.
Yes, the main point of a BufferedReader is to buffer the data. It is reading input from its underlying Reader in bigger chunks to avoid having multiple small reads.
That it has a readLine() method is just a nice bonus which is made easily possible by the buffering.
You may want to use a DataInputStream (on top of a BufferedInputStream) and it's readLine() method, if you really have to mix text and binary data over the same connection - read the data from the same DataInputStream. (But take care about the encoding here.)
Call flush() on the OutputStream after you've written data that you want to be certain has been sent. So essentially at the end of each file call flush().
I guess you must flush your output stream in order to make sure any buffered bytes are properly sent down the stream. Closing the stream will equally have this process run.
The Javadocs for flush say:
Flushes this output stream and forces
any buffered output bytes to be
written out. The general contract of
flush is that calling it is an
indication that, if any bytes
previously written have been buffered
by the implementation of the output
stream, such bytes should immediately
be written to their intended
destination.