Develop Reference

First two letters missing in file transfer in java

I am working on a client/server transfer protocol in java. The client is sending a simple text file, everything is going across the wire fine in wireshark, but once it gets to the server side, the first two letters are missing from the text file. I believe that it is overwriting the first buffer for some reason.
My goal is to make a while loop that reads the bytes in the buffer and then increments a count that'll place the next set of bytes....in the place if the ones already written
Here is the server's code that I currently have:
int bytesRead;
int current = 0;
InputStream in = s.getInputStream();
// Instantiating a new output stream object
OutputStream output = new FileOutputStream(myFile);
PrintStream stream = new PrintStream(output);
// Receive file 1024 bytes at a time
byte[] buffer = new byte[1024];
while ((bytesRead = in.read(buffer)) != -1) {
output.write(buffer, 0, bytesRead);
System.out.println(output.toString());
}

send large files over socket java

I am trying to transfer larger files over socket.I will be transferring the file in chunks.As shown in the code.link
int count;
byte[] buffer = new byte[8192];
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
before sending this file,i want to send an object which holds the details of the file.Using which stream should i send Object + File.
I am new to streams,can i get any sample code.
Can i send the byte length of the object first to read the object ,save it and send file data.is it possible,any sample code ?
Thanks
421

I'm not sure whether using Java's serialization mechanism is the best way to do a simple file transfer. As your question suggest, you try to avoid keeping the whole file in memory at any time. This can be done with objects using the Proxy pattern but if all you want to do is transfer the file, this might not be the most straight-forward solution. (Also, it will effectively tie your peer to be implemented in Java too.)
Instead, why not take a look at an extremely successful protocol that does exactly what you need: HTTP.
Content-Type: application/octet-stream
Content-Length: 542183
542183 bytes of data follow...
It should not be too hard for you to write a parser for the meta-data header.

You need to ensure the order of writing/reading. If write an object -> write raw bytes on client, then read an object -> read raw bytes on server. When reading, ObjectInputStream should be able to find the boundary of the serialized object data.
If you want to keep a socket connection long-live and use its streams multiple times, wrapping socket's Output/InputStream in a ObjectOutput/InputStream is not a good idea IMO. When you close a object stream, it closes the underlying stream as well.
So you may want to write the length of serialized object data first (file length is contained in the object so you don't need to write it explictly), e.g. 4 bytes of BigEndian encoded int. Then serialize the object into a ByteArrayOutputStream, and write the bytes in its buffer. On the server, read 4 bytes first, decode the bytes back to an int, and read that many bytes into a byte[], wrap the byte array with a ByteArrayInputStream and deserialize the object from it.
Write like this:
......
OutputStream out = socket.getOutputStream();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(fileInfoObject);
oos.close();
byte[] header = encodeInteger(baos.size());
out.write(header, 0, 4);
baos.writeTo(out);
// write the file to out just as your question shows
On the receiving side:
......
InputStream in = socket.getInputStream();
// read the int
byte[] header = new byte[4];
in.read(header, 0, 4);
int size = decodeInteger(header);
// read the object
byte[] objectbuf = new byte[size];
int count;
while((count += in.read(objectbuf)) < size); // not sure if this works...
ObjectInputStram ois = new ObjectInputStream(new ByteArrayInputStream(objectbuf));
Object fileInfoObject = ois.readObject();
ois.close();
// read the file
FileOutputStream fos = new FileOutputStream(new File("somefile"));
byte[] buffer = new byte[8192];
count = 0;
long left = castedFileInfoObject.fileSize;
// also not sure if this works, not tested.
int maxRead = buffer.length;
while (true) {
count = in.read(buffer, 0, maxRead);
left -= count;
if (left < 8192) {
maxRead = (int)left;
}
fos.write(buffer, 0, count);
if (left == 0) {
break;
}
}
I haven't tested the sample code in my answer.. just to show the idea.

Class MyClass should implement the Serializable interface. Then, an object of this class can be written to an ObjectOutputStream and read back from ObjectInputStream using writeObject and readObject methods (See below).
On Client:
Socket socket = new Socket(url, port);
OutputStream os = socket.getOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(os);
MyClass obj = new Myclass();
oos.writeObject(obj);
int count;
byte[] buffer = new byte[8192];
while ((count = in.read(buffer)) > 0) {
out.write(buffer, 0, count);
}
On server:
ServerSocket sSocket = new ServerSocket(port);
Socket socket = sSocket.accept();
InputStream is = socket.getInputStream();
ObjectInputStream ois = new ObjectInputStream(is);
MyClass obj = (MyClass)ois.readObject();
byte arr[];
try {
while(arr = (byte[])ois.readObject()) {
//do something with arr
}
} catch(java.io.EOFException) {
// End of data
}
If you need to send more data after the file is finished, you need a way to figure out the number of bytes the file consists of. Then, you can send the number of bytes beforehand over the socket to the server. On the server, read only that many bytes of information for the file and then do the same for the rest of the data you are going to send. This strategy of pre-sending the file size is recommended and is mostly used while doing any data transfer. If you can do that, you don't have to rely on catching java.io.EOFException to detect end of data.

How can I make sure I received whole file through socket stream?

Ok, So I'm making a Java program that has a server and client and I'm sending a Zip file from server to client. I have sending the file down, almost. But recieving I've found some inconsistency. My code isn't always getting the full archive. I'm guessing it's terminating before the BufferedReader has the full thing. Here's the code for the client:
public void run(String[] args) {
try {
clientSocket = new Socket("jacob-custom-pc", 4444);
out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedInputStream(clientSocket.getInputStream());
BufferedReader inRead = new BufferedReader(new InputStreamReader(in));
int size = 0;
while(true) {
if(in.available() > 0) {
byte[] array = new byte[in.available()];
in.read(array);
System.out.println(array.length);
System.out.println("recieved file!");
FileOutputStream fileOut = new FileOutputStream("out.zip");
fileOut.write(array);
fileOut.close();
break;
}
}
}
} catch(IOException e) {
e.printStackTrace();
System.exit(-1);
}
}
So how can I be sure the full archive is there before it writes the file?

On the sending side write the file size before you start writing the file. On the reading side Read the file size so you know how many bytes to expect. Then call read until you have gotten everything you expect. With network sockets it may take more than one call to read to get everything that was sent. This is especially true as your data gets larger.

HTTP sends a content-length: x+\n in bytes. This is elegant, it might throw a TimeoutException if the conn is broken.

You are using a TCP socket. The ZIP file is probably larger than the network MTU, so it will be split up into multiple packets and reassembled at the other side. Still, something like this might happen:
client connects
server starts sending. The ZIP file is bigger than the MTU and therefore split up into multiple packets.
client busy-waits in the while (true) until it gets the first packets.
client notices that data has arrived (in.available() > 0)
client reads all available data, writes it to the file and exits
the last packets arrive
So as you can see: Unless the client machine is crazily slow and the network is crazily fast and has a huge MTU, your code simply won't receive the entire file by design. That's how you built it.
A different approach: Prefix the data with the length.
Socket clientSocket = new Socket("jacob-custom-pc", 4444);
DataInputStream dataReader = new DataInputStream(clientSocket.getInputStream());
FileOutputStream out = new FileOutputStream("out.zip");
long size = dataReader.readLong();
long chunks = size / 1024;
int lastChunk = (int)(size - (chunks * 1024));
byte[] buf = new byte[1024];
for (long i = 0; i < chunks; i++) {
dataReader.read(buf);
out.write(buf);
}
dataReader.read(buf, 0, lastChunk);
out.write(buf, 0, lastChunk);
And the server uses DataOutputStream to send the size of the file before the actual file. I didn't test this, but it should work.

How can I make sure I received whole file through socket stream?
By fixing your code. You are using InputStream.available() as a test for end of stream. That's not what it's for. Change your copy loop to this, which is also a whole lot simpler:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Use with any buffer size greater than zero, typically 8192.

In.available() just tells you that there is no data to be consumed by in.read() without blocking (waiting) at the moment but it does not mean the end of stream. But, they may arrive into your PC at any time, with TCP/IP packet. Normally, you never use in.available(). In.read() suffices everything for the reading the stream entirely. The pattern for reading the input streams is
byte[] buf;
int size;
while ((size = in.read(buf)) != -1)
process(buf, size);
// end of stream has reached
This way you will read the stream entirely, until its end.
update If you want to read multiple files, then chunk you stream into "packets" and prefix every one with an integer size. You then read until size bytes is received instead of in.read = -1.
update2 Anyway, never use in.available for demarking between the chunks of data. If you do that, you imply that there is a time delay between incoming data pieces. You can do this only in the real-time systems. But Windows, Java and TCP/IP are all these layers incompatible with real-time.

sending image from java server to android app

I would like to send image file from java server to android app using this code:
Server(Java):
File file = new File("./clique.jpg");
FileInputStream stream = new FileInputStream(file);
DataOutputStream writer = new DataOutputStream(socket.getOutputStream());
byte[] contextB = new byte[4096];
int n;
int i = 0;
while ( (n=stream.read(contextB))!=-1 ){
writer.write(contextB, 0, n);
writer.flush();
System.out.println(n);
i+=n;
}
writer.flush();
stream.close();
android app:
DataInputStream reader = new DataInputStream(socket.getInputStream());
byte[] buffer = new byte[4096];
ByteArrayOutputStream content = new ByteArrayOutputStream();
int n;
int i = 0;
reader = new DataInputStream(socket.getInputStream());
while ( (n=reader.read(buffer)) != null){
content.write(buffer, 0, n);
content.flush();
}
Utility.CreateImageFile(content.toByteArray());
What I noticed is that in the android app n which is the number of read bytes is not 4096 while I am sending from server byte blocks of 4096 size,also I can not get n=-1 which is the end of stream,it blocks until I close the app then I get n=-1.

Regarding the number of bytes you read at a time has nothing to do with the number of bytes you write -it very much depends on the network conditions and will be variable with every chunk (basically as many bytes managed to be transmitted in short period of time between your reads as many you will get in the read chunk.
Regarding the end of stream - in your server code you have forgotten to close the output stream (you only close the stream which is input stream - you should also close the writer which in turn will close the underlying output stream.
Two comments:
1) I would really recommend to use Buffered Readers/Writers wrapping the writes/readers - the code you will get will be nicer and you will not have to create/manage buffers yourself.
2) Use try {} finally and close your streams in finally clauses - this is the best practice that will make sure that you will close the streams and free resources even in case of problems while reading/writing.

You got a problem in your android code:
while ( (n=reader.read(buffer)) != null) {
n can not be null.
Use writer.close() instead of writer.flush() after your loop on the server.

How to write file data correctly?

My application is unable to transfer data properly over a socket connection and write it to a file properly. Files over about 65,535 bytes get corrupted and are no longer recognized by the programs designed to run them.
I have been able to send small .doc and .txt files successfully, but .mp3 .wmv .m4a .avi and just about anything else does not work. Neither do larger docs.
I have looked all over the internet for a solution to this problem. I have repeatedly tweaked the I/O code to fix the problem but it still doesn't work! Here is the I/O code in the super class that handles sending and receiving files. If you need anymore information/other parts of code, let me know.
protected void sendFile() throws IOException {
byte[] bytes = new byte[(int) file.length()];
buffin = new BufferedInputStream(new FileInputStream(file));
int bytesRead = buffin.read(bytes,0,bytes.length);
System.out.println(bytesRead);
out = sock.getOutputStream();
out.write(bytes,0,fileBytes);
out.flush();
out.close();
}
protected void receiveFile() throws IOException {
byte[] bytes = new byte[fileBytes];
in = sock.getInputStream();
for(int i=0;i<fileBytes;i++) {
in.read(bytes);
}
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
buffout.write(bytes,0,fileBytes);
buffout.flush();
buffout.close();
}
UPDATED CODE (that works):
protected void sendFile() throws IOException {
if((file.length())<63000) {
byte[] bytes = new byte[(int)file.length()];
buffin = new BufferedInputStream(new FileInputStream(file));
buffin.read(bytes,0,bytes.length);
out = sock.getOutputStream();
out.write(bytes,0,bytes.length);
out.close();
} else {
byte[] bytes = new byte[32000];
buffin = new BufferedInputStream(new FileInputStream(file));
out = sock.getOutputStream();
int bytesRead;
while((bytesRead = buffin.read(bytes))>0) {
out.write(bytes,0,bytesRead);
}
out.close();
}
}
protected void receiveFile() throws IOException {
if(fileBytes<63000) {
byte[] bytes = new byte[32000];
in = sock.getInputStream();
System.out.println(in.available());
in.read(bytes,0,fileBytes);
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
buffout.write(bytes,0,bytes.length);
buffout.close();
} else {
byte[] bytes = new byte[16000];
in = sock.getInputStream();
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
int bytesRead;
while((bytesRead = in.read(bytes))>0) {
buffout.write(bytes,0,bytesRead);
}
buffout.close();
}
}

The issue is that you are sending only chunks of it. That is, you are only sending 64k of the file ever. If the file is ever larger then 64k the other end will never see it.
You want to continously read from the BufferedInputStream until the read() returns either less then the length or -1.

Your code is completely wrong. This is how to copy a stream in Java:
int count;
byte[] buffer = new byte[8192]; // more if you like but no need for it to be the entire file size
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
You should use this both when sending the file and when receiving the file. At present your sending method hopes that the entire file fits into memory; fits into INTEGER_MAX bytes; and is read in one chunk by the read method, without even checking the result. You can't assume any of those things. Your receive method is complete rubbish: it just keeps overwriting the same array, again without checking any read() results.
EDIT: Your revised code is just as bad, or worse. You are calling read() to check for EOS and then throwing that byte away, and then calling read() again and throwing away the read count it returns. You pointlessly have a different path for files < 64000, or 63000, or whatever it is, that has zero benefit except to give you two code paths to test, or possibly four, instead of one. The network only gives you 1460 bytes at a time at best anyway so what is the point? You already have (a) a BufferedInputStream with a default buffersize of 8192, and (b) my code that uses a byte[] buffer of any size you like. My code above works for any amount of data in two lines of executable code. Yours is 20. QED.

I suggest that you use some good library to read and write file contents as well as socket read/write. For example Apache Commons IO. If you insist on writig code yourself, do it smaller chunks rather than the whole file at once.

You have to consider that InputStream.read returns the number of bytes read which may be less than the total number of bytes in the file.
You would probably be better off just letting something like CopyUtils.copy take care of this for you.

You need to loop until bytesRead < 0. You need to make sure that fileBytes is => than the transferred file.
protected void receiveFile() throws IOException {
byte [] bytes = new byte [fileBytes];
InputStream is = sock.getInputStream();
FileOutputStream fos = new FileOutputStream("/Datawire/"+fileName);
BufferedOutputStream bos = new BufferedOutputStream(fos);
int bytesRead = is.read(bytes,0,bytes.length);
int current = bytesRead;
do {
bytesRead =
is.read(bytes, current, (bytes.length-current));
if(bytesRead >= 0) current += bytesRead;
} while(bytesRead > -1);
bos.write(bytes, 0 , current);
bos.flush();
bos.close();
}