Develop Reference

send large files over socket java

I am trying to transfer larger files over socket.I will be transferring the file in chunks.As shown in the code.link
int count;
byte[] buffer = new byte[8192];
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
before sending this file,i want to send an object which holds the details of the file.Using which stream should i send Object + File.
I am new to streams,can i get any sample code.
Can i send the byte length of the object first to read the object ,save it and send file data.is it possible,any sample code ?
Thanks
421

I'm not sure whether using Java's serialization mechanism is the best way to do a simple file transfer. As your question suggest, you try to avoid keeping the whole file in memory at any time. This can be done with objects using the Proxy pattern but if all you want to do is transfer the file, this might not be the most straight-forward solution. (Also, it will effectively tie your peer to be implemented in Java too.)
Instead, why not take a look at an extremely successful protocol that does exactly what you need: HTTP.
Content-Type: application/octet-stream
Content-Length: 542183
542183 bytes of data follow...
It should not be too hard for you to write a parser for the meta-data header.

You need to ensure the order of writing/reading. If write an object -> write raw bytes on client, then read an object -> read raw bytes on server. When reading, ObjectInputStream should be able to find the boundary of the serialized object data.
If you want to keep a socket connection long-live and use its streams multiple times, wrapping socket's Output/InputStream in a ObjectOutput/InputStream is not a good idea IMO. When you close a object stream, it closes the underlying stream as well.
So you may want to write the length of serialized object data first (file length is contained in the object so you don't need to write it explictly), e.g. 4 bytes of BigEndian encoded int. Then serialize the object into a ByteArrayOutputStream, and write the bytes in its buffer. On the server, read 4 bytes first, decode the bytes back to an int, and read that many bytes into a byte[], wrap the byte array with a ByteArrayInputStream and deserialize the object from it.
Write like this:
......
OutputStream out = socket.getOutputStream();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(fileInfoObject);
oos.close();
byte[] header = encodeInteger(baos.size());
out.write(header, 0, 4);
baos.writeTo(out);
// write the file to out just as your question shows
On the receiving side:
......
InputStream in = socket.getInputStream();
// read the int
byte[] header = new byte[4];
in.read(header, 0, 4);
int size = decodeInteger(header);
// read the object
byte[] objectbuf = new byte[size];
int count;
while((count += in.read(objectbuf)) < size); // not sure if this works...
ObjectInputStram ois = new ObjectInputStream(new ByteArrayInputStream(objectbuf));
Object fileInfoObject = ois.readObject();
ois.close();
// read the file
FileOutputStream fos = new FileOutputStream(new File("somefile"));
byte[] buffer = new byte[8192];
count = 0;
long left = castedFileInfoObject.fileSize;
// also not sure if this works, not tested.
int maxRead = buffer.length;
while (true) {
count = in.read(buffer, 0, maxRead);
left -= count;
if (left < 8192) {
maxRead = (int)left;
}
fos.write(buffer, 0, count);
if (left == 0) {
break;
}
}
I haven't tested the sample code in my answer.. just to show the idea.

Class MyClass should implement the Serializable interface. Then, an object of this class can be written to an ObjectOutputStream and read back from ObjectInputStream using writeObject and readObject methods (See below).
On Client:
Socket socket = new Socket(url, port);
OutputStream os = socket.getOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(os);
MyClass obj = new Myclass();
oos.writeObject(obj);
int count;
byte[] buffer = new byte[8192];
while ((count = in.read(buffer)) > 0) {
out.write(buffer, 0, count);
}
On server:
ServerSocket sSocket = new ServerSocket(port);
Socket socket = sSocket.accept();
InputStream is = socket.getInputStream();
ObjectInputStream ois = new ObjectInputStream(is);
MyClass obj = (MyClass)ois.readObject();
byte arr[];
try {
while(arr = (byte[])ois.readObject()) {
//do something with arr
}
} catch(java.io.EOFException) {
// End of data
}
If you need to send more data after the file is finished, you need a way to figure out the number of bytes the file consists of. Then, you can send the number of bytes beforehand over the socket to the server. On the server, read only that many bytes of information for the file and then do the same for the rest of the data you are going to send. This strategy of pre-sending the file size is recommended and is mostly used while doing any data transfer. If you can do that, you don't have to rely on catching java.io.EOFException to detect end of data.

How to (without reconnecting and closing socket everytime) receive many files from socket in Android or java [duplicate]

This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 7 years ago.
Presently My Server program can able to receive the file from client socket and able to save that received file in server machine.
But I am need to receive many files from client socket to server socket without closing and opening the socket connection every time.
I have written the code, this is working fine. But in this I am closing and opening the server and client socket connection in every iteration. But I need to do this without connecting and disconnecting both the sockets every time.
Please guide me seniors...
My Server code:
int img_count=1;
int bytesRead;
int current = 0;
byte [] mybytearray = new byte [100000];
InputStream is = null;
FileOutputStream fos = null;
BufferedOutputStream bos = null;
Socket sock=null;
// create socket
ServerSocket servsock = new ServerSocket(6668);
System.out.println("Waiting... for client req");
int i=0;
for ( i=0; i<9; i++)
{
sock = servsock.accept(); // Waiting for Client
String fname = "Image000"+(img_count++)+".JPG";
String fpath = "C:/RX_images/"+fname; // Image saving path
File myFile = new File (fpath);
is = sock.getInputStream();
fos = new FileOutputStream(myFile);
bos = new BufferedOutputStream(fos);
bytesRead = is.read(mybytearray,0,mybytearray.length);
current = bytesRead;
do {
bytesRead =
is.read(mybytearray, current, (mybytearray.length-current));
if(bytesRead >= 0) current += bytesRead;
} while(bytesRead > -1);
bos.write(mybytearray, 0 , current);
bos.flush();
fos.flush();
fos.close();
bos.close();
is.close();
sock.close();
} // End of for loop
servsock.close();
System.out.println("Received : "+ (i++)+ " Images");
My Client Code:
int i=0;
int img_count=1;
FileInputStream fis=null;
BufferedInputStream bis=null;
OutputStream os=null;
Socket client=null;
System.out.println("Sending...");
for ( i=0; i<9; i++)
{
client = new Socket("192.168.1.54",6668);
String fname = "Image000"+(img_count++)+".JPG";
String fpath = "C:/Tx_Images/"+fname; // Image path
File myFile = new File (fpath);
byte [] mybytearray = new byte [(int)myFile.length()];
fis = new FileInputStream(myFile);
bis = new BufferedInputStream(fis);
bis.read(mybytearray,0,mybytearray.length);
os = client.getOutputStream();
os.write(mybytearray,0,mybytearray.length);
bis.close();
fis.close();
os.flush();
os.close();
client.close();
Thread.sleep(2000);
} // End of for loop
System.out.println("\n Sent : "+(i++)+" Images");
I am very new to java,
Help me please....

Since the socket is just a stream of bytes, in order to handle more than one file you are going to have to construct a simple protocol of some sort. In other words, the sender will have to send bytes that differentiate between the bytes in one file and the bytes in another. Since you are sending binary data, there is no series of bytes you can send to "mark" the beginning and/or/ending -- for example if you send 4 zero bytes at the end, that might be data and so the receiver cannot be sure if it's a marker or data. Two ways to handle it come to mind offhand -- break your file up into sections that are a maximum of N bytes, and send the sections one at a time. You will have to have a count of the bytes in each section, since at least one section will not have the same number of bytes as all other sections. Alternately,y you could count the bytes in the file and start with bytes that give that count, so the receiver knows how many bytes to expect. While you are giving the count, you could also give information such as the name and the type of file, if you wanted. Good luck.

This question really depends on whether you need the client to keep the connection open, or not. Typically you just need to keep the server side listening, and it's ok for the client to reconnect each time it needs to send a file.
Use an ExecutorService to keep the server side going and handle multiple connections with separate threads. Then just have the client connect and send what it needs to send and disconnect. See this question for a quick example: Multithreading Socket communication Client/Server
Also, look at how they close resources (finally) and stop the server in that example too. That is not related to your question, but you'll want to make your I/O and error handling more robust as well.
If you really do require that the server and client stay connected and send multiple files (or whatever data) then you'll need to implement some sort of a protocol as rcook notes, and you'll need to go deeper into networking and have a heartbeat and such. And, even if you do that, the client still needs to be smart enough to try to reconnect if the socket is closed, etc.

Just make simple protocol like:
File Name\r\n
File Size\r\n
File Data\r\n
File Name\r\n
File Size\r\n
File Data\r\n
....
I hope you will understand this. You can send file information initially then server will parse this file information, and make your server to read number bytes as you specified in file information. These will enable you to see file end marker and when to begin new file. BUT you must know file size before.
This will not work for data streams which have unknown length.
Make your server to read number of bytes you will be specifying, so server can know when to end file writing and begin new file or whether file is fully received before socket closes...

Sending big file using FileInputStream/ObjectOutputStream

I need help on my homework, any help will be much appreciated. I can send small files without a problem. But when i try to send let’s say a 1GB file byte array sends OutOfMemoryError so i need a better solution to send file from server to client. How can i improve this code and send big files, please help me.
Server Code:
FileInputStream fis = new FileInputStream(file);
byte[] fileByte = new byte[fis.available()]; //This causes the problem.
bytesRead = fis.read(fileByte);
oos = new ObjectOutputStream(sock.getOutputStream());
oos.writeObject(fileByte);
Client Code:
ois = new ObjectInputStream(sock.getInputStream());
byte[] file = (byte[]) ois.readObject();
fos = new FileOutputStream(file);
fos.write(file);

Don't read the whole file into memory, use a small buffer and write while you are reading the file:
BufferedOutputStream bos = new BufferedOutputStream(sock.getOutputStream())
File file = new File("asd");
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
byte[] buffer = new byte[1024*1024*10];
int n = -1;
while((n = bis.read(buffer))!=-1) {
bos.write(buffer,0,n):
}
Use Buffered* to optimize the writing and reading from Streams

Just split the array to smaller chunks so that you don't need to allocate any big array.
For example you could split the array into 16Kb chunks, eg new byte[16384] and send them one by one. On the receiving side you would have to wait until a chunk can be fully read and then store them somewhere and start with next chunk.
But if you are not able to allocate a whole array of the size you need on server side you won't be able to store all the data that you are going to receive anyway.
You could also compress the data before sending it to save bandwidth (and time), take a look at ZipOutputStream and ZipInputStream.

Here's how I solved it:
Client Code:
bis=new BufferedInputStream(sock.getInputStream());
fos = new FileOutputStream(file);
int n;
byte[] buffer = new byte[8192];
while ((n = bis.read(buffer)) > 0){
fos.write(buffer, 0, n);}
Server Code:
bos= new BufferedOutputStream(sock.getOutputStream());
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
int n=-1;
byte[] buffer = new byte[8192];
while((n = bis.read(buffer))>-1)
bos.write(buffer,0,n);

Depending on whether or not you have to write the code yourself, there are existing libraries which solve this problem, e.g. rmiio. If you are not using RMI, just plain java serialization, you can use the DirectRemoteInputStream, which is kind of like a Serializable InputStream. (this library also has support for things like auto-magically compressing the data).
Actually, if you are only sending file data, you would be better off ditching the Object streams and use DataInput/DataOutput streams. first write an integer indicating the file length, then copy the bytes directly to the stream. on the receiving side, read the integer file length, then read exactly that many bytes.
when you copy the data between streams, use a small, fixed size byte[] to move chunks of data between the input and output streams in a loop. there are numerous examples of how to do this correctly available online (e.g. #ErikFWinter's answer).

File transfer and object serialization in Java

Im working on a Client/server chat application which allows user to send files (images / videos...) through a socket connection.
In order to manage all kind of communication, I use an Object "Packet" which stores all information that I want to send. (Sender, receivers, file ...).
Here is a code sample where I write in the stream :
private void write(Packet packet) throws IOException {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(bos);
os.writeObject(packet);
this.outStream.write(bos.toByteArray());
}
And outStream is an OutputStream.
Here is my Connection run :
public void run() {
while (isRunning()) {
try {
byte[] buffer = new byte[65536];
// Read from the InputStream
inStream.read(buffer);
ObjectInputStream in = new ObjectInputStream(new ByteArrayInputStream(buffer));
Packet p = (Packet) in.readObject();
} catch (IOException e) {
e.printStackTrace();
this.disconnect();
}
}
}
It works very well for all purpose except files transfer !
I put the file in a byte[] (with filestream) and store the array in my Packet Object.
When the server receive the communication it breaks on the "in.readObject()" and give me a pretty "java io streamcorruptedexception wrong format : 0" exception.
I tried the transfer with a custom byte[] (filled by a string.getBytes()) and it worked very well.
So, what am I doing wrong ?

You're reading from the InputStream to a byte array (with an arbitrary size which could be too small). Then you construct an ObjectInputStream to read from this byte array. Why don't you read your object directly from the InputStream?
ObjectInputStream in = new ObjectInputStream(inStream);
Packet p = (Packet) in.readObject();
No need for a buffer.
Moreover, InputStream.read() doesn't read everything from the InputStream. It reads what is available, and returns the number of bytes read. If you don't loop until it returns -1, you only read a part of what has been sent on the other side.
BTW, you're doing the same mistake on the sending side. Instead of writing your object directly to the output stream, you write it to a byte array, adn then send this byte array. Write your object directly to the stream:
ObjectOutputStream os = new ObjectOutputStream(this.outputStream);
os.writeObject(packet);
No need for a buffer.

sending image from java server to android app

I would like to send image file from java server to android app using this code:
Server(Java):
File file = new File("./clique.jpg");
FileInputStream stream = new FileInputStream(file);
DataOutputStream writer = new DataOutputStream(socket.getOutputStream());
byte[] contextB = new byte[4096];
int n;
int i = 0;
while ( (n=stream.read(contextB))!=-1 ){
writer.write(contextB, 0, n);
writer.flush();
System.out.println(n);
i+=n;
}
writer.flush();
stream.close();
android app:
DataInputStream reader = new DataInputStream(socket.getInputStream());
byte[] buffer = new byte[4096];
ByteArrayOutputStream content = new ByteArrayOutputStream();
int n;
int i = 0;
reader = new DataInputStream(socket.getInputStream());
while ( (n=reader.read(buffer)) != null){
content.write(buffer, 0, n);
content.flush();
}
Utility.CreateImageFile(content.toByteArray());
What I noticed is that in the android app n which is the number of read bytes is not 4096 while I am sending from server byte blocks of 4096 size,also I can not get n=-1 which is the end of stream,it blocks until I close the app then I get n=-1.

Regarding the number of bytes you read at a time has nothing to do with the number of bytes you write -it very much depends on the network conditions and will be variable with every chunk (basically as many bytes managed to be transmitted in short period of time between your reads as many you will get in the read chunk.
Regarding the end of stream - in your server code you have forgotten to close the output stream (you only close the stream which is input stream - you should also close the writer which in turn will close the underlying output stream.
Two comments:
1) I would really recommend to use Buffered Readers/Writers wrapping the writes/readers - the code you will get will be nicer and you will not have to create/manage buffers yourself.
2) Use try {} finally and close your streams in finally clauses - this is the best practice that will make sure that you will close the streams and free resources even in case of problems while reading/writing.

You got a problem in your android code:
while ( (n=reader.read(buffer)) != null) {
n can not be null.
Use writer.close() instead of writer.flush() after your loop on the server.