I've been trying to get this working for a few days now, but I've had no success.
I want to send a file over socket client/server. The only difference is: I want to send an object that contains the file bytes.
So the client loads a file, reads chunks of 1024 bytes, store them in a object, and send the object to the server. Since the file can be larger than 1024 bytes, I want to send the object repeatedly but with different bytes stored in them (as the buffer reads it). On the server, I want to compose the array of bytes and save it as a file.
The reason I'm using 1024 is because I want to avoid any sort of out of memory error, if the file is, let's say, 4 GB in size.
I tried doing the following on the client:
File file = new File("C:\\test\\test.txt");
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
ObjectOutputStream oos = new ObjectOutputStream(socket.getOutputStream());
byte[] bytes = new byte[1024];
FileTest ft = new FileTest();
ft.setName("Testing");
int counttest = 1;
while (bis.read(bytes) > 0) {
ft.setCounttest(counttest);
ft.setBytes(bytes);
oos.writeObject(ft);
counttest += 1;
}
On the server:
int bufferSize = socket.getReceiveBufferSize();
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream("C:\\test\\test2.txt"));
ObjectInputStream ois = new ObjectInputStream(socket.getInputStream());
byte[] bytes = new byte[bufferSize];
while (true) {
FileTest ft = (FileTest) ois.readObject();
if (ft != null) {
System.out.println(ft.getName());
bos.write(ft.getBytes());
}
}
So I tested sending a txt file with a sequence of numbers and the test2.txt file produced by the server came out only with the first 1024 chunk of bytes repeated twice. Also, the counttest integer never increases when received in the server.
Any idea how to accomplish this?
Thanks in advance.
You are running into the effects of the ObjectOutputStream's attempt to preserve object identity. Repeatedly writing the same object instance will result on the same instance on the receiver's end. This is generally a good thing, but confusing if you are modifying the object on the sender's end and expecting those modifications to show up on the receiver's end.
Thus, you have two issues:
In order to send the data each time, you either need to create a new FileTest instance each time, or use the writeUnshared() method.
Due to this identity preserving behavior, you will need to periodically reset() the ObjectOutputStream in order to keep all these instances from being held forever (and potentially leading to an OOME on the client or server).
Related
Ok, So I'm making a Java program that has a server and client and I'm sending a Zip file from server to client. I have sending the file down, almost. But recieving I've found some inconsistency. My code isn't always getting the full archive. I'm guessing it's terminating before the BufferedReader has the full thing. Here's the code for the client:
public void run(String[] args) {
try {
clientSocket = new Socket("jacob-custom-pc", 4444);
out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedInputStream(clientSocket.getInputStream());
BufferedReader inRead = new BufferedReader(new InputStreamReader(in));
int size = 0;
while(true) {
if(in.available() > 0) {
byte[] array = new byte[in.available()];
in.read(array);
System.out.println(array.length);
System.out.println("recieved file!");
FileOutputStream fileOut = new FileOutputStream("out.zip");
fileOut.write(array);
fileOut.close();
break;
}
}
}
} catch(IOException e) {
e.printStackTrace();
System.exit(-1);
}
}
So how can I be sure the full archive is there before it writes the file?
On the sending side write the file size before you start writing the file. On the reading side Read the file size so you know how many bytes to expect. Then call read until you have gotten everything you expect. With network sockets it may take more than one call to read to get everything that was sent. This is especially true as your data gets larger.
HTTP sends a content-length: x+\n in bytes. This is elegant, it might throw a TimeoutException if the conn is broken.
You are using a TCP socket. The ZIP file is probably larger than the network MTU, so it will be split up into multiple packets and reassembled at the other side. Still, something like this might happen:
client connects
server starts sending. The ZIP file is bigger than the MTU and therefore split up into multiple packets.
client busy-waits in the while (true) until it gets the first packets.
client notices that data has arrived (in.available() > 0)
client reads all available data, writes it to the file and exits
the last packets arrive
So as you can see: Unless the client machine is crazily slow and the network is crazily fast and has a huge MTU, your code simply won't receive the entire file by design. That's how you built it.
A different approach: Prefix the data with the length.
Socket clientSocket = new Socket("jacob-custom-pc", 4444);
DataInputStream dataReader = new DataInputStream(clientSocket.getInputStream());
FileOutputStream out = new FileOutputStream("out.zip");
long size = dataReader.readLong();
long chunks = size / 1024;
int lastChunk = (int)(size - (chunks * 1024));
byte[] buf = new byte[1024];
for (long i = 0; i < chunks; i++) {
dataReader.read(buf);
out.write(buf);
}
dataReader.read(buf, 0, lastChunk);
out.write(buf, 0, lastChunk);
And the server uses DataOutputStream to send the size of the file before the actual file. I didn't test this, but it should work.
How can I make sure I received whole file through socket stream?
By fixing your code. You are using InputStream.available() as a test for end of stream. That's not what it's for. Change your copy loop to this, which is also a whole lot simpler:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Use with any buffer size greater than zero, typically 8192.
In.available() just tells you that there is no data to be consumed by in.read() without blocking (waiting) at the moment but it does not mean the end of stream. But, they may arrive into your PC at any time, with TCP/IP packet. Normally, you never use in.available(). In.read() suffices everything for the reading the stream entirely. The pattern for reading the input streams is
byte[] buf;
int size;
while ((size = in.read(buf)) != -1)
process(buf, size);
// end of stream has reached
This way you will read the stream entirely, until its end.
update If you want to read multiple files, then chunk you stream into "packets" and prefix every one with an integer size. You then read until size bytes is received instead of in.read = -1.
update2 Anyway, never use in.available for demarking between the chunks of data. If you do that, you imply that there is a time delay between incoming data pieces. You can do this only in the real-time systems. But Windows, Java and TCP/IP are all these layers incompatible with real-time.
This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 7 years ago.
Presently My Server program can able to receive the file from client socket and able to save that received file in server machine.
But I am need to receive many files from client socket to server socket without closing and opening the socket connection every time.
I have written the code, this is working fine. But in this I am closing and opening the server and client socket connection in every iteration. But I need to do this without connecting and disconnecting both the sockets every time.
Please guide me seniors...
My Server code:
int img_count=1;
int bytesRead;
int current = 0;
byte [] mybytearray = new byte [100000];
InputStream is = null;
FileOutputStream fos = null;
BufferedOutputStream bos = null;
Socket sock=null;
// create socket
ServerSocket servsock = new ServerSocket(6668);
System.out.println("Waiting... for client req");
int i=0;
for ( i=0; i<9; i++)
{
sock = servsock.accept(); // Waiting for Client
String fname = "Image000"+(img_count++)+".JPG";
String fpath = "C:/RX_images/"+fname; // Image saving path
File myFile = new File (fpath);
is = sock.getInputStream();
fos = new FileOutputStream(myFile);
bos = new BufferedOutputStream(fos);
bytesRead = is.read(mybytearray,0,mybytearray.length);
current = bytesRead;
do {
bytesRead =
is.read(mybytearray, current, (mybytearray.length-current));
if(bytesRead >= 0) current += bytesRead;
} while(bytesRead > -1);
bos.write(mybytearray, 0 , current);
bos.flush();
fos.flush();
fos.close();
bos.close();
is.close();
sock.close();
} // End of for loop
servsock.close();
System.out.println("Received : "+ (i++)+ " Images");
My Client Code:
int i=0;
int img_count=1;
FileInputStream fis=null;
BufferedInputStream bis=null;
OutputStream os=null;
Socket client=null;
System.out.println("Sending...");
for ( i=0; i<9; i++)
{
client = new Socket("192.168.1.54",6668);
String fname = "Image000"+(img_count++)+".JPG";
String fpath = "C:/Tx_Images/"+fname; // Image path
File myFile = new File (fpath);
byte [] mybytearray = new byte [(int)myFile.length()];
fis = new FileInputStream(myFile);
bis = new BufferedInputStream(fis);
bis.read(mybytearray,0,mybytearray.length);
os = client.getOutputStream();
os.write(mybytearray,0,mybytearray.length);
bis.close();
fis.close();
os.flush();
os.close();
client.close();
Thread.sleep(2000);
} // End of for loop
System.out.println("\n Sent : "+(i++)+" Images");
I am very new to java,
Help me please....
Since the socket is just a stream of bytes, in order to handle more than one file you are going to have to construct a simple protocol of some sort. In other words, the sender will have to send bytes that differentiate between the bytes in one file and the bytes in another. Since you are sending binary data, there is no series of bytes you can send to "mark" the beginning and/or/ending -- for example if you send 4 zero bytes at the end, that might be data and so the receiver cannot be sure if it's a marker or data. Two ways to handle it come to mind offhand -- break your file up into sections that are a maximum of N bytes, and send the sections one at a time. You will have to have a count of the bytes in each section, since at least one section will not have the same number of bytes as all other sections. Alternately,y you could count the bytes in the file and start with bytes that give that count, so the receiver knows how many bytes to expect. While you are giving the count, you could also give information such as the name and the type of file, if you wanted. Good luck.
This question really depends on whether you need the client to keep the connection open, or not. Typically you just need to keep the server side listening, and it's ok for the client to reconnect each time it needs to send a file.
Use an ExecutorService to keep the server side going and handle multiple connections with separate threads. Then just have the client connect and send what it needs to send and disconnect. See this question for a quick example: Multithreading Socket communication Client/Server
Also, look at how they close resources (finally) and stop the server in that example too. That is not related to your question, but you'll want to make your I/O and error handling more robust as well.
If you really do require that the server and client stay connected and send multiple files (or whatever data) then you'll need to implement some sort of a protocol as rcook notes, and you'll need to go deeper into networking and have a heartbeat and such. And, even if you do that, the client still needs to be smart enough to try to reconnect if the socket is closed, etc.
Just make simple protocol like:
File Name\r\n
File Size\r\n
File Data\r\n
File Name\r\n
File Size\r\n
File Data\r\n
....
I hope you will understand this. You can send file information initially then server will parse this file information, and make your server to read number bytes as you specified in file information. These will enable you to see file end marker and when to begin new file. BUT you must know file size before.
This will not work for data streams which have unknown length.
Make your server to read number of bytes you will be specifying, so server can know when to end file writing and begin new file or whether file is fully received before socket closes...
I need help on my homework, any help will be much appreciated. I can send small files without a problem. But when i try to send let’s say a 1GB file byte array sends OutOfMemoryError so i need a better solution to send file from server to client. How can i improve this code and send big files, please help me.
Server Code:
FileInputStream fis = new FileInputStream(file);
byte[] fileByte = new byte[fis.available()]; //This causes the problem.
bytesRead = fis.read(fileByte);
oos = new ObjectOutputStream(sock.getOutputStream());
oos.writeObject(fileByte);
Client Code:
ois = new ObjectInputStream(sock.getInputStream());
byte[] file = (byte[]) ois.readObject();
fos = new FileOutputStream(file);
fos.write(file);
Don't read the whole file into memory, use a small buffer and write while you are reading the file:
BufferedOutputStream bos = new BufferedOutputStream(sock.getOutputStream())
File file = new File("asd");
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
byte[] buffer = new byte[1024*1024*10];
int n = -1;
while((n = bis.read(buffer))!=-1) {
bos.write(buffer,0,n):
}
Use Buffered* to optimize the writing and reading from Streams
Just split the array to smaller chunks so that you don't need to allocate any big array.
For example you could split the array into 16Kb chunks, eg new byte[16384] and send them one by one. On the receiving side you would have to wait until a chunk can be fully read and then store them somewhere and start with next chunk.
But if you are not able to allocate a whole array of the size you need on server side you won't be able to store all the data that you are going to receive anyway.
You could also compress the data before sending it to save bandwidth (and time), take a look at ZipOutputStream and ZipInputStream.
Here's how I solved it:
Client Code:
bis=new BufferedInputStream(sock.getInputStream());
fos = new FileOutputStream(file);
int n;
byte[] buffer = new byte[8192];
while ((n = bis.read(buffer)) > 0){
fos.write(buffer, 0, n);}
Server Code:
bos= new BufferedOutputStream(sock.getOutputStream());
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
int n=-1;
byte[] buffer = new byte[8192];
while((n = bis.read(buffer))>-1)
bos.write(buffer,0,n);
Depending on whether or not you have to write the code yourself, there are existing libraries which solve this problem, e.g. rmiio. If you are not using RMI, just plain java serialization, you can use the DirectRemoteInputStream, which is kind of like a Serializable InputStream. (this library also has support for things like auto-magically compressing the data).
Actually, if you are only sending file data, you would be better off ditching the Object streams and use DataInput/DataOutput streams. first write an integer indicating the file length, then copy the bytes directly to the stream. on the receiving side, read the integer file length, then read exactly that many bytes.
when you copy the data between streams, use a small, fixed size byte[] to move chunks of data between the input and output streams in a loop. there are numerous examples of how to do this correctly available online (e.g. #ErikFWinter's answer).
Im working on a Client/server chat application which allows user to send files (images / videos...) through a socket connection.
In order to manage all kind of communication, I use an Object "Packet" which stores all information that I want to send. (Sender, receivers, file ...).
Here is a code sample where I write in the stream :
private void write(Packet packet) throws IOException {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(bos);
os.writeObject(packet);
this.outStream.write(bos.toByteArray());
}
And outStream is an OutputStream.
Here is my Connection run :
public void run() {
while (isRunning()) {
try {
byte[] buffer = new byte[65536];
// Read from the InputStream
inStream.read(buffer);
ObjectInputStream in = new ObjectInputStream(new ByteArrayInputStream(buffer));
Packet p = (Packet) in.readObject();
} catch (IOException e) {
e.printStackTrace();
this.disconnect();
}
}
}
It works very well for all purpose except files transfer !
I put the file in a byte[] (with filestream) and store the array in my Packet Object.
When the server receive the communication it breaks on the "in.readObject()" and give me a pretty "java io streamcorruptedexception wrong format : 0" exception.
I tried the transfer with a custom byte[] (filled by a string.getBytes()) and it worked very well.
So, what am I doing wrong ?
You're reading from the InputStream to a byte array (with an arbitrary size which could be too small). Then you construct an ObjectInputStream to read from this byte array. Why don't you read your object directly from the InputStream?
ObjectInputStream in = new ObjectInputStream(inStream);
Packet p = (Packet) in.readObject();
No need for a buffer.
Moreover, InputStream.read() doesn't read everything from the InputStream. It reads what is available, and returns the number of bytes read. If you don't loop until it returns -1, you only read a part of what has been sent on the other side.
BTW, you're doing the same mistake on the sending side. Instead of writing your object directly to the output stream, you write it to a byte array, adn then send this byte array. Write your object directly to the stream:
ObjectOutputStream os = new ObjectOutputStream(this.outputStream);
os.writeObject(packet);
No need for a buffer.
I'm creating a TFTP server. I've got it tranfering files fine but most of the files wont open when they arrive at the other end. This is because the output of the ArrayList im using to store file bytes from every packet received adds a load of bytes to the start of the file. eg. "¬í sr java.util.ArrayListxÒ™Ça I sizexp w ur [B¬óøTà xp ü!". The reason for using the List in the first place is that the server im creating has no way to tell the file size of the file which is being received. Therefore as far as I can tell I cant use a byte[] as this needs to be initialised with a set length. Is there any way round this?
WRQ WRQ = new WRQ();
ACK ACK = new ACK();
DatagramPacket outPacket;
byte[] bytes;
byte[] fileOut;
List fileBytes = new ArrayList();
outPacket = WRQ.firstPacket(packet);
socket.send(outPacket);
socket.receive(packet);
while (packet.getLength() == 516){
bytes = WRQ.doWRQ(packet);
fileBytes.add(bytes);
outPacket = ACK.doACK(packet);
socket.send(outPacket);
socket.receive(packet);
}
bytes = WRQ.doWRQ(packet);
fileBytes.add(bytes);
outPacket = ACK.doACK(packet);
socket.send(outPacket);
ObjectOutputStream os;
ByteArrayOutputStream byteStream = new ByteArrayOutputStream(5000);
os = new ObjectOutputStream(new BufferedOutputStream(byteStream));
os.flush();
os.writeObject(fileBytes);
os.flush();
byte[] outFile = byteStream.toByteArray();
os.close();
FileOutputStream foStream = new FileOutputStream(filename);
foStream.write(outFile);
foStream.close();
You store byte arrays in an ArrayList, and then you write the whole ArrayList to a ByteArrayOutputStream wrapped in an ObjectOutputStream, using the writeObject() method.
This uses the native Object serialization mechanism to save the ArrayList object. It doesn't write every byte array in the list one after the other. To make it clear: it writes the class name, and the internal structure of the ArrayList, using the object serialization protocol.
You don't need an ArrayList. Write directly to a ByteArrayOutputStream, or even directly to a FileOutputStream. As is, you're trying to
write bytes to a list
write the bytes in the list to a byte array
write the byte array to a file.
It would be much more straightforward (and efficient) to write directly to the output file (wrapped in a BufferedOutputStream for buffering)