Develop Reference

Java reading a file and sending it via DatagramSocket

Using Java, I am trying to send some file data over a DatagramSocket. I need to read a file in 1000-byte chunks and send them over as packets. My code:
reads a file into a byte array wrapped in a byte buffer
places the data in a packet and sends it
has the receiver open the packet and re-write the contents to a new file.
I am having a problem with writing the byte array back to a file. Please see my code below.
Client/Sender:
byte[] data = new byte[1000];
ByteBuffer b = ByteBuffer.wrap(data);
DatagramPacket pkt;
File file = new File(sourceFile);
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
CRC32 crc = new CRC32();
while(true){
b.clear();
b.putLong(0); // I need to put the checksum at the beginning for easy retrieval
bytesRead = bis.read(data);
if(bytesRead==-1) { break; }
crc.reset();
crc.update(data, 8, data.length-8);
long chksum = crc.getValue();
b.rewind();
b.putLong(chksum);
pkt = new DatagramPacket(data, 1000, addr); // addr is valid, works fine
sk.send(pkt);
}
bis.close();
fis.close();
Server/Receiver:
DatagramSocket sk = new DatagramSocket(port);
File destfile = new File("hello.txt");
FileOutputStream fos = new FileOutputStream(destfile);
BufferedOutputStream bos = new BufferedOutputStream(fos);
PrintStream ps = new PrintStream(fos);
byte[] data = new byte[1000];
DatagramPacket pkt = new DatagramPacket(data, data.length);
ByteBuffer b = ByteBuffer.wrap(data);
CRC32 crc = new CRC32();
while(true) {
pkt.setLength(data.length);
sk.receive(pkt);
b.rewind();
// compare checksum, print error if checksum is different
// if checksum is the same:
bos.write(data); // Where the problem seems to be occurring.
// send acknowledgement packet.
}
bos.close();
fos.close();
Here, I am mainly having issues with writing the file back. With a small text file that says Hello World!, I get a strange output that says vˇ]rld!. Also, the input file is only 12 bytes but the file that the receiver creates is 1KB.
I think my issue is dealing with a byte buffer - I've written a program that copies files using file streams and buffered streams, which worked well. But I'm confused with how streams work in this sort of situation, and I would really appreciate any help. Thanks!

In the sender's data[] you overwrite the text, which was read from the file by the crc! You have to read the text in a position after the long. When correcting this in the Sender, it works:
//int bytesRead = bis.read(data); --old code
int bytesRead=bis.read(data,8,data.length-8);
Furthermore you send 1000 bytes, so will receive 1000 bytes, which will go into the destfile.
BTW: you do not check the crc in the server.... so why send it ?

Java TCP File Transfer Only Complete On First Attempt

Despite hours of researching this problem, I have made very little progress. According to my professor, the code should be working as written...
I have a server that stays open, and a client that requests a file. Once the client receives the file, the client closes.
When I open the server, I am able to transfer a complete .jpg image file. The client then closes while the server remains open. I start up another client and try to transfer the same image, and only a portion of the bytes are transferred/written to the disk. The file transfer is only completely successful for the first file transferred by the server!
Additionally strange, a simple .txt text file never successfully transfers. I believe the cause is on the server side because it remains open as opposed to the client, which starts over each time.
Server Code:
import java.io.*;
import java.net.*;
import java.util.Arrays;
class ft_server {
public static void main(String args[]) throws Exception {
/*
* Asks user for port number and listens on that port
*/
BufferedReader portFromUser = new BufferedReader(new InputStreamReader(System.in));
System.out.println("Enter the port you'd like to use: ");
int portNumber = Integer.valueOf(portFromUser.readLine());
if (portNumber < 1 || portNumber > 65535) {
System.out.println("Please choose a port number between 1 and 65535.");
return;
}
portFromUser.close();
ServerSocket listenSocket = new ServerSocket(portNumber);
/*
* Finished with user input
*/
/*
* Continuously listens for clients:
*/
while (true) {
Socket clientSocket = listenSocket.accept();
BufferedReader inFromClient = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
DataOutputStream outToClient = new DataOutputStream(clientSocket.getOutputStream());
String clientIP = clientSocket.getRemoteSocketAddress().toString();
System.out.println("The client " + clientIP + " connected!");
String clientMessage = inFromClient.readLine();
System.out.println("The client requested file: " + clientMessage);
// Get file. If doesn't exist, let's client know.
// Otherwise informs client of file size.
File myFile = new File(clientMessage);
if (!myFile.exists()) {
outToClient.writeBytes("File does not exist!\n");
return;
} else {
outToClient.writeBytes(String.valueOf((int)myFile.length()) + "\n");
}
// Create array for storage of file bytes:
byte[] byteArray = new byte[(int)myFile.length()];
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(myFile));
// Read file into array:
bis.read(byteArray, 0, byteArray.length);
// Send the file:
outToClient.write(byteArray, 0, byteArray.length);
outToClient.close();
clientSocket.close();
}
}
}
Client Code:
import java.io.*;
import java.net.*;
class ft_client {
public static void main(String args[]) throws Exception {
int byteSize = 2022386;
int bytesRead;
/*
* Asks user for IP and port:
*/
BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in));
System.out.println("Enter an IP address: ");
String ipAddress = inFromUser.readLine();
System.out.println("Enter a port: ");
String port = inFromUser.readLine();
Socket clientSocket;
try {
// Makes socket, port, and calls connect. Assumes it's TCP:
clientSocket = new Socket(ipAddress, Integer.valueOf(port));
} catch (Exception e) {
System.out.println(e.getMessage());
return;
}
// Creates InputStream from server to get file size and other messages:
BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
// Anything written to this will be sent to the server:
DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());
// Asks for a file name to download from the server:
System.out.println("What file do you want?: ");
String message = inFromUser.readLine();
outToServer.writeBytes(message + "\n");
inFromUser.close();
// Listens for confirmation from server.
// If the file exists, the file size is delivered here:
String response = inFromServer.readLine();
System.out.println("File size: " + response);
if (response.equals("File does not exist!")) {
return;
}
// Receives file from server:
byteSize = (int) Integer.valueOf(response);
byte[] byteArray = new byte[byteSize];
InputStream is = clientSocket.getInputStream(); // calling clientSocket.getInputStream() twice???
FileOutputStream fos = new FileOutputStream(message);
BufferedOutputStream bos = new BufferedOutputStream(fos);
// Continuously writes the file to the disk until complete:
int total = 0;
while ((bytesRead = is.read(byteArray)) != -1) {
bos.write(byteArray, 0, bytesRead);
total += bytesRead;
}
bos.close();
System.out.println("File downloaded (" + total + " bytes read)");
clientSocket.close();
}
}
Are buffered readers interfering with output streams? Is there a better way to transfer files?

It's worth checking, in your server code, what value comes back from the file read() call, so:
int bytesRead = bis.read(byteArray, 0, byteArray.length);
System.out.println("File bytes read: " + bytesRead + " from file size: " + myFile.length());
The read() method is under no obligation to fill the byteArray - only to return something and to tell you how many bytes it read. From the docs, it:
Reads up to len bytes of data from this input stream into an array of
bytes. If len is not zero, the method blocks until some input is
available; otherwise, no bytes are read and 0 is returned.
You need to keep reading in a loop. I'd do this (actually, same as your client!):
int n;
while ((n = bis.read(byteArray, 0, byteArray.length)) != -1) {
// Send the chunk of n bytes
outToClient.write(byteArray, 0, n);
}
bis.close();
outToClient.close();
or something similar. I've closed the file too: it'd close on GC/finalize, but that could be a while, and meanwhile you're holding the file open.
EDIT
The specific problem with your image-read in this case is in your client code. You read the file size near the top of the code:
// Creates InputStream from server to get file size and other messages:
BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
and then you access the client again:
InputStream is = clientSocket.getInputStream(); // calling clientSocket.getInputStream() twice???
and as your comment suggests, this is bad! Thank you to #EJP for highlighting this!
This causes a problem of buffer over-ingestion: the BufferedReader consumes more bytes into its belly than you extract from it, so when you visit the clientSocket inputstream the second time, the read-pointer has moved on. You never look again at what the BufferedReader consumed.
As a general rule, once you plug buffering code onto something, you must be careful to read only from that buffer. In this case, it's difficult, because you can't read image (raw binary) data from a Reader, because it will busily interpret the binary values as characters and read them as UTF-8 or something.
Even without buffers, it's a minor sin to mix Readers (text oriented) and binary data (DataStreams) on the same stream. HTTP and email does this, so you are in good company, but they get away with it by being very tightly specified. Problem is, you can easily get snarled with questions of local/default character encoding at each end, whether you're reading Unix "LF" vs Windows "CR/LF" line endings etc.
In this case, try not using BufferedReaders at all, and try using DataInput/Output streams all the way. Try writeUTF(s) and readUTF() for transferring the String data. Ideally, create them like this:
DataInputStream inFromServer = new DataInputStream (new BufferedInputStream(clientSocket.getInputStream()));
so you still get the benefits of buffering.
EDIT 2
So seeing the new client code:
byteSize = (int) Integer.valueOf(response);
byte[] byteArray = new byte[byteSize];
FileOutputStream fos = new FileOutputStream(message);
int readBytes = inFromServer.read(byteArray);
// Continuously writes the file to the disk until complete:
int total = 0;
for (int i=0; i<byteArray.length; i++) {
fos.write(byteArray[i]);
total++;
}
fos.close();
Here, we're assuming that because the byteArray array is set to the right size, that the inFromServer.read(byteArray) will populate it - it won't. It's good to assume that any and all read operations will return you just as much data as the system has to hand: in this case, it's probably going to return as soon as it gets the first packet or two, with an underfilled array. This is same as C and Unix read behaviour too.
Try this - I'm repeatedly reading and writing a 4K buffer, until the byte count is reached (as determined by summing the return values of the reads):
byteSize = (int) Integer.valueOf(response);
byte[] byteArray = new byte[4096];
FileOutputStream fos = new FileOutputStream(message);
int total = 0;
// Continuously writes the file to the disk until complete:
while (total < byteSize && (readBytes = inFromServer.read(byteArray)) != -1) {
fos.write(byteArray, 0, readBytes);
total += readBytes;
}
fos.close();
A variant is this - same thing, but byte at a time. Might be a bit clearer. It's going to be slow - all those reads and writes are hitting the OS, but if you put a BufferedInputStream/BufferedOutputStream around the socket/file streams, it'll iron that out. I've added them:
DataInputStream inFromServer =
new DataInputStream(new BufferedInputStream(clientSocket.getInputStream()));
...
byteSize = (int) Integer.valueOf(response);
OutputStream fos = new BufferedOutputStream(FileOutputStream(message));
int total = 0;
int ch;
// Continuously writes the file to the disk until complete:
while (total < byteSize && (ch = inFromServer.read()) != -1) {
fos.write(ch);
total ++;
}
fos.close();
And finally! the simplest answer is this. Your code, but changed to:
int readBytes = inFromServer.readFully(byteArray);
Yes! Those nice people in 1990's Javasoft added a DataInput.readFully method, which does what you want! - basically wraps the code above. It's the simplest solution, and arguably most correct approach: "use existing libraries where possible". OTOH, it's the least educational, and the time you spend getting used to read/writes like this is not deducted from your life-expectancy!
And in fact, the readFully approach has severe limitations. Try pointing it at a 1GB file and see what happens (after you've fixed up the array size at the top): you'll a) run out memory, and b) wish that while you were ingesting a huge blob, you could at least be spooling it out to disk. If you try a 2.5G file, you'll notice that some of those ints should become longs to cope with numbers >= 2^31.
If it was me, I'd do the 4K buffer one. (BTW I'm writing this on a laptop with no Java compiler installed, so I haven't actually run the above! DO respond if there are any difficulties.)

ZipOutputStream creating corrupt (unzippable) zip file

I am trying to send zipped bytes to another server and then have that server receive them and write out the zipped filed. When I do the zipping and writing on the same server it works great. The local version looks something like this:
ZipOutputStream zout = new ZipOutputStream(FileOutputStream);
zout.write(byteBuffer, 0, len);
zout.flush()
FileOutputStream.flush();
zout.close();
The cross server implementation creates a bad output though. The sending code is: (magic string tells the server it has received all of the data.
ZipOutputStream zout = new ZipOutputStream(out);
ZipEntry entry = new ZipEntry(fileName);
zout.putNextEntry(entry);
System.out.println("sending zipped bytes...");
zout.write(inputBuffer, contentBegin, len);
zout.flush();
zout.closeEntry();
out.flush();
byte[] magicStringData = "--------MagicStringCSE283Miami".getBytes("US-ASCII");
out.write(magicStringData, 0, magicStringData.length);
out.flush();
System.out.println("Done writing file and sending zipped bytes.");
Thread.sleep(10000);
zout.close();
clntSock.close(); // Close the socket. We are done with this client!
The receiving code looks like this:
System.out.println("receiving zipped bytes...");
byte[] inputBuffer = new byte[BUF_SIZE];
int total2 = 0, count = 0;
while(count != -1) { // read from origin's buffer into byteBuffer until origin is out of data
count = inFromCompression.read(inputBuffer, total2, BUF_SIZE - total - 1);
String msg = new String(inputBuffer, total2, count, "US-ASCII");
total2 += count;
if(msg.contains("-------MagicString")){
System.out.println("full message received...");
break;
}
}
String inputString = new String(inputBuffer, 0, total2, "US-ASCII");
int contentEnd = inputString.indexOf("--------MagicString");
FileOutputStream fout2 = new FileOutputStream(outputFileName + ".zip");
fout2.write(inputBuffer, 0, contentEnd);
fout2.flush();
fout2.close();
System.out.println("Done writing zipped bytes.");
//Thread.sleep(10000);
//socketToCompression.close();
Any ideas? I am thinking it might be something like I am not closing the ZipOutputStream before sending the magic string that signals the end of the data, but every time I call zout.close() immediately after flushing zout it closes the entire socket.

Get rid of the magic string and just send and receive the actual data. You're presently throwing away any buffer that contains the magic string, including whatever ZIP data may have been before it.
You don't need a ByteArrayOutputStream.

How to send a list of files over a socket in Java [duplicate]

This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 7 years ago.
I have used the code here to send an individual file over a socket. However, I need to be able to send multiple files (basically all files in a directory) over the socket and have the client recognize how the separation between files. Frankly, I am at a complete loss for what to do. Any tips would be helpful.
NOTE 1: I need a way to send the files in one continuous stream that the client can segregate into individual files. It cannot rely on individual requests from the client.
NOTE 2: To answer a question I am pretty sure I will get in the comments, no, this is NOT homework.
EDIT it has been suggested that I could send the size of the file before the file itself. How can I do this, as sending a file over the socket is always done in either a predetermined array of bytes, or a single byte individually, rather than the long returned by File.length()

Here is a full implementation:
Sender Side:
String directory = ...;
String hostDomain = ...;
int port = ...;
File[] files = new File(directory).listFiles();
Socket socket = new Socket(InetAddress.getByName(hostDomain), port);
BufferedOutputStream bos = new BufferedOutputStream(socket.getOutputStream());
DataOutputStream dos = new DataOutputStream(bos);
dos.writeInt(files.length);
for(File file : files)
{
long length = file.length();
dos.writeLong(length);
String name = file.getName();
dos.writeUTF(name);
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
int theByte = 0;
while((theByte = bis.read()) != -1) bos.write(theByte);
bis.close();
}
dos.close();
Receiver Side:
String dirPath = ...;
ServerSocket serverSocket = ...;
Socket socket = serverSocket.accept();
BufferedInputStream bis = new BufferedInputStream(socket.getInputStream());
DataInputStream dis = new DataInputStream(bis);
int filesCount = dis.readInt();
File[] files = new File[filesCount];
for(int i = 0; i < filesCount; i++)
{
long fileLength = dis.readLong();
String fileName = dis.readUTF();
files[i] = new File(dirPath + "/" + fileName);
FileOutputStream fos = new FileOutputStream(files[i]);
BufferedOutputStream bos = new BufferedOutputStream(fos);
for(int j = 0; j < fileLength; j++) bos.write(bis.read());
bos.close();
}
dis.close();
I did not test it, but I hope it will work!

You could send the size of the file first before each file, that way the client will know when the current file is over and expect the next (size). This will allow you to use one contiguous stream for all files.

A very simple way to do it would be to send the file length before sending each file so that you can determine the separation between files.
Of course, if the receiving process is Java, you can just send Objects.

You could zip the files on the client side and send this zipped stream to the server.
e.g: http://www.exampledepot.com/egs/java.util.zip/CreateZip.html
with ...
OutputStream output = connection.getOutputStream();
ZipOutputStream out = new ZipOutputStream(output);

Maybe the fastest way is to automatically zip and unzip the files in your directory into one file, see java.util.zip package

How to write file data correctly?

My application is unable to transfer data properly over a socket connection and write it to a file properly. Files over about 65,535 bytes get corrupted and are no longer recognized by the programs designed to run them.
I have been able to send small .doc and .txt files successfully, but .mp3 .wmv .m4a .avi and just about anything else does not work. Neither do larger docs.
I have looked all over the internet for a solution to this problem. I have repeatedly tweaked the I/O code to fix the problem but it still doesn't work! Here is the I/O code in the super class that handles sending and receiving files. If you need anymore information/other parts of code, let me know.
protected void sendFile() throws IOException {
byte[] bytes = new byte[(int) file.length()];
buffin = new BufferedInputStream(new FileInputStream(file));
int bytesRead = buffin.read(bytes,0,bytes.length);
System.out.println(bytesRead);
out = sock.getOutputStream();
out.write(bytes,0,fileBytes);
out.flush();
out.close();
}
protected void receiveFile() throws IOException {
byte[] bytes = new byte[fileBytes];
in = sock.getInputStream();
for(int i=0;i<fileBytes;i++) {
in.read(bytes);
}
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
buffout.write(bytes,0,fileBytes);
buffout.flush();
buffout.close();
}
UPDATED CODE (that works):
protected void sendFile() throws IOException {
if((file.length())<63000) {
byte[] bytes = new byte[(int)file.length()];
buffin = new BufferedInputStream(new FileInputStream(file));
buffin.read(bytes,0,bytes.length);
out = sock.getOutputStream();
out.write(bytes,0,bytes.length);
out.close();
} else {
byte[] bytes = new byte[32000];
buffin = new BufferedInputStream(new FileInputStream(file));
out = sock.getOutputStream();
int bytesRead;
while((bytesRead = buffin.read(bytes))>0) {
out.write(bytes,0,bytesRead);
}
out.close();
}
}
protected void receiveFile() throws IOException {
if(fileBytes<63000) {
byte[] bytes = new byte[32000];
in = sock.getInputStream();
System.out.println(in.available());
in.read(bytes,0,fileBytes);
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
buffout.write(bytes,0,bytes.length);
buffout.close();
} else {
byte[] bytes = new byte[16000];
in = sock.getInputStream();
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
int bytesRead;
while((bytesRead = in.read(bytes))>0) {
buffout.write(bytes,0,bytesRead);
}
buffout.close();
}
}

The issue is that you are sending only chunks of it. That is, you are only sending 64k of the file ever. If the file is ever larger then 64k the other end will never see it.
You want to continously read from the BufferedInputStream until the read() returns either less then the length or -1.

Your code is completely wrong. This is how to copy a stream in Java:
int count;
byte[] buffer = new byte[8192]; // more if you like but no need for it to be the entire file size
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
You should use this both when sending the file and when receiving the file. At present your sending method hopes that the entire file fits into memory; fits into INTEGER_MAX bytes; and is read in one chunk by the read method, without even checking the result. You can't assume any of those things. Your receive method is complete rubbish: it just keeps overwriting the same array, again without checking any read() results.
EDIT: Your revised code is just as bad, or worse. You are calling read() to check for EOS and then throwing that byte away, and then calling read() again and throwing away the read count it returns. You pointlessly have a different path for files < 64000, or 63000, or whatever it is, that has zero benefit except to give you two code paths to test, or possibly four, instead of one. The network only gives you 1460 bytes at a time at best anyway so what is the point? You already have (a) a BufferedInputStream with a default buffersize of 8192, and (b) my code that uses a byte[] buffer of any size you like. My code above works for any amount of data in two lines of executable code. Yours is 20. QED.

I suggest that you use some good library to read and write file contents as well as socket read/write. For example Apache Commons IO. If you insist on writig code yourself, do it smaller chunks rather than the whole file at once.

You have to consider that InputStream.read returns the number of bytes read which may be less than the total number of bytes in the file.
You would probably be better off just letting something like CopyUtils.copy take care of this for you.

You need to loop until bytesRead < 0. You need to make sure that fileBytes is => than the transferred file.
protected void receiveFile() throws IOException {
byte [] bytes = new byte [fileBytes];
InputStream is = sock.getInputStream();
FileOutputStream fos = new FileOutputStream("/Datawire/"+fileName);
BufferedOutputStream bos = new BufferedOutputStream(fos);
int bytesRead = is.read(bytes,0,bytes.length);
int current = bytesRead;
do {
bytesRead =
is.read(bytes, current, (bytes.length-current));
if(bytesRead >= 0) current += bytesRead;
} while(bytesRead > -1);
bos.write(bytes, 0 , current);
bos.flush();
bos.close();
}