Develop Reference

First two letters missing in file transfer in java

I am working on a client/server transfer protocol in java. The client is sending a simple text file, everything is going across the wire fine in wireshark, but once it gets to the server side, the first two letters are missing from the text file. I believe that it is overwriting the first buffer for some reason.
My goal is to make a while loop that reads the bytes in the buffer and then increments a count that'll place the next set of bytes....in the place if the ones already written
Here is the server's code that I currently have:
int bytesRead;
int current = 0;
InputStream in = s.getInputStream();
// Instantiating a new output stream object
OutputStream output = new FileOutputStream(myFile);
PrintStream stream = new PrintStream(output);
// Receive file 1024 bytes at a time
byte[] buffer = new byte[1024];
while ((bytesRead = in.read(buffer)) != -1) {
output.write(buffer, 0, bytesRead);
System.out.println(output.toString());
}

Java TCP File Transfer Only Complete On First Attempt

Despite hours of researching this problem, I have made very little progress. According to my professor, the code should be working as written...
I have a server that stays open, and a client that requests a file. Once the client receives the file, the client closes.
When I open the server, I am able to transfer a complete .jpg image file. The client then closes while the server remains open. I start up another client and try to transfer the same image, and only a portion of the bytes are transferred/written to the disk. The file transfer is only completely successful for the first file transferred by the server!
Additionally strange, a simple .txt text file never successfully transfers. I believe the cause is on the server side because it remains open as opposed to the client, which starts over each time.
Server Code:
import java.io.*;
import java.net.*;
import java.util.Arrays;
class ft_server {
public static void main(String args[]) throws Exception {
/*
* Asks user for port number and listens on that port
*/
BufferedReader portFromUser = new BufferedReader(new InputStreamReader(System.in));
System.out.println("Enter the port you'd like to use: ");
int portNumber = Integer.valueOf(portFromUser.readLine());
if (portNumber < 1 || portNumber > 65535) {
System.out.println("Please choose a port number between 1 and 65535.");
return;
}
portFromUser.close();
ServerSocket listenSocket = new ServerSocket(portNumber);
/*
* Finished with user input
*/
/*
* Continuously listens for clients:
*/
while (true) {
Socket clientSocket = listenSocket.accept();
BufferedReader inFromClient = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
DataOutputStream outToClient = new DataOutputStream(clientSocket.getOutputStream());
String clientIP = clientSocket.getRemoteSocketAddress().toString();
System.out.println("The client " + clientIP + " connected!");
String clientMessage = inFromClient.readLine();
System.out.println("The client requested file: " + clientMessage);
// Get file. If doesn't exist, let's client know.
// Otherwise informs client of file size.
File myFile = new File(clientMessage);
if (!myFile.exists()) {
outToClient.writeBytes("File does not exist!\n");
return;
} else {
outToClient.writeBytes(String.valueOf((int)myFile.length()) + "\n");
}
// Create array for storage of file bytes:
byte[] byteArray = new byte[(int)myFile.length()];
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(myFile));
// Read file into array:
bis.read(byteArray, 0, byteArray.length);
// Send the file:
outToClient.write(byteArray, 0, byteArray.length);
outToClient.close();
clientSocket.close();
}
}
}
Client Code:
import java.io.*;
import java.net.*;
class ft_client {
public static void main(String args[]) throws Exception {
int byteSize = 2022386;
int bytesRead;
/*
* Asks user for IP and port:
*/
BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in));
System.out.println("Enter an IP address: ");
String ipAddress = inFromUser.readLine();
System.out.println("Enter a port: ");
String port = inFromUser.readLine();
Socket clientSocket;
try {
// Makes socket, port, and calls connect. Assumes it's TCP:
clientSocket = new Socket(ipAddress, Integer.valueOf(port));
} catch (Exception e) {
System.out.println(e.getMessage());
return;
}
// Creates InputStream from server to get file size and other messages:
BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
// Anything written to this will be sent to the server:
DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());
// Asks for a file name to download from the server:
System.out.println("What file do you want?: ");
String message = inFromUser.readLine();
outToServer.writeBytes(message + "\n");
inFromUser.close();
// Listens for confirmation from server.
// If the file exists, the file size is delivered here:
String response = inFromServer.readLine();
System.out.println("File size: " + response);
if (response.equals("File does not exist!")) {
return;
}
// Receives file from server:
byteSize = (int) Integer.valueOf(response);
byte[] byteArray = new byte[byteSize];
InputStream is = clientSocket.getInputStream(); // calling clientSocket.getInputStream() twice???
FileOutputStream fos = new FileOutputStream(message);
BufferedOutputStream bos = new BufferedOutputStream(fos);
// Continuously writes the file to the disk until complete:
int total = 0;
while ((bytesRead = is.read(byteArray)) != -1) {
bos.write(byteArray, 0, bytesRead);
total += bytesRead;
}
bos.close();
System.out.println("File downloaded (" + total + " bytes read)");
clientSocket.close();
}
}
Are buffered readers interfering with output streams? Is there a better way to transfer files?

It's worth checking, in your server code, what value comes back from the file read() call, so:
int bytesRead = bis.read(byteArray, 0, byteArray.length);
System.out.println("File bytes read: " + bytesRead + " from file size: " + myFile.length());
The read() method is under no obligation to fill the byteArray - only to return something and to tell you how many bytes it read. From the docs, it:
Reads up to len bytes of data from this input stream into an array of
bytes. If len is not zero, the method blocks until some input is
available; otherwise, no bytes are read and 0 is returned.
You need to keep reading in a loop. I'd do this (actually, same as your client!):
int n;
while ((n = bis.read(byteArray, 0, byteArray.length)) != -1) {
// Send the chunk of n bytes
outToClient.write(byteArray, 0, n);
}
bis.close();
outToClient.close();
or something similar. I've closed the file too: it'd close on GC/finalize, but that could be a while, and meanwhile you're holding the file open.
EDIT
The specific problem with your image-read in this case is in your client code. You read the file size near the top of the code:
// Creates InputStream from server to get file size and other messages:
BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
and then you access the client again:
InputStream is = clientSocket.getInputStream(); // calling clientSocket.getInputStream() twice???
and as your comment suggests, this is bad! Thank you to #EJP for highlighting this!
This causes a problem of buffer over-ingestion: the BufferedReader consumes more bytes into its belly than you extract from it, so when you visit the clientSocket inputstream the second time, the read-pointer has moved on. You never look again at what the BufferedReader consumed.
As a general rule, once you plug buffering code onto something, you must be careful to read only from that buffer. In this case, it's difficult, because you can't read image (raw binary) data from a Reader, because it will busily interpret the binary values as characters and read them as UTF-8 or something.
Even without buffers, it's a minor sin to mix Readers (text oriented) and binary data (DataStreams) on the same stream. HTTP and email does this, so you are in good company, but they get away with it by being very tightly specified. Problem is, you can easily get snarled with questions of local/default character encoding at each end, whether you're reading Unix "LF" vs Windows "CR/LF" line endings etc.
In this case, try not using BufferedReaders at all, and try using DataInput/Output streams all the way. Try writeUTF(s) and readUTF() for transferring the String data. Ideally, create them like this:
DataInputStream inFromServer = new DataInputStream (new BufferedInputStream(clientSocket.getInputStream()));
so you still get the benefits of buffering.
EDIT 2
So seeing the new client code:
byteSize = (int) Integer.valueOf(response);
byte[] byteArray = new byte[byteSize];
FileOutputStream fos = new FileOutputStream(message);
int readBytes = inFromServer.read(byteArray);
// Continuously writes the file to the disk until complete:
int total = 0;
for (int i=0; i<byteArray.length; i++) {
fos.write(byteArray[i]);
total++;
}
fos.close();
Here, we're assuming that because the byteArray array is set to the right size, that the inFromServer.read(byteArray) will populate it - it won't. It's good to assume that any and all read operations will return you just as much data as the system has to hand: in this case, it's probably going to return as soon as it gets the first packet or two, with an underfilled array. This is same as C and Unix read behaviour too.
Try this - I'm repeatedly reading and writing a 4K buffer, until the byte count is reached (as determined by summing the return values of the reads):
byteSize = (int) Integer.valueOf(response);
byte[] byteArray = new byte[4096];
FileOutputStream fos = new FileOutputStream(message);
int total = 0;
// Continuously writes the file to the disk until complete:
while (total < byteSize && (readBytes = inFromServer.read(byteArray)) != -1) {
fos.write(byteArray, 0, readBytes);
total += readBytes;
}
fos.close();
A variant is this - same thing, but byte at a time. Might be a bit clearer. It's going to be slow - all those reads and writes are hitting the OS, but if you put a BufferedInputStream/BufferedOutputStream around the socket/file streams, it'll iron that out. I've added them:
DataInputStream inFromServer =
new DataInputStream(new BufferedInputStream(clientSocket.getInputStream()));
...
byteSize = (int) Integer.valueOf(response);
OutputStream fos = new BufferedOutputStream(FileOutputStream(message));
int total = 0;
int ch;
// Continuously writes the file to the disk until complete:
while (total < byteSize && (ch = inFromServer.read()) != -1) {
fos.write(ch);
total ++;
}
fos.close();
And finally! the simplest answer is this. Your code, but changed to:
int readBytes = inFromServer.readFully(byteArray);
Yes! Those nice people in 1990's Javasoft added a DataInput.readFully method, which does what you want! - basically wraps the code above. It's the simplest solution, and arguably most correct approach: "use existing libraries where possible". OTOH, it's the least educational, and the time you spend getting used to read/writes like this is not deducted from your life-expectancy!
And in fact, the readFully approach has severe limitations. Try pointing it at a 1GB file and see what happens (after you've fixed up the array size at the top): you'll a) run out memory, and b) wish that while you were ingesting a huge blob, you could at least be spooling it out to disk. If you try a 2.5G file, you'll notice that some of those ints should become longs to cope with numbers >= 2^31.
If it was me, I'd do the 4K buffer one. (BTW I'm writing this on a laptop with no Java compiler installed, so I haven't actually run the above! DO respond if there are any difficulties.)

Second packet read from input stream has incomplete data,

I a simple server that receives bytes using TCP and then saves them to a file stream. Through many tests I have seen that the first packet received is always just the filename with no other data. The second packet received only has one byte and it is the first letter of the input text file. After this all packets are sent correctly, but I can't seem to figure out what is messing up the second packet. It also appears that the last packet is written twice. Can anyone see what I am doing wrong? Here is an example Input/Output: https://www.diffchecker.com/srclclrx
InputStream in = clntSock.getInputStream(); //server's input stream - gets data from the client
OutputStream out = clntSock.getOutputStream(); //server's output stream - server sends data to the client
byte[] byteBuffer = new byte[BUFSIZE];
int count = in.read(byteBuffer, 0, BUFSIZE);
String firstRead = new String(byteBuffer, 0, count);
int fileNameEnd = firstRead.indexOf("\r\n");
String fileName = firstRead.substring(0, fileNameEnd);
FileOutputStream fout = new FileOutputStream(fileName); //unzipped file output stream
int contentBegin = fileNameEnd+2;
byte[] oldBuffer = Arrays.copyOfRange(byteBuffer, contentBegin, count);
int oldCount = count-contentBegin;
String oldString = new String(byteBuffer, contentBegin, count-contentBegin, "US-ASCII");
while((count = in.read(byteBuffer, 0, BUFSIZE)) != -1) { // read from origin's buffer into byteBuffer until origin is out of data
String newString = new String(byteBuffer, 0, count, "US-ASCII");
String combinedString = oldString + newString;
int index = combinedString.indexOf("--------MagicStringCSE283Miami");
if(index != -1){
System.out.println("Final Print");
byte[] combinedBuffer = concat(oldBuffer, byteBuffer);
for(int i=0; i<index; i++){
System.out.print((char)combinedBuffer[i]);
}
System.out.println("");
fout.write(combinedBuffer, 0, index);
fout.flush();
fout.close();
break;
}
System.out.println(+ oldCount);
fout.write(oldBuffer, 0, oldCount); //write the byteBuffer's data to the client via the zip output stream
fout.flush(); //push all data out of the zipOutputStream before continuing
if(count == 1){
for(int i=0; i<count; i++){
System.out.println((char)byteBuffer[i]);
}
}
oldBuffer = byteBuffer;
oldCount = count;
oldString = newString;
}
Edit: Another peculiarity to me is that the second to last packet is always just "-" and then the last packet has the remainder of the magic string which terminates the file output stream.

Are you really sure that you are taking the full content of the data you receive ?
while((count = in.read(byteBuffer, 0, BUFSIZE)) != -1) { // read from origin's buffer into byteBuffer until origin is out of data
add logic here to print count
add logic here to print the content of the byteBuffer
}
It is very likely that in your logic you are mistreating what you receive and somehow loose part of the data.
For instance your second packet where you claim to receive only '-' is the count then just equal to 1 ? It is possible that indeed this is the case with TCP, but you really have to verify that you are indeed processing everything that you receive. Based on your explanation I think you are dropping data, well not processing it correctly really.

There are no messages in TCP and no guarantees about how much data each read() will return. It is only specified to transfer at least one byte in blocking mode, unless an error occurs.
One of the consequences of this is that it is impossible to write correct networking code without storing the result of read() in a variable, testing it, and then using it to delimit the amount of data processed.
Your expectations are at fault.

Client-Server not sending the entire file

When I run my client server - which is connecting and I try to send a file it doesn't won't send the whole file which is pulling errors else where, it gets about halfway through and constantly stops at the same part. This set up works when running the server-client on the same machine so I am completely confused
Server --->
// output (a DataOutputstream) is set up elsewhere and messages are sent and received properly
output.writeInt((int)file.length());
// send file
byte [] mybytearray = new byte [(int)file.length()];
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
bis.read(mybytearray,0,mybytearray.length);
System.out.println("Sending " + file + "(" + mybytearray.length + " bytes)");
output.write(mybytearray,0,mybytearray.length);
output.flush();
System.out.println("Done.");
Client --->
// input (a DataInputstream) is set up elsewhere and messages are sent and received properly
String FILE_TO_RECEIVED = "Load_From.xml";
File file = new File(FILE_TO_RECEIVED);
int FILE_SIZE = input.readInt();
if(FILE_SIZE!=0){
// receive file
System.out.println("received file size : " + FILE_SIZE);
byte [] mybytearray = new byte [FILE_SIZE];
FileOutputStream fos = new FileOutputStream(file);
BufferedOutputStream bos = new BufferedOutputStream(fos);
int bytesRead = input.read(mybytearray, 0, mybytearray.length);
bos.write(mybytearray, 0, bytesRead);

You are only reading part of the data, both when reading the file and when reading the socket. The read method returns the number of bytes read, and you need a loop to read everything. For example,
int read = 0, offset = 0;
while ((read = bis.read(mybytearray, offset, mybytearray.length - offset) != -1) {
offset += read;
}
Or you can use classes from the standard library, for example DataInputStream has a readFully method.
DataInputStream dis = new DataInputStream(fis);
dis.readFully(mybytearray);

Please see the documentation of read() at: http://docs.oracle.com/javase/7/docs/api/java/io/InputStream.html#read(byte[],%20int,%20int).
In particular:
An attempt is made to read as many as len bytes, but a smaller number may be read. The number of bytes actually read is returned as an integer.
This means that if the length returned is not what you expect and you haven't encountered the end of file yet, you should call read() again (in a loop).

Send file length with outputstream and receive length and byte[] with inputstream for streaming frames from one device to the other Android/Java

I have searched and searched and everything I have found has been helpful but I keep getting an out of memory error. The images I send are .06 MB so I know the problem isn't from decoding the byte[] into a bitmap. When I remove the while loops this works like a charm for one frame but I want multiple frames. I am getting a byte[] and sending it to a different device using sockets but I am at a loss how to do this. My problem is that I don't send and receive the correct byte[] length. This is what i am doing currently.
while (count != -1) {
//first send the byte[] length
dataOutputStream.writeInt(sendPackage.length);
//pass a byte array
publishProgress("sending file to client");
showMyToastOnUiThread(String.valueOf(sendPackage.length));
outputStream.write(sendPackage, 0, sendPackage.length);
outputStream.flush();
}
Receive byte[] on different device:
int count = inputStream.read();
while (count != -1) {
int byteArrayLength = dataInputStream.readInt();
Log.i(MainActivity.TAG, "Starting convert to byte array");
byte[] receivedBytes = convertInputStreamToByteArray(inputStream, byteArrayLength);
Bitmap bitmap = BitmapFactory.decodeByteArray(receivedBytes, 0, receivedBytes.length);
publishProgress(bitmap);
}
//convert inputstream to byte[]
public byte[] convertInputStreamToByteArray(InputStream inputStream, int readLength) {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] data = new byte[readLength];
try {
Log.i(MainActivity.TAG, "Starting convert to byte array while loop");
int readTotal = 0;
int count = 0;
while (count >= 0 && readTotal < readLength) {
count = inputStream.read(data, readTotal, readLength - readTotal);
if (readLength > 0) {
readTotal += count;
}
}
Log.i(MainActivity.TAG, "Finished convert to byte array while loop");
} catch (IOException e) {
Log.e(MainActivity.TAG, "error: " + e.getMessage());
e.printStackTrace();
}
return data;
}

This is the problem:
int count = inputStream.read();
while (count != -1) {
You're consuming a byte and then ignoring it. That means the next value you read (the size) will be incorrect. You need a different way of telling whether you're at the end of the stream. Some options:
Send a -1 when you're finished; that way you can stop as soon as readInt returns -1
If you know it, send the number of images you're going to send before you start sending them
Use mark(1), then read(), then reset() - if your stream supports marking. I don't know whether it will or not. You could always wrap it in BufferedInputStream if not.
Reimplement DataInputStream.readInt yourself in a way which detects the end of the stream as being an expected possibility instead of throwing an exception
Just catch an exception in readInt (not nice - getting to the end of the stream isn't really exceptional)