Develop Reference

Why socket send only 1KB (425B) of data?

I am learning sockets and now I want to write file transfer program. I have server part and client part. Server part contains 2 ports: 5000 (commands) and 5001 (files). Now I want to send a file via socket and when I did something is wrong because only 425B of data is sending.
Here is client send method:
private void sendFile(Socket socket) {
File file2 = new File("C:\\Users\\barte\\Desktop\\dos.png");
byte[] bytes = new byte[16 * 1024];
System.out.println(file2.exists());
try (InputStream inputStream = new FileInputStream(file2);
OutputStream outputStream = socket.getOutputStream();
OutputStream secondOutput = new FileOutputStream("C:\\Users\\barte\\Desktop\\received\\dos.png")) {
int count;
while ((count = inputStream.read(bytes)) > 0) {
outputStream.write(bytes, 0, count);
secondOutput.write(bytes, 0, count);
}
socket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
As you can see (image below) I am writing this file also locally and everything is ok, all of 73KB of data is writed.
Now, on server side I am trying to receive this file:
case SEND: {
new Thread(() -> {
printWriter.println("Server is receiving files right now...");
try (ServerSocket serverSocket = new ServerSocket(5001)) {
while (true) {
new FilesTransfer(serverSocket.accept()).start();
}
} catch (IOException e) {
e.printStackTrace();
}
}).start();
break;
}
And inside FilesTransfer run method:
#Override
public void run() {
System.out.println("Hello there");
try {
InputStream inputStream = inSocket.getInputStream();
OutputStream outputStream = new FileOutputStream("C:\\Users\\barte\\Desktop\\received\\file");
byte[] bytes = new byte[16 * 1024];
int count;
while ((count = inputStream.read()) > 0) {
outputStream.write(bytes, 0, count);
}
outputStream.close();
inputStream.close();
inSocket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Where is a bug? Why only empty bytes are sending when locally everything it's fine?

The problem is:
while ((count = inputStream.read()) > 0) {
Your code uses InputStream.read(), which reads individual bytes (or -1 when end-of-stream). Right now, you are reading individual bytes, interpreting that as a length, and then writing that number of 0x00 bytes from bytes to the file. This stops when you read a 0x00 byte from the stream.
You need to change this to use InputStream.read(byte[]):
while ((count = inputStream.read(bytes)) != -1) {
That is, you need to pass bytes in, and check for the result being unequal to -1, not if it is greater than zero (0), although read(byte[]) will only return 0 if the passed in byte array has length zero, so that is not a real concern.

You could do it in this way:
#Override
public void run() {
System.out.println("Hello there");
try {
InputStream inputStream = inSocket.getInputStream();
OutputStream outputStream = new FileOutputStream("C:\\Users\\barte\\Desktop\\received\\file");
byte[] bytes = new byte[16 * 1024];
int byteRead= 1;
while (byteRead > -1) {
byteRead= inputStream.read();
outputStream.write(byteRead);
}
outputStream.close();
inputStream.close();
inSocket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Actually END OF FILE or EOF means -1 and you did > 0 so 0 was taken and it stopped the connection saving the file.
I also recommend to write a logic to transfer the filename as a command to the server so that the file is saved with the correct name and extension!

File md5 hash changes when chunking it (for netty transfer)

Question at the bottom
I'm using netty to transfer a file to another server.
I limit my file-chunks to 1024*64 bytes (64KB) because of the WebSocket protocol. The following method is a local example what will happen to the file:
public static void rechunck(File file1, File file2) {
FileInputStream is = null;
FileOutputStream os = null;
try {
byte[] buf = new byte[1024*64];
is = new FileInputStream(file1);
os = new FileOutputStream(file2);
while(is.read(buf) > 0) {
os.write(buf);
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
if(is != null && os != null) {
is.close();
os.close();
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
}
The file is loaded by the InputStream into a ByteBuffer and directly written to the OutputStream.
The content of the file cannot change while this process.
To get the md5-hashes of the file I've wrote the following method:
public static String checksum(File file) {
InputStream is = null;
try {
is = new FileInputStream(file);
MessageDigest digest = MessageDigest.getInstance("MD5");
byte[] buffer = new byte[8192];
int read = 0;
while((read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
return new BigInteger(1, digest.digest()).toString(16);
} catch(IOException | NoSuchAlgorithmException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
is.close();
} catch(IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
return null;
}
So: just in theory it should return the same hash, shouldn't it? The problem is that it returns two different hashes that do not differ with every run.. file size stays the same and the content either.
When I run the method once for in: file-1, out: file-2 and again with in: file-2 and out: file-3 the hashes of file-2 and file-3 are the same! This means the method will properly change the file every time the same way.
1. 58a4a9fbe349a9e0af172f9cf3e6050a
2. 7b3f343fa1b8c4e1160add4c48322373
3. 7b3f343fa1b8c4e1160add4c48322373
Here is a little test that compares all buffers if they are equivalent. Test is positive. So there aren't any differences.
File file1 = new File("controller/templates/Example.zip");
File file2 = new File("controller/templates2/Example.zip");
try {
byte[] buf1 = new byte[1024*64];
byte[] buf2 = new byte[1024*64];
FileInputStream is1 = new FileInputStream(file1);
FileInputStream is2 = new FileInputStream(file2);
boolean run = true;
while(run) {
int read1 = is1.read(buf1), read2 = is2.read(buf2);
String result1 = Arrays.toString(buf1), result2 = Arrays.toString(buf2);
boolean test = result1.equals(result2);
System.out.println("1: " + result1);
System.out.println("2: " + result2);
System.out.println("--- TEST RESULT: " + test + " ----------------------------------------------------");
if(!(read1 > 0 && read2 > 0) || !test) run = false;
}
} catch (IOException e) {
e.printStackTrace();
}
Question: Can you help me chunking the file without changing the hash?

while(is.read(buf) > 0) {
os.write(buf);
}
The read() method with the array argument will return the number of files read from the stream. When the file doesn't end exactly as a multiple of the byte array length, this return value will be smaller than the byte array length because you reached the file end.
However your os.write(buf); call will write the whole byte array to the stream, including the remaining bytes after the file end. This means the written file gets bigger in the end, therefore the hash changed.
Interestingly you didn't make the mistake when you updated the message digest:
while((read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
You just have to do the same when you "rechunk" your files.

Your rechunk method has a bug in it. Since you have a fixed buffer in there, your file is split into ByteArray-parts. but the last part of the file can be smaller than the buffer, which is why you write too many bytes in the new file. and that's why you do not have the same checksum anymore. the error can be fixed like this:
public static void rechunck(File file1, File file2) {
FileInputStream is = null;
FileOutputStream os = null;
try {
byte[] buf = new byte[1024*64];
is = new FileInputStream(file1);
os = new FileOutputStream(file2);
int length;
while((length = is.read(buf)) > 0) {
os.write(buf, 0, length);
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
if(is != null)
is.close();
if(os != null)
os.close();
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
}
Due to the length variable, the write method knows that until byte x of the byte array, only the file is off, then there are still old bytes in it that no longer belong to the file.

inputStream read method constantly reading 0 [duplicate]

This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 6 years ago.
I am first transferring a file from a client to my master, the stores the byte array and then sends to the slave. Where the slave stores the byte array. But when The file is sent properly from client to master but when I send the byte array to the slave it to the slave the read method in input stream constantly reads 0.
// This method writes the file to the master
public void writeFile(File file) {
try {
this.write(String.valueOf(file.length()));
byte[] bytearray = new byte[(int) file.length()];
FileInputStream fin = new FileInputStream(file);
BufferedInputStream bin = new BufferedInputStream(fin);
bin.read(bytearray, 0, bytearray.length);
BufferedOutputStream bos;
OutputStream os = socket.getOutputStream();
bos= new BufferedOutputStream(os);
bos.write(bytearray, 0, bytearray.length);
bos.flush();
} catch (IOException e) {
e.printStackTrace();
}
}
//This method reads the file into the master as a byte array and the byte array from the master into slave
public byte[] readFile() {
byte[] bytearray = null;
try {
int currentTot = 0;
int filesize = Integer.parseInt(this.read());
System.out.println(filesize);
bytearray = new byte[filesize];
InputStream is = socket.getInputStream();
int bytesRead;
bytesRead = is.read(bytearray, 0, bytearray.length);
currentTot = bytesRead;
int count = 0;
do {
bytesRead = is.read(bytearray, currentTot, (bytearray.length - currentTot));
if (bytesRead > 0) {
currentTot += bytesRead;
count = 0;
} else {
count++;
System.out.println("count " + count);
}
} while (bytesRead > -1);
System.out.println(currentTot);
// bos.write(bytearray, 0, currentTot);
// bos.flush();
// bos.close();
} catch (IOException e) {
e.printStackTrace();
}
return bytearray;
}
//This method writes from the master to the slave
public void writeByte(byte[] m) {
this.write(String.valueOf(m.length));
System.out.println("File side inside sender" + m.length);
// byte[] bytearray = m;
OutputStream os;
try {
os = socket.getOutputStream();
os.write(m, 0, m.length);
os.flush();
//os.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Interestingly if I close my output stream after I send my byte array from my master it works well. But I cannot close stream because the slave needs to communicate with the master further. Thanks in advance.
public void write(String output) {
if (pw == null)
this.openWriter();
pw.println(output);
}
public String read() {
try {
if (br == null) {
if (this.socket != null)
br = new BufferedReader(new InputStreamReader(this.socket.getInputStream()));
}
return br.readLine();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}

You're misreading the file length in the receiver. You are getting zero, so you're constructing a zero length byte array, so read() returns zero.
You need to send the length via DataOutputStream.writeLong() and read it via DataInputStream.readLong(). And then your sending and receiving code is all wrong as well. See my answer here for complete code.

Java Server/Client socket stops responding

I am having trouble with my client/server program. The server is supposed to get a file from the client then convert it to xml, and then stream the xml file back to the client.
As of now everything works up to the point of streaming the xml file back to the client. I can successfully send my file to the server and convert it to XML.
The server does not send anything to the client. The client is stuck in the area of reading bytes in from the InputStream. On line 57 in the Client class.
public class Server {
private ServerSocket serverSocket;
private Socket clientSocket;
private PrintWriter out;
private BufferedReader in;
public static void main(String[] args) {
int port = 8081;
Server srv = new Server(port);
}
public Server(int portNumber) {
try {
serverSocket = new ServerSocket(portNumber);
clientSocket = serverSocket.accept();
out = new PrintWriter(clientSocket.getOutputStream(), true);
//in = new BufferedReader(
//new InputStreamReader(clientSocket.getInputStream()));
byte[] myArray = new byte[22000]; // should be file size
InputStream is = clientSocket.getInputStream();
FileOutputStream fos = new FileOutputStream("file.csv");
BufferedOutputStream bos = new BufferedOutputStream(fos);
int bytesRead;
int current = 0;
bytesRead = is.read(myArray, 0, myArray.length);
current = bytesRead;
do {
bytesRead = is.read(myArray, current, (myArray.length - current) );
if (bytesRead >= 0)
current += bytesRead;
} while (bytesRead > -1);
bos.write(myArray, 0, myArray.length);
bos.flush();
boolean flag = false;
System.out.println("Server: finished receiving file");
XMLWriter xmlWrite = new XMLWriter();
xmlWrite.createXmlDocument("file_copy.csv");
sendXML("server_file.XML");
clientSocket.close();
System.out.println("Server: disconnected with client");
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void sendXML(String fileName) {
System.out.println("Sending file");
File file = new File(fileName);
BufferedInputStream bis;
try {
FileInputStream fis = new FileInputStream(file);
bis = new BufferedInputStream(fis);
DataInputStream dataIn = new DataInputStream( bis );
OutputStream outStream = clientSocket.getOutputStream();
int length;
try {
length = dataIn.readInt();
System.out.println("S: " + dataIn.readInt());
byte[] data = new byte[ length ];
dataIn.readFully(data, 0, data.length);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
}
public class Client {
private Socket mainSocket;
public static void main(String[] args) {
int port = 8081;
Client cli = new Client(port);
}
public Client(int portNumber) {
try {
mainSocket = new Socket("localhost", portNumber);
// send file to server
File file = new File("Passengers.csv");
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
OutputStream outStream = mainSocket.getOutputStream();
byte[] myBytes = new byte[(int) file.length()];
bis.read(myBytes, 0, myBytes.length);
outStream.write(myBytes, 0, myBytes.length);
outStream.flush();
System.out.println("Client: Done sending file");
byte[] fileData = new byte[30000];
InputStream is = mainSocket.getInputStream();
FileOutputStream fos = new FileOutputStream("client_file.XML");
BufferedOutputStream bos = new BufferedOutputStream(fos);
DataOutputStream dataOut = new DataOutputStream( bos );
dataOut.writeInt( fileData.length );
dataOut.write( fileData );
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

This loop will run until the stream is closed, not until the client finishes sending data.
do {
bytesRead = is.read(myArray, current, (myArray.length - current) );
if (bytesRead >= 0)
current += bytesRead;
} while (bytesRead > -1);
This is because InputStream.read will block until at least one byte of data is available or the stream is closed, causing a -1 to be returned. Since the client is done sending data but the stream is still open it will block forever.
A solution is to first send the length of the file in the first 4 bytes using something like DataOutputStream/DataInputStream, and then have the client send exactly that many bytes and the server read exactly that many bytes before moving on.
Client:
// Connect to the server and read in file data
byte[] fileData = ...;
DataOutputStream dataOut = new DataOutputStream( outStream );
dataOut.writeInt( fileData.length );
dataOut.write( fileData );
// Get reply from server
Server:
// Create ServerSocket and get Client connection
DataInputStream dataIn = new DataInputStream( is );
int length = dataIn.readInt();
byte[] data = new byte[ length ];
dataIn.readFully( data );
// Continue to process client connection
The general idea behind this kind of network communication is that every block of data, or packet, is prefixed with it's length before being send. This means that the recipient of the data can read in that number and know exactly how many bytes remain to form the complete block or packet. Communication goes along something like this for both sides of the connection. The only difference is in how you process the data.
public byte[] readPacket( DataInputStream dataIn ) throws IOException {
int length = dataIn.readInt();
byte[] packet = new byte[ length ];
dataIn.readFully( packet );
return packet;
}
public void writePacket( DataOutputStream dataOut, byte[] packet ) throws IOException {
dataOut.writeInt( packet.length );
dataOut.write( packet );
}
Only you would call these from some sort of loop preferably on background threads but it's not required for what you want to do. For your case you want to read in the file on the client side into a byte[] and then use writePacket to send it to the server. On the server side you would use readPacket to read in the file from the client. The same thing takes place when sending data back to the client but with the roles switched.

DataOutputStream writing too much

What I currently have
I'm currently trying to create a little download manager in Java and I have a problem with writing the loaded bytes in a file. I'm using a DataOutputStream to write the byte-array which I read from a DataInputStream. Here is the class I created to do that:
public class DownloadThread extends Thread{
private String url_s;
private File datei;
public DownloadThread(String url_s, File datei){
this.url_s = url_s;
this.datei = datei;
}
public void run(){
// Connect:
int size = 0;
URLConnection con = null;
try {
URL url = new URL(url_s);
con = url.openConnection();
size = con.getContentLength();
// Set Min and Max of the JProgressBar
prog.setMinimum(0);
prog.setMaximum(size);
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
// Download:
if (con != null || size != 0){
byte[] buffer = new byte[4096];
DataInputStream down_reader = null;
// Output:
DataOutputStream out = null;
try {
out = new DataOutputStream(new FileOutputStream(datei));
} catch (FileNotFoundException e1) {
e1.printStackTrace();
}
// Load:
try {
down_reader = new DataInputStream(con.getInputStream());
int byte_counter = 0;
int tmp = 0;
int progress = 0;
// Read:
while (true){
tmp = down_reader.read(buffer);
// Check for EOF
if (tmp == -1){
break;
}
out.write(buffer);
out.flush();
// Set Progress:
byte_counter += tmp;
progress = (byte_counter * 100) / size;
prog.setValue( byte_counter );
prog.setString(progress+"% - "+byte_counter+"/"+size+" Bytes");
}
// Check Filesize:
prog.setString("Checking Integrity...");
if (size == out.size()){
prog.setString("Integrity Check passed!");
} else {
prog.setString("Integrity Check failed!");
System.out.println("Size: "+size+"\n"+
"Read: "+byte_counter+"\n"+
"Written: "+out.size() );
}
// ENDE
} catch (IOException e) {
e.printStackTrace();
} finally{
try {
out.close();
down_reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
// Clean Up...
load.setEnabled(true);
try {
this.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
This is currently an inner-class and the prog-Object is a JProgressBar from it's mother-class, so it can be accessed directly.
Example
I'm trying to download the Windows .exe Version of the TinyCC, which should be 281KB size. The file i downloaded with my download manager is 376KB big.
The Output from the Script looks like this:
Size: 287181
Read: 287181
Written: 385024
So it seems that the read bytes match the file-size but there are more bytes written. What am I missing here?

This is wrong:
out.write(buffer);
It should be
out.write(buffer, 0, tmp);
You need to specify how many bytes to write, a read doesn't always read a full buffer.

Memorize this. It is the canonical way to copy a stream in Java.
int count;
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}