Develop Reference

File md5 hash changes when chunking it (for netty transfer)

Question at the bottom
I'm using netty to transfer a file to another server.
I limit my file-chunks to 1024*64 bytes (64KB) because of the WebSocket protocol. The following method is a local example what will happen to the file:
public static void rechunck(File file1, File file2) {
FileInputStream is = null;
FileOutputStream os = null;
try {
byte[] buf = new byte[1024*64];
is = new FileInputStream(file1);
os = new FileOutputStream(file2);
while(is.read(buf) > 0) {
os.write(buf);
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
if(is != null && os != null) {
is.close();
os.close();
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
}
The file is loaded by the InputStream into a ByteBuffer and directly written to the OutputStream.
The content of the file cannot change while this process.
To get the md5-hashes of the file I've wrote the following method:
public static String checksum(File file) {
InputStream is = null;
try {
is = new FileInputStream(file);
MessageDigest digest = MessageDigest.getInstance("MD5");
byte[] buffer = new byte[8192];
int read = 0;
while((read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
return new BigInteger(1, digest.digest()).toString(16);
} catch(IOException | NoSuchAlgorithmException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
is.close();
} catch(IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
return null;
}
So: just in theory it should return the same hash, shouldn't it? The problem is that it returns two different hashes that do not differ with every run.. file size stays the same and the content either.
When I run the method once for in: file-1, out: file-2 and again with in: file-2 and out: file-3 the hashes of file-2 and file-3 are the same! This means the method will properly change the file every time the same way.
1. 58a4a9fbe349a9e0af172f9cf3e6050a
2. 7b3f343fa1b8c4e1160add4c48322373
3. 7b3f343fa1b8c4e1160add4c48322373
Here is a little test that compares all buffers if they are equivalent. Test is positive. So there aren't any differences.
File file1 = new File("controller/templates/Example.zip");
File file2 = new File("controller/templates2/Example.zip");
try {
byte[] buf1 = new byte[1024*64];
byte[] buf2 = new byte[1024*64];
FileInputStream is1 = new FileInputStream(file1);
FileInputStream is2 = new FileInputStream(file2);
boolean run = true;
while(run) {
int read1 = is1.read(buf1), read2 = is2.read(buf2);
String result1 = Arrays.toString(buf1), result2 = Arrays.toString(buf2);
boolean test = result1.equals(result2);
System.out.println("1: " + result1);
System.out.println("2: " + result2);
System.out.println("--- TEST RESULT: " + test + " ----------------------------------------------------");
if(!(read1 > 0 && read2 > 0) || !test) run = false;
}
} catch (IOException e) {
e.printStackTrace();
}
Question: Can you help me chunking the file without changing the hash?

while(is.read(buf) > 0) {
os.write(buf);
}
The read() method with the array argument will return the number of files read from the stream. When the file doesn't end exactly as a multiple of the byte array length, this return value will be smaller than the byte array length because you reached the file end.
However your os.write(buf); call will write the whole byte array to the stream, including the remaining bytes after the file end. This means the written file gets bigger in the end, therefore the hash changed.
Interestingly you didn't make the mistake when you updated the message digest:
while((read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
You just have to do the same when you "rechunk" your files.

Your rechunk method has a bug in it. Since you have a fixed buffer in there, your file is split into ByteArray-parts. but the last part of the file can be smaller than the buffer, which is why you write too many bytes in the new file. and that's why you do not have the same checksum anymore. the error can be fixed like this:
public static void rechunck(File file1, File file2) {
FileInputStream is = null;
FileOutputStream os = null;
try {
byte[] buf = new byte[1024*64];
is = new FileInputStream(file1);
os = new FileOutputStream(file2);
int length;
while((length = is.read(buf)) > 0) {
os.write(buf, 0, length);
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
if(is != null)
is.close();
if(os != null)
os.close();
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
}
Due to the length variable, the write method knows that until byte x of the byte array, only the file is off, then there are still old bytes in it that no longer belong to the file.

Files downloaded as Binary with Java are corrupted

I have written an downloader which should be used to download text files, as well as images. So I download the files as binaries. Many of the downloads work very well, but some parts of the text files and many image files are corrupted. The errors occur always at the same files and at the same places (as long as I can tell when analysing the text files). I used this code for downloading:
public File downloadFile(HttpURLConnection connection) {
return writeFileDataToFile(getFileData(connection));
}
//downloads the data of the file and returns the content as string
private List<Byte> getFileData(HttpURLConnection connection) {
List<Byte> fileData = new ArrayList<>();
try (InputStream input = connection.getInputStream()) {
byte[] fileChunk = new byte[8*1024];
int bytesRead;
do {
bytesRead = input.read(fileChunk);
if (bytesRead != -1) {
fileData.addAll(Bytes.asList(fileChunk));
fileChunk = new byte[8*1024];
}
} while (bytesRead != -1);
return fileData;
} catch (IOException e) {
System.out.println("Receiving file at " + url.toString() + " failed");
System.exit(1);
return null; //shouldn't be reached
}
}
//writes data to the file
private File writeFileDataToFile(List<Byte> fileData) {
if (!this.file.exists()) {
try {
this.file.getParentFile().mkdirs();
this.file.createNewFile();
} catch (IOException e) {
System.out.println("Error while creating file at " + file.getPath());
System.exit(1);
}
}
try (OutputStream output = new FileOutputStream(file)) {
output.write(Bytes.toArray(fileData));
return file;
} catch (IOException e) {
System.out.println("Error while accessing file at " + file.getPath());
System.exit(1);
return null;
}
}

I could suggest you to not pass through List of Byte, since you create a list of Byte from an array, to get it back to an array of Byte, which is not really efficient.
Moreover you wrongly assume the chunk size (not necesseraly 8192 bytes).
Why don't you just do something as:
private File writeFileDataToFile(HttpURLConnection connection) {
if (!this.file.exists()) {
try {
this.file.getParentFile().mkdirs();
//this.file.createNewFile(); // not needed, will be created at FileOutputStream
} catch (IOException e) {
System.out.println("Error while creating file at " + file.getPath());
//System.exit(1);
// instead do a throw of error or return null
throw new YourException(message);
}
}
OutputStream output = null;
InputStream input = null;
try {
output = new FileOutputStream(file):
input = connection.getInputStream();
byte[] fileChunk = new byte[8*1024];
int bytesRead;
while ((bytesRead = input.read(fileChunk )) != -1) {
output.write(fileChunk , 0, bytesRead);
}
return file;
} catch (IOException e) {
System.out.println("Receiving file at " + url.toString() + " failed");
// System.exit(1); // you should avoid such exit
// instead do a throw of error or return null
throw new YourException(message);
} finally {
if (input != null) {
try {
input.close();
} catch (Execption e2) {} // ignore
}
if (output != null) {
try {
output.close();
} catch (Execption e2) {} // ignore
}
}
}

The failure was adding the whole fileChunk Array to file data, even if it wasn't completely filled by the read operation.
Fix:
//downloads the data of the file and returns the content as string
private List<Byte> getFileData(HttpURLConnection connection) {
List<Byte> fileData = new ArrayList<>();
try (InputStream input = connection.getInputStream()) {
byte[] fileChunk = new byte[8*1024];
int bytesRead;
do {
bytesRead = input.read(fileChunk);
if (bytesRead != -1) {
fileData.addAll(Bytes.asList(Arrays.copyOf(fileChunk, bytesRead)));
}
} while (bytesRead != -1);
return fileData;
} catch (IOException e) {
System.out.println("Receiving file at " + url.toString() + " failed");
System.exit(1);
return null; //shouldn't be reached
}
}
Where the relevant change is changing
if (bytesRead != -1) {
fileData.addAll(Bytes.asList(fileChunk));
fileChunk = new byte[8*1024];
}
into
if (bytesRead != -1) {
fileData.addAll(Bytes.asList(Arrays.copyOf(fileChunk, bytesRead)));
}

Blocking multithreaded TCP Server NullPointerException debugging

I have a TCP server that accepts data and saves it to a text file. It then uses that text file to create an image and sends it back to the client. Every couple of hours I will get a NullPointerException that gets thrown to every client that connects after that. I am not sure how to go about debugging this as I cannot replicate it on my own.
Does anyone have any debugging practices to help me figure out why this is becoming a problem?
The server running is running Ubuntu 12.04 i386 with 2 gigs of RAM. My initial suspicion is that something is not getting closed properly and creating issues but everything should be getting closed as far as I can tell.
ServerSocket echoServer = null;
Socket clientSocket = null;
try {
echoServer = new ServerSocket(xxx);
} catch (IOException e) {
System.out.println(e);
}
while(true)
{
InputStream is = null;
FileOutputStream fos = null;
BufferedOutputStream bos = null;
int bufferSize = 0;
FileInputStream fis = null;
BufferedInputStream bis = null;
BufferedOutputStream out = null;
try {
//Receieve text file
is = null;
fos = null;
bos = null;
bufferSize = 0;
String uid = createUid();
try {
clientSocket = echoServer.accept();
clientSocket.setKeepAlive(true);
clientSocket.setSoTimeout(10000);
System.out.println("Client accepted from: " + clientSocket.getInetAddress());
} catch (IOException ex) {
System.out.println("Can't accept client connection. ");
}
try {
is = clientSocket.getInputStream();
bufferSize = clientSocket.getReceiveBufferSize();
System.out.println("Buffer size: " + bufferSize);
} catch (IOException ex) {
System.out.println("Can't get socket input stream. ");
}
try {
fos = new FileOutputStream("/my/diretory/" + uid + ".txt");
bos = new BufferedOutputStream(fos);
} catch (FileNotFoundException ex) {
System.out.println("File not found. ");
}
byte[] bytes = new byte[bufferSize];
int count;
while ((count = is.read(bytes)) > 0) {
bos.write(bytes, 0, count);
System.out.println("Receiving... " + count);
}
System.out.println("Done receiving text file");
bos.flush();
bos.close();
fos.close();
//image
String[] command = new String[3];
command[0] = "python";
command[1] = "imagecreationfile.py";
command[2] = uid;
System.out.println("Starting python script");
Boolean success = startScript(command);
if(success)
{
System.out.println("Script completed successfully");
//Send image here
String image = "/my/directory/" + uid + ".png";
File imageFile = new File(image);
long length = imageFile.length();
if (length > Integer.MAX_VALUE) {
System.out.println("File is too large.");
}
bytes = new byte[(int) length];
fis = new FileInputStream(imageFile);
bis = new BufferedInputStream(fis);
out = new BufferedOutputStream(clientSocket.getOutputStream());
count = 0;
while ((count = bis.read(bytes)) > 0) {
out.write(bytes, 0, count);
System.out.println("Writing... " + count);
}
out.flush();
out.close();
fis.close();
bis.close();
}
else
{
System.out.println("Script failed");
}
System.out.println("Closing connection");
is.close();
clientSocket.close();
} catch (Exception e) {
System.out.println(e); //This is where the exception is being caught
}
if(!clientSocket.isClosed())
{
try {
clientSocket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
try {
if(is != null)
is.close();
if(fos != null)
fos.close();
if(bos != null)
bos.close();
if(fis != null)
fis.close();
if(bis != null)
bis.close();
if(out != null)
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}

Maybe exception was thrown in one of your try-catch scope.
And the next try-catch scope found null variables.
for example
//Receieve text file
is = null;
fos = null;
bos = null;
bufferSize = 0;
String uid = createUid();
try {
clientSocket = echoServer.accept();
clientSocket.setKeepAlive(true);
clientSocket.setSoTimeout(10000);
System.out.println("Client accepted from: " + clientSocket.getInetAddress());
} catch (IOException ex) {
System.out.println("Can't accept client connection. ");
}
try {
is = clientSocket.getInputStream();
bufferSize = clientSocket.getReceiveBufferSize();
System.out.println("Buffer size: " + bufferSize);
} catch (IOException ex) {
System.out.println("Can't get socket input stream. ");
}
if IOException was thrown in "clientSocket = echoServer.accept();" , it will print "Can't accept client connection. ".
When, "is = clientSocket.getInputStream();" executed, it will throw NullPointer because "clientSocket" was not initialized properly.
My suggestion, dont break a sequenced statement in different try-catch scope until it necessary.

FileInputStream / FileOutputStream blocking?

I have the following code which successfully copies a file. However, there are two problems with it:
The System.out.println() immediately after the progressBar.setValue() does not print intervals between 0 and 100 (just prints "0" till the end where it prints "100")
Besides the fact that the value for the progress bar might be wrong somehow due to question #1, in the actual code I am making other visual changes too, but they don't show until the entire file is processed. I thought the FileInputStream/FileOutputStream functions were non-blocking. How can I change the following code so that the progress bar is in fact updated during the operation?
startJob method:
private void startJob(File inFile, File outFile) {
long offset = 0;
int numRead = 0;
byte[] bytes = new byte[8192];
long fileLength = inFile.length();
Boolean keepGoing = true;
progressBar.setValue(0);
try {
inputStream = new FileInputStream(inFile);
outputStream = new FileOutputStream(outFile, false);
System.out.println("Total file size to read (in bytes) : " + inputStream.available());
} catch (FileNotFoundException err) {
inputStream = null;
outputStream = null;
err.printStackTrace();
} catch (IOException err) {
inputStream = null;
outputStream = null;
err.printStackTrace();
}
if (inputStream != null && outputStream != null) {
while (keepGoing) {
try {
numRead = inputStream.read(bytes);
outputStream.write(bytes, 0, numRead);
} catch (IOException err) {
keepGoing = false;
err.printStackTrace();
}
if (numRead > 0) {
offset += numRead;
}
if (offset >= fileLength) {
keepGoing = false;
}
progressBar.setValue(Math.round(offset / fileLength) * 100);
System.out.println(Integer.toString(Math.round(offset / fileLength) * 100));
}
}
if (offset < fileLength) {
//error
} else {
//success
}
try {
inputStream.close();
outputStream.close();
} catch (IOException err) {
err.printStackTrace();
}
}

I suspect you are calling your lengthy method from the EDT. Remove your operation from the EDT by placing it in it's own Runnable for instance and then call
SwingUtilities.invokeLater(new Runnable() {
#Override
public void run() {
progressBar.setValue(value);
// or any other GUI changes you want to make
}
});
Otherwise, your operation blocks the EDT until it is done, and with the EDT blocked no events like repaint etc will can be processed -> no GUI changes visible until the end.

The value of expression Math.round(offset / fileLength) will always equal 0 (zero), because offset < fileLength.
UPD:
If you want to do this calculation correctly, you have to change it to:
Math.round(((double)offset / (double)fileLength) * 100)

Android - Jave Server AES encryption Not working - Different keys

I am trying to encrypt images on android, send them to the server so that it process them. The server has to decrypt the received message. I already posted the code in this question
I have called the encrypt function on the Android side and the decrypt function the java server side (image is sent via TCP).
However, I am receiving the error:
javax.crypto.BadPaddingException: Given final block not properly padded
I output the key on android and got:
javax.crypto.spec.SecretKeySpec#652
whereas on the Java server (developped using Netbeans) I got:
javax.crypto.spec.SecretKeySpec#148dd
I thought that the padding is different so I used
AES/ECB/PKCS5Padding
instead of
AES
but the Java server output an error:
should use AES.
How can i solve this issue?
For send on android side:
public void send(Bitmap mRgbImage1_array, int port_number)
throws IOException {
socket = new Socket("192.168.0.107", port_number);
boolean encrypt = HomeScreen.checkbox2.isChecked();
ByteArrayOutputStream stream = new ByteArrayOutputStream();
mRgbImage1_array.compress(CompressFormat.JPEG, 100, stream);
InputStream photoStream = new ByteArrayInputStream(stream.toByteArray());
BufferedInputStream bis = new BufferedInputStream(photoStream);
byte[] mybytearray = new byte[photoStream.available()];
bis.read(mybytearray, 0, mybytearray.length);
photoStream.close();
if(encrypt)
{
try {
byte[] dst = Security.encrypt(mybytearray);
mybytearray = new byte[10000];
for(int i=0; i<dst.length;i++)
{
mybytearray[i] = dst[i];
}
} catch (Exception e1) {
/// TODO Auto-generated catch block
e1.printStackTrace();
}
}
os = socket.getOutputStream();
os.write(mybytearray);
os.flush();
os.close();
if (os != null)
{
try {
os.close();
}
catch (IOException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
For receive on Java server side:
public static void receive(int port_number) {
boolean received = false;
Socket socket = null;
InputStream is = null;
int bytesRead;
int current = 0;
BufferedOutputStream bos = null;
try {
if (serverSocket == null) {
serverSocket = new ServerSocket(port_number);
}
System.out.println("Listening :" + port_number);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
while (!received) {
try {
socket = serverSocket.accept();
InetAddress ip = socket.getInetAddress();
String[] ip1 = ip.toString().split("/");
ip2 = ip1[1];
System.out.println("ip is " + ip2);
byte[] mybytearray = new byte[10000000];
is = socket.getInputStream();
FileOutputStream fos = new FileOutputStream("source-image.jpeg");
bos = new BufferedOutputStream(fos);
bytesRead = is.read(mybytearray, 0, mybytearray.length);
current = bytesRead;
do {
bytesRead =
is.read(mybytearray, current, (mybytearray.length - current));
if (bytesRead >= 0) {
current += bytesRead;
}
} while (bytesRead > -1);
if(Networker.should_encrypt)
{
try {
mybytearray = Security1.decrypt(mybytearray);
} catch (Exception e1) {
/// TODO Auto-generated catch block
e1.printStackTrace();
}
}
bos.write(mybytearray, 0, current);
bos.flush();
bos.close();
received = true;
} catch (IOException ex) {
Logger.getLogger(MyServer1.class.getName()).log(Level.SEVERE, null, ex);
} finally {
if (socket != null) {
try {
socket.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
if (is != null) {
try {
is.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
}

I believe this is the problem:
mybytearray = Security1.decrypt(mybytearray);
That's always going to decrypt 10000000 bytes, even if you've only actually written a small amount of data. You should change your decrypt method to say how much data to decrypt, then call doFinal(byte[], int, int).
I'd also suggest trying to handle the encryption/decryption in a streaming manner rather than preallocating 10MB (which is going to be wasteful in most cases, and could be too short in others), but that's a larger change.
Additionally, this is a bad idea in the encrypting code:
byte[] mybytearray = new byte[photoStream.available()];
bis.read(mybytearray, 0, mybytearray.length);
photoStream.close();
You're assuming that the amount available to start with is the whole file. That may not be the case. You should generally loop round, reading from the file stream and writing to an encrypting stream. Oh, and the close() call should be in a finally block.
If you really want to do all the encryption/decryption in one call, you can loop round reading from the file or network stream and writing into a ByteArrayOutputStream, so that you don't need to hard-code the size or assume it from available(). Then use ByteArrayOutputStream.toByteArray() to get a byte array of the right size. (That involves an extra copy, admittedly.)