Develop Reference

File md5 hash changes when chunking it (for netty transfer)

Question at the bottom
I'm using netty to transfer a file to another server.
I limit my file-chunks to 1024*64 bytes (64KB) because of the WebSocket protocol. The following method is a local example what will happen to the file:
public static void rechunck(File file1, File file2) {
FileInputStream is = null;
FileOutputStream os = null;
try {
byte[] buf = new byte[1024*64];
is = new FileInputStream(file1);
os = new FileOutputStream(file2);
while(is.read(buf) > 0) {
os.write(buf);
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
if(is != null && os != null) {
is.close();
os.close();
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
}
The file is loaded by the InputStream into a ByteBuffer and directly written to the OutputStream.
The content of the file cannot change while this process.
To get the md5-hashes of the file I've wrote the following method:
public static String checksum(File file) {
InputStream is = null;
try {
is = new FileInputStream(file);
MessageDigest digest = MessageDigest.getInstance("MD5");
byte[] buffer = new byte[8192];
int read = 0;
while((read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
return new BigInteger(1, digest.digest()).toString(16);
} catch(IOException | NoSuchAlgorithmException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
is.close();
} catch(IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
return null;
}
So: just in theory it should return the same hash, shouldn't it? The problem is that it returns two different hashes that do not differ with every run.. file size stays the same and the content either.
When I run the method once for in: file-1, out: file-2 and again with in: file-2 and out: file-3 the hashes of file-2 and file-3 are the same! This means the method will properly change the file every time the same way.
1. 58a4a9fbe349a9e0af172f9cf3e6050a
2. 7b3f343fa1b8c4e1160add4c48322373
3. 7b3f343fa1b8c4e1160add4c48322373
Here is a little test that compares all buffers if they are equivalent. Test is positive. So there aren't any differences.
File file1 = new File("controller/templates/Example.zip");
File file2 = new File("controller/templates2/Example.zip");
try {
byte[] buf1 = new byte[1024*64];
byte[] buf2 = new byte[1024*64];
FileInputStream is1 = new FileInputStream(file1);
FileInputStream is2 = new FileInputStream(file2);
boolean run = true;
while(run) {
int read1 = is1.read(buf1), read2 = is2.read(buf2);
String result1 = Arrays.toString(buf1), result2 = Arrays.toString(buf2);
boolean test = result1.equals(result2);
System.out.println("1: " + result1);
System.out.println("2: " + result2);
System.out.println("--- TEST RESULT: " + test + " ----------------------------------------------------");
if(!(read1 > 0 && read2 > 0) || !test) run = false;
}
} catch (IOException e) {
e.printStackTrace();
}
Question: Can you help me chunking the file without changing the hash?

while(is.read(buf) > 0) {
os.write(buf);
}
The read() method with the array argument will return the number of files read from the stream. When the file doesn't end exactly as a multiple of the byte array length, this return value will be smaller than the byte array length because you reached the file end.
However your os.write(buf); call will write the whole byte array to the stream, including the remaining bytes after the file end. This means the written file gets bigger in the end, therefore the hash changed.
Interestingly you didn't make the mistake when you updated the message digest:
while((read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
You just have to do the same when you "rechunk" your files.

Your rechunk method has a bug in it. Since you have a fixed buffer in there, your file is split into ByteArray-parts. but the last part of the file can be smaller than the buffer, which is why you write too many bytes in the new file. and that's why you do not have the same checksum anymore. the error can be fixed like this:
public static void rechunck(File file1, File file2) {
FileInputStream is = null;
FileOutputStream os = null;
try {
byte[] buf = new byte[1024*64];
is = new FileInputStream(file1);
os = new FileOutputStream(file2);
int length;
while((length = is.read(buf)) > 0) {
os.write(buf, 0, length);
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
if(is != null)
is.close();
if(os != null)
os.close();
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
}
Due to the length variable, the write method knows that until byte x of the byte array, only the file is off, then there are still old bytes in it that no longer belong to the file.

How to return and delete file?

I want to return file (read or load) from method and then remove this file.
public File method() {
File f = loadFile();
f.delete();
return f;
}
But when I delete a file, I delete it from disk and then exists only descriptor to non-existing file on return statement. So what is the most effective way for it.

You can't keep the File handle of deleted file, rather you can keep the data in a byte array temporarily, delete the file and then return the byte array
public byte[] method() {
File f =loadFile();
FileInputStream fis = new FileInputStream(f);
byte[] data = new byte[fis.available()];
fis.read(data);
f.delete();
return data;
}
// Edit Aproach 2
FileInputStream input = new FileInputStream(f);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int bytesRead = input.read(buf);
while (bytesRead != -1) {
baos.write(buf, 0, bytesRead);
bytesRead = input.read(buf);
}
baos.flush();
byte[] bytes = baos.toByteArray();
you can construct the file data from byte array
However, my suggestion is to use IOUtils.toByteArray(InputStream input) from Jakarta commons, why do you want re write when already in plate

Assuming you want to return the file to the browser, this is how I did it :
File pdf = new File("file.pdf");
if (pdf.exists()) {
try {
InputStream inputStream = new FileInputStream(pdf);
httpServletResponse.setContentType("application/pdf");
httpServletResponse.addHeader("content-disposition", "inline;filename=file.pdf");
copy(inputStream, httpServletResponse.getOutputStream());
inputStream.close();
pdf.delete();
} catch (Exception e) {
e.printStackTrace();
}
}
private static int copy(InputStream input, OutputStream output) throws IOException {
byte[] buffer = new byte[512];
int count = 0;
int n = 0;
while (-1 != (n = input.read(buffer))) {
output.write(buffer, 0, n);
count += n;
}
return count;
}

Joining two mp3 files into one

I have this code to read bytes to another file.
But I'm not able to concatenate two mp3 files into one.
Am I missing something?
public static void main(String[] args) {
String strFileName = ("D:/Music/Assb/Love.mp3");
BufferedOutputStream bos = null;
try
{
//create an object of FileOutputStream
FileOutputStream fos = new FileOutputStream(new File(strFileName));
//create an object of BufferedOutputStream
bos = new BufferedOutputStream(fos);
String str = "D:/Music/Assembled/Heart001.mp3"
+ "D:/Music/Assembled/Heart002.mp3";
/*
* To write byte array to file use,
* public void write(byte[] b) method of BufferedOutputStream
* class.
*/
System.out.println("Writing byte array to file");
bos.write(str.getBytes());
System.out.println("File written");

It`s suck. Mp3 file starts with headers. For correct merging you have to skip first 32 bytes. Try this.
try {
FileInputStream fistream1 = new FileInputStream(_file_name);
File f = new File(new File(_file_name).getParent()+"/final.mp3");
if(!f.exists())
{
f.createNewFile();
}
FileOutputStream sistream = new FileOutputStream((new File(_file_name)).getParent()+"/final.mp3");
int temp;
int size = 0;
temp = fistream1.read();
while( temp != -1)
{
sistream.write(temp);
temp = fistream1.read();
};
fistream1.close();
FileInputStream fistream2 = new FileInputStream(temp_file);
fistream2.read(new byte[32],0,32);
temp = fistream2.read();
while( temp != -1)
{
sistream.write(temp);
temp = fistream2.read();
};
fistream2.close();
sistream.close();
} catch (IOException e) {
e.printStackTrace();
}

You need to do this in two steps
String str = "D:/Music/Assembled/Heart001.mp3";
>>> ADD code to open the file given by str <<<<
bos.write(strFile.getBytes());
>>> Add code to close the file
str = "D:/Music/Assembled/Heart002.mp3";
>>> ADD code to open the file given by str <<<<
bos.write(strFile.getBytes());
>>> Add code to close the file
And as you can see you need code to open the mp3 file to read it

What Are You Trying For...Actually..if You Want To Read 2 Files to Byte Stream the dont String str = "D:/Music/Assembled/Heart001.mp3"
+ "D:/Music/Assembled/Heart002.mp3";
make str1=D:/Music/Assembled/Heart001.mp3 and str2=D:/Music/Assembled/Heart002.mp3 and read str1,str2 seperately through bufferedoutputsream

This code will work well and merge audio of similar type with in seconds...
try {
InputStream in = new FileInputStream("C:\\a.mp3");//firstmp3
byte[] buffer = new byte[1 << 20]; // loads 1 MB of the file
OutputStream os = new FileOutputStream(new File("C:\\output.mp3", true);//output mp3
int count;
while ((count = in.read(buffer)) != -1) {
os.write(buffer, 0, count);
os.flush();
}
in.close();
in = new FileInputStream("C:\\b.mp3");//second mp3
while ((count = in.read(buffer)) != -1) {
os.write(buffer, 0, count);
os.flush();
}
in.close();
os.close();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

Limit size byte[] Java android

I have to fill a byte[] in my Android application. Sometime, this one is bigger than 4KB.
I initialize my byte[] like this :
int size = ReadTools.getPacketSize(ptr.dataInputStream);
byte[] myByteArray = new byte[size];
Here, my size = 22625. But when I fill up my byte[] like this :
while (i != size) {
myByteArray[i] = ptr.dataInputStream.readByte();
i++;
}
But when I print the content of my byte[], I have a byte[] with size = 4060.
Does Java split my byte[] if this one is bigger than 4060 ? And if yes, how can I have a byte[] superior to 4060 ?
Here is my full code:
public class ReadSocket extends Thread{
DataInputStream inputStream;
BufferedReader reader;
GlobalContent ptr;
public ReadSocket(DataInputStream inputStream, GlobalContent ptr)
{
this.inputStream = inputStream;
this.ptr = ptr;
}
public void run() {
int i = 0;
int j = 0;
try {
ptr.StatusThreadReadSocket = 1;
while(ptr.dataInputStream.available() == 0)
{
if(ptr.StatusThreadReadSocket == 0)
{
ptr.dataInputStream.close();
break;
}
}
if(ptr.StatusThreadReadSocket == 1)
{
int end = ReadTools.getPacketSize(ptr.dataInputStream);
byte[] buffer = new byte[end];
while (i != end) {
buffer[j] = ptr.dataInputStream.readByte();
i++;
j++;
}
ptr.StatusThreadReadSocket = 0;
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
...
}

Java doesn't split anything. You should post the minimal code which reproduces your error, and tell where ReadTools comes from.
There are two options here:
ReadTools.getPacketSize() returns 4096
You inadevertedly reassign myByteArray to another array
You should really post your full code and tell what library you use. Likely, it will have a method like
read(byte[] buffer, int offset, int length);
Which will save you some typing and also give better performance if all you need is bulk reading the content of the input in memory

Downloading with BufferInputStream not working properly

The following code doesn't work to download a file (btw clen is file's length):
int pos = 0, total_pos = 0;
byte[] buffer = new byte[BUFFER_SIZE];
while (pos != -1) {
pos = in.read(buffer, 0, BUFFER_SIZE);
total_pos += pos;
out.write(buffer);
setProgress((int) (total_pos * 100 / clen));
}
...but this works fine:
int buf;
while ((buf = in.read()) != -1)
out.write(buf);
I'm wondering why, even though the second code segment works quickly. On that note, is there any particular reason to use a byte[] buffer (since it doesn't seem to be faster, and BufferedInputStream already uses a buffer of its own....?)

Here's how it should be done.
public static void copyStream(InputStream is, OutputStream os)
{
byte[] buff = new byte[4096];
int count;
try {
while((count = is.read(buff)) > 0)
os.write(buff, 0, count);
}catch (Exception e) {
e.printStackTrace();
}finally {
try {
if(is != null)
is.close();
} catch (IOException e) {
e.printStackTrace();
}
try {
if(os != null)
os.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}

I've tried to make the minimum changes necessary to your code to get it working. st0le did a good job of providing a neater version of stream copying.
public class Test {
private static final String FORMAT = "UTF-8";
private static final int BUFFER_SIZE = 10; // for demonstration purposes.
public static void main(String[] args) throws Exception {
String string = "This is a test of the public broadcast system";
int clen = string.length();
ByteArrayInputStream in = new ByteArrayInputStream(string.getBytes(FORMAT));
OutputStream out = System.out;
int pos = 0, total_pos = 0;
byte[] buffer = new byte[BUFFER_SIZE];
while (pos != -1) {
pos = in.read(buffer, 0, BUFFER_SIZE);
if (pos > 0) {
total_pos += pos;
out.write(buffer, 0, pos);
setProgress((int) (total_pos * 100 / clen));
}
}
}
private static void setProgress(int i) {
}
}
You were ignoring the value of pos when you were writing out the buffer to the output stream.
You also need to re-check the value of pos because it may have just read the end of the file. You don't increment the total_pos in that case (although you should probably report that you are 100% complete)
Be sure to handle your resources correctly with close()s in the appropriate places.
-edit-
The general reason for using an array as a buffer is so that the output stream can do as much work as it can with a larger set of data.
Writing to a console there might not be much of a delay, but it might be a network socket being written to or some other slow device. As the JavaDoc states
The write method of OutputStream calls the write method of one argument on each of the bytes to be written out. Subclasses are encouraged to override this method and provide a more efficient implementation.
The benefit of using it when using a Buffered Input/Output Stream are probably minimal.