Java reading file to a byte array - most efficient implementation - java

Im reading image files off an sdcard of an android device. This results in an out of memory fatal error due to a memory leak. I have narrowed it down to an allocation of 1 byte arrays that are not being removed by the GC. I use FileInputStream to read in the file is there a more efficient way to do this? Can you see the cause of the memory leak? Thanks
private String getHexFileString(File _file)
{
byte[] byteStream = new byte[(int) _file.length()];
String fileHexString = null;
try
{
FileInputStream fis = new FileInputStream(_file);
fis.read(byteStream);
fis.close();
fis = null;
fileHexString = byteArrayToHexString(byteStream);
}
catch (FileNotFoundException e1)
{ actLog.addMessage(new ErrorMessage(e1)); }
catch (IOException e2)
{ actLog.addMessage(new ErrorMessage(e2)); }
catch(OutOfMemoryError e3)
{ actLog.addMessage(new ErrorMessage(e3)); }
return fileHexString;
}
/**
* This method formats a byte-array into a hex string
*
* #param b byte-array
* #return hex string
*/
public String byteArrayToHexString(byte[] b)
{
char[] hexVal = new char[b.length * 2];
int value = 0;
for (int i = 0; i < b.length; i++)
{
value = (b[i] + 256) % 256;
hexVal[i * 2 + 0] = kDigits[value >> 4];
hexVal[i * 2 + 1] = kDigits[value & 0x0f];
}
return new String(hexVal);
}

If some exception occure while you read you will not close the file. It's better to use block finally

You are likely to find that converting to a hex String is far more expensive in time and memory than reading the file. i.e. the String will be 4x bigger.
If you want to improve efficiency and avoid running out of memory you should use a method which returns a byte[] and you operate on this array.

Related

Mixing wave files

I can not mix two audio extension files wav. My work:
byte[] bufData1 = null;
byte[] bufData2 = null;
ArrayList<Byte> bufData3 = new ArrayList<Byte>();
Creating two arrays with raw audio data
public void bootloadInputData(String p1, String p2) throws IOException {
bufData1 = bootloadReadFileByte(p1);
bufData2 = bootloadReadFileByte(p2);
System.arraycopy(bufData1, 44, bufData1, 0, (bufData1.length - 44));
System.arraycopy(bufData2, 44, bufData2, 0, (bufData2.length - 44));
}
public byte[] bootloadReadFileByte(String path) throws IOException{
ByteArrayOutputStream out = null;
InputStream input = null;
try{
out = new ByteArrayOutputStream();
input = new BufferedInputStream(new FileInputStream(path));
int data = 0;
while((data = input.read()) != -1){
out.write(data);
}
}
finally{
if(null != input){
input.close();
}
if(null != out){
out.close();
}
}
return out.toByteArray();
}
Mixing the bytes of raw audio data
public void bootloadOutputData() throws IOException {
for(int i = 0; i < ((bufData1.length + bufData2.length) / 4); i += 4) {
if(i < bufData1.length){
bufData3.add(bufData1[i]);
bufData3.add(bufData1[i+1]);
bufData3.add(bufData1[i+2]);
bufData3.add(bufData1[i+3]);
}
if(i < bufData2.length){
bufData3.add(bufData2[i]);
bufData3.add(bufData2[i+1]);
bufData3.add(bufData2[i+2]);
bufData3.add(bufData2[i+3]);
}
}
}
Create a new file, fill in the header and raw audio data.
private void bootloadCreateWaveMix(String p1, String p2, String p3) throws IOException {
int size1 = 0;
int size2 = 0;
FileInputStream fis1 = null;
FileInputStream fis2 = null;
try {
fis1 = new FileInputStream(p1);
fis2 = new FileInputStream(p2);
size1 = fis1.available();
size2 = fis2.available();
} finally {
if(fis1 != null){
fis1.close();
}
if(fis2 != null){
fis2.close();
}
}
int mNumBytes = (size1 + size2);
DataOutputStream out = null;
try {
out = new DataOutputStream(new FileOutputStream(p3));
writeId(out, "RIFF");
writeInt(out, 36 + mNumBytes);
writeId(out, "WAVE");
writeId(out, "fmt ");
writeInt(out, 16);
writeShort(out, (short) 1);
writeShort(out, (short) 4);
writeInt(out, (int) 44100);
writeInt(out, 2 * 44100 * 16 / 8);
writeShort(out, (short)(2 * 16 / 8));
writeShort(out, (short) 16);
writeId(out, "data");
writeInt(out, mNumBytes);
out.write(toByteArray(bufData3));
} finally {
if(out != null){
out.close();
}
}
}
private static void writeId(OutputStream out, String id) throws IOException {
for (int i = 0; i < id.length(); i++) out.write(id.charAt(i));
}
private static void writeInt(OutputStream out, int val) throws IOException {
out.write(val >> 0);
out.write(val >> 8);
out.write(val >> 16);
out.write(val >> 24);
}
private static void writeShort(OutputStream out, short val) throws IOException {
out.write(val >> 0);
out.write(val >> 8);
}
public static byte[] toByteArray(ArrayList<Byte> in) {
byte[] data = new byte[in.size()];
for (int i = 0; i < data.length; i++) {
data[i] = (byte) in.get(i);
}
return data;
}
Question:
This code does not correctly create a file that the computer can not
play, but the device can. Reproduction is bad, there is some kind of
interference at the end of the merged files. Also, playback ends when
the first file ends, even if the second file is larger than the first
one. Another problem with the channels on the idea is two stereo
files, and in the title I indicate 4 life even though 2. The files
will always be 44100/16 bit / stereo
If I understand correctly, you want to do the following:
Given 2 input WAV files, mix them together to a single WAV file.
The contents of the output will be the input files played at the same time, not one after the other.
The length of the new file will be the length of the longest of the input files.
All files, input and output, are 16 bit, stereo 44100Hz.
If that's the case, here are (some of) your mistakes:
You need to parse the incoming files so that you don't read their headers as audio data (Do not skip this step just because you already know the format of the audio. You need to read the headers to confirm the data format and accurately determine the number of samples in your input. Also, note that 2/16/44100 WAV files can have different size headers because they can contain various chunks, so you can't just skip over X bytes and then read the file -- you must parse the header!).
If the WAV files are all 16-bit, you need to convert the incoming data from bytes to shorts (note, this is not a simple typecasting -- you must pack 2 bytes into each short. I believe you can use a DataInputStream for this, but be sure to take endianness into account -- WAV files are little-endian and Java is big-endian). Once you've got the shorts representing your samples, average the shorts from the separate files to do the mixing. Your averaged values must then be converted back to bytes (DataOutputStream) to save the resulting file. When you've run out of data from one file, substitute zero.
Your calculation of numBytes is incorrect -- it is not the sum of raw bytes in both files, but a somewhat more complex calculation. In your case, you want it to be equal to something like this:
n1 = number of samples in file 1
n2 = number of samples in file 2
n = MAX( n1 + n2 )
numBytes = n * (number of channels) * (number of bytes per channel) = n * 2 * 2
I strongly urge you to consider using a library like JMF to tackle 1 & 2.

Stream of short[]

Hi I need to calculate the entropy of order m of a file where m is the number of bit (m <= 16).
So:
H_m(X)=-sum_i=0 to i=2^m-1{(p_i,m)(log_2 (p_i,m))}
So, I thought to create an input stream to read the file and then calculate the probability of each sequence composed by m bit.
For m = 8 it's easy because I consider a byte.
Since that m<=16 I tought to consider as primitive type short, save each short of the file in an array short[] and then manipulate bits using bitwise operators to obtain all the sequences of m bit in the file.
Is this a good idea?
Anyway, I'm not able to create a stream of short. This is what I've done:
public static void main(String[] args) {
readFile(FILE_NAME_INPUT);
}
public static void readFile(String filename) {
short[] buffer = null;
File a_file = new File(filename);
try {
File file = new File(filename);
FileInputStream fis = new FileInputStream(filename);
DataInputStream dis = new DataInputStream(fis);
int length = (int)file.length() / 2;
buffer = new short[length];
int count = 0;
while(dis.available() > 0 && count < length) {
buffer[count] = dis.readShort();
count++;
}
System.out.println("length=" + length);
System.out.println("count=" + count);
for(int i = 0; i < buffer.length; i++) {
System.out.println("buffer[" + i + "]: " + buffer[i]);
}
fis.close();
}
catch(EOFException eof) {
System.out.println("EOFException: " + eof);
}
catch(FileNotFoundException fe) {
System.out.println("FileNotFoundException: " + fe);
}
catch(IOException ioe) {
System.out.println("IOException: " + ioe);
}
}
But I lose a byte and I don't think this is the best way to proced.
This is what I think to do using bitwise operator:
int[] list = new int[l];
foreach n in buffer {
for(int i = 16 - m; i > 0; i-m) {
list.add( (n >> i) & 2^m-1 );
}
}
I'm assuming in this case to use shorts.
If I use bytes, how can I do a cycle like that for m > 8?
That cycle doesn't work because I have to concatenate multiple bytes and each time varying the number of bits to be joined..
Any ideas?
Thanks
I think you just need to have a byte array:
public static void readFile(String filename) {
ByteArrayOutputStream outputStream=new ByteArrayOutputStream();
try {
FileInputStream fis = new FileInputStream(filename);
byte b=0;
while((b=fis.read())!=-1) {
outputStream.write(b);
}
byte[] byteData=outputStream.toByteArray();
fis.close();
}
catch(IOException ioe) {
System.out.println("IOException: " + ioe);
}
Then you can manipulate byteData as per your bitwise operations.
--
If you want to work with shorts you can combine bytes read this way
short[] buffer=new short[(int)(byteData.length/2.)+1];
j=0;
for(i=0; i<byteData.length-1; i+=2) {
buffer[j]=(short)((byteData[i]<<8)|byteData[i+1]);
j++;
}
To check for odd bytes do this
if((byteData.length%2)==1) last=(short)((0x00<<8)|byteData[byteData.length-1]]);
last is a short so it could be placed in buffer[buffer.length-1]; I'm not sure if that last position in buffer is available or occupied; I think it is but you need to check j after exiting the loop; if j's value is buffer.length-1 then it is available; otherwise might be some problem.
Then manipulate buffer.
The second approach with working with bytes is more involved. It's a question of its own. So try this above.

Convert Hex to Byte with big string

I tried the different ways to convert hex to byte, there are four methods in the code, three of them I comment it out, only one there is no error when I run it, but I confused that when I repeated to run the code, it gave me the different result (should be generate "byte").
There is a question is when I use "method1", it gave me the result (byte), but once I changed to "method2", it will not generate the result, I don't know why. I thought it should generate same result, when I have same string.
public class Convert {
/**
* #param args
* #throws IOException
*/
// String everything;
public static void main(String[] args) throws IOException {
//String everything;
// TODO Auto-generated method stub
BufferedReader br = null;
try {
br = new BufferedReader(new FileReader("C:\\TEMP1\\Doctor.txt"));
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
try {
StringBuilder sb = new StringBuilder();
String line = null;
try {
line = br.readLine();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
while (line != null) {
sb.append(line);
sb.append('\n');
line = br.readLine();
}
//*********Method 1****************
//String r="1ee079746828d7c6f9af46f93c1ef2555ff4b14b2378ad53a258d18dc6a8363fb57f3448783833722bd9ef291ba53153afca31a96de404755e78f68b76fd5a77e4be3b984ea25244842e92a8ed40da1f1a588fb3da26b8bc21d74cd8476534f26ee454df086567c4d7cf3334f794cede41a9b051a5c393a35584afcf";
//byte[] b = new BigInteger(r,16).toByteArray();
//System.out.println("Byte for public key: "+b);
//*********Method 2****************
//String r2 = sb.toString();
//System.out.println("Doctor contect file: "+r2);
//byte[] b = new BigInteger(r2,16).toByteArray();
//System.out.println("Byte for public key: "+b);
//********Method 3*****************
String r="1ee079746828d7c6f9af46f93c1ef2555ff4b14b2378ad53a258d18dc6a8363fb57f3448783833722bd9ef291ba53153afca31a96de404755e78f68b76fd5a77e4be3b984ea25244842e92a8ed40da1f1a588fb3da26b8bc21d74cd8476534f26ee454df086567c4d7cf3334f794cede41a9b051a5c393a35584afcf";
int len = r.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(r.charAt(i), 16) << 4)
+ Character.digit(r.charAt(i+1), 16));
System.out.println(data);
}
//********Method4******************
/*
String r2 = sb.toString();
int len = r2.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(r2.charAt(i), 16) << 4)
+ Character.digit(r2.charAt(i+1), 16));
System.out.println(data);
}
*/
//String r=everything;
// String r="1ee079746828d7c6f9af46f93c1ef2555ff4b14b2378ad53a258d18dc6a8363fb57f3448783833722bd9ef291ba53153afca31a96de404755e78f68b76fd5a77e4be3b984ea25244842e92a8ed40da1f1a588fb3da26b8bc21d74cd8476534f26ee454df086567c4d7cf3334f794cede41a9b051a5c393a35584afcf";
// double convert=Double.parseDouble(r);
// long convert=(long)(Integer.parseInt(r,32)&0xFF);
// byte convert=Byte.parseByte(r,32);
// byte convert=Integer.parseInt(everything,16);
// System.out.println("Byte for public key: "+convert);
} finally {
br.close();
}
}
}
You're printing the result of calling toString on a byte[]. That's not going to give you what you want.
For diagnostic purposes, use System.out.println(Arrays.toString(data)). And do that at the end of the loop rather than within it:
for (int i = 0; i < len; i += 2) {
...
}
System.out.println(Arrays.toString(data));
There are plenty of alternative approaches to parsing a hex string, mind you. I don't personally like the idea of using an XML-focused API (as recommended in the question comments) when you're not dealing with XML, but it would certainly work - and any number of third party APIs have hex conversion routines.
EDIT: As noted in comments, I believe your hex conversion code is also broken at the moment - but that should probably be fixed by using a prebuilt one from elsewhere. The main purpose of this answer was to explain why you're getting results such as "[B#40a0dcd9". Once you can see the data, you can verify it.
I'm unsure on why you do that, so this answer may not reflect your intend.
I tried to comprehend your stuff and came to the conclusion that you want to split that string into two character blocks, treat them as a hex number and convert them to a byte.
That wont work, as the second block e0 is larger that Byte.MAX_VALUE. So, heres my latest guess on what could be the code that you are looking for (using Integer).
public static void main(String[] args) {
String r = "1ee079746828d7c6f9af46f93c1ef2555ff4b14b2378ad53a258d18dc6a8363fb57f3448783833722bd9ef291ba53153afca31a96de404755e78f68b76fd5a77e4be3b984ea25244842e92a8ed40da1f1a588fb3da26b8bc21d74cd8476534f26ee454df086567c4d7cf3334f794cede41a9b051a5c393a35584afcf";
char[] rA = r.toCharArray();
int len = r.length();
int[] data = new int[len / 2];
for (int i = 0; i < len; i += 2) {
String base = "#" + rA[i] + rA[i+1];
System.out.println("base: " + base);
data[i / 2] = Integer.decode(base);
System.out.println(data[i/2]);
}
}
After reading your question again, it seems that you describe that when you use the String directly (method1) it works, but if you read if from a file (method2) it does not.
The reason is simple: You add a \n at the end of the String you read from file. You do not do so in your method1.

Getting MD5 Hash of File from URL

The result I'm getting is that files of the same type are returning the same md5 hash value. For example two different jpgs are giving me the same result. However, a jpg vs a apk are giving different results.
Here is my code...
public static String checkHashURL(String input) {
try {
MessageDigest md = MessageDigest.getInstance("MD5");
InputStream is = new URL(input).openStream();
try {
is = new DigestInputStream(is, md);
int b;
while ((b = is.read()) > 0) {
;
}
} finally {
is.close();
}
byte[] digest = md.digest();
StringBuffer sb = new StringBuffer();
for (int i = 0; i < digest.length; i++) {
sb.append(
Integer.toString((digest[i] & 0xff) + 0x100, 16).substring(
1));
}
return sb.toString();
} catch (Exception ex) {
throw new RuntimeException(ex);
}
}
This is broken:
while ((b = is.read()) > 0)
Your code will stop at the first byte of the stream which is 0. If the two files have the same values before the first 0 byte, you'll fail. If you really want to call the byte-at-a-time version of read, you want:
while (is.read() != -1) {}
The parameterless InputStream.read() method returns -1 when it reaches the end of the stream.
(There's no need to assign a value to b, as you're not using it.)
Better would be to read a buffer at a time:
byte[] ignoredBuffer = new byte[8 * 1024]; // Up to 8K per read
while (is.read(ignoredBuffer) > 0) {}
This time the condition is valid, because InputStream.read(byte[]) would only ever return 0 if you pass in an empty buffer. Otherwise, it will try to read at least one byte, returning the length of data read or -1 if the end of the stream has been reached.

How can I write a sequence of strings and then a byte array to a file?

I want to write first a sequence of strings and then a sequence of bytes into a file, using Java. I started by using FileOutputStream because of the array of bytes. After searching the API, I realised that FileOutputStream cannot write Strings, only ints and bytes, so I switched to DataOutputStream. When I run the program, I get an exception. Why?
Here's a portion of my code:
try {
// Create the file
FileOutputStream fos;
DataOutputStream dos; // = new DataOutputStream("compressedfile.ecs_h");
File file= new File("C:\\MyFile.txt");
fos = new FileOutputStream(file);
dos=new DataOutputStream(fos);
/* saves the characters as a dictionary into the file before the binary seq*/
for (int i = 0; i < al.size(); i++) {
String name= al.get(i).name; //gets the string from a global arraylist, don't pay attention to this!
dos.writeChars(name); //saving the name in the file
}
System.out.println("\nIS SUCCESFULLY WRITTEN INTO FILE! ");
dos.writeChars("><");
String strseq;
/*write all elements from the arraylist into a string variable*/
strseq= seq.toString();
System.out.println("sTringSeq: " + strseq);
/*transpose the sequence string into a byte array*/
byte[] data = new byte[strseq.length() / 8];
for (int i = 0; i < data.length; i++) {
data[i] = (byte) Integer.parseInt(strseq.substring(i * 8, (i + 1) * 8), 2);
dos.write(data[i]);
}
dos.flush();
//Close the output stream
dos.close();
} catch(Exception e){}
The problem with your code is that the last for loop was counting over the wrong number of bytes. The code below fixes your problem writing your test data to a file. This works on my machine.
public static void main(String[] args) {
ArrayList<String> al = new ArrayList<String>();
al.add("String1");
al.add("String2");
try {
// Create the file
FileOutputStream fos = new FileOutputStream("MyFile.txt");
DataOutputStream dos = new DataOutputStream(fos);
/* saves the characters as a dictionary into the file before the binary seq */
for (String str : al) {
dos.writeChars(str);
}
System.out.println("\nIS SUCCESFULLY WRITTEN INTO FILE! ");
dos.writeChars("><");
String strseq = "001100111100101000101010111010100100111000000000";
// Ensure that you have a string of the correct size
if (strseq.length() % 8 != 0) {
throw new IllegalStateException(
"Input String is cannot be converted to bytes - wrong size: "
+ strseq.length());
}
int numBytes = strseq.length() / 8;
for (int i = 0; i < numBytes; i++) {
int start = i * 8;
int end = (i + 1) * 8;
byte output = (byte) Integer.parseInt(strseq.substring(start, end), 2);
dos.write(output);
}
dos.writeChars("> Enf of File");
dos.flush();
// Close the output stream
dos.close();
} catch (Exception e) {
e.printStackTrace();
}
}
The approach of writing bytes directly to a test file does have a few problems (I assume that it's a text file in that your test file name ends with .txt), the most obvious one being that some text editors don't handle/display null characters very well (your last test byte was: 00000000 or null). If you want to see the bytes as readable bytes then you could investigate encoding them using Base64 encoding.
Line:
data[i] = (byte) Integer.parseInt(strseq.substring(i * 8, (i + 1) * 8), 2);
looks very suspiciously...
can you provide move details about strseq and its value?
What about this code ?
this code :
byte[] data = new byte[strseq.length() / 8];
for (int i = 0; i < data.length; i++) {
data[i] = (byte) Integer.parseInt(strseq.substring(i * 8, (i + 1) * 8), 2);
dos.write(data[i]);
}
becomes
byte[] data = strseq.getBytes();
With the FileWriter class you have a nice abstraction of a file writing operation.
May this class can help you to write your file...
You can substitute the other OutputStreams by only this class. It have all the methods of you want for write a string and a byte array in a file.

Categories

Resources