Develop Reference

Stream of short[]

Hi I need to calculate the entropy of order m of a file where m is the number of bit (m <= 16).
So:
H_m(X)=-sum_i=0 to i=2^m-1{(p_i,m)(log_2 (p_i,m))}
So, I thought to create an input stream to read the file and then calculate the probability of each sequence composed by m bit.
For m = 8 it's easy because I consider a byte.
Since that m<=16 I tought to consider as primitive type short, save each short of the file in an array short[] and then manipulate bits using bitwise operators to obtain all the sequences of m bit in the file.
Is this a good idea?
Anyway, I'm not able to create a stream of short. This is what I've done:
public static void main(String[] args) {
readFile(FILE_NAME_INPUT);
}
public static void readFile(String filename) {
short[] buffer = null;
File a_file = new File(filename);
try {
File file = new File(filename);
FileInputStream fis = new FileInputStream(filename);
DataInputStream dis = new DataInputStream(fis);
int length = (int)file.length() / 2;
buffer = new short[length];
int count = 0;
while(dis.available() > 0 && count < length) {
buffer[count] = dis.readShort();
count++;
}
System.out.println("length=" + length);
System.out.println("count=" + count);
for(int i = 0; i < buffer.length; i++) {
System.out.println("buffer[" + i + "]: " + buffer[i]);
}
fis.close();
}
catch(EOFException eof) {
System.out.println("EOFException: " + eof);
}
catch(FileNotFoundException fe) {
System.out.println("FileNotFoundException: " + fe);
}
catch(IOException ioe) {
System.out.println("IOException: " + ioe);
}
}
But I lose a byte and I don't think this is the best way to proced.
This is what I think to do using bitwise operator:
int[] list = new int[l];
foreach n in buffer {
for(int i = 16 - m; i > 0; i-m) {
list.add( (n >> i) & 2^m-1 );
}
}
I'm assuming in this case to use shorts.
If I use bytes, how can I do a cycle like that for m > 8?
That cycle doesn't work because I have to concatenate multiple bytes and each time varying the number of bits to be joined..
Any ideas?
Thanks

I think you just need to have a byte array:
public static void readFile(String filename) {
ByteArrayOutputStream outputStream=new ByteArrayOutputStream();
try {
FileInputStream fis = new FileInputStream(filename);
byte b=0;
while((b=fis.read())!=-1) {
outputStream.write(b);
}
byte[] byteData=outputStream.toByteArray();
fis.close();
}
catch(IOException ioe) {
System.out.println("IOException: " + ioe);
}
Then you can manipulate byteData as per your bitwise operations.
--
If you want to work with shorts you can combine bytes read this way
short[] buffer=new short[(int)(byteData.length/2.)+1];
j=0;
for(i=0; i<byteData.length-1; i+=2) {
buffer[j]=(short)((byteData[i]<<8)|byteData[i+1]);
j++;
}
To check for odd bytes do this
if((byteData.length%2)==1) last=(short)((0x00<<8)|byteData[byteData.length-1]]);
last is a short so it could be placed in buffer[buffer.length-1]; I'm not sure if that last position in buffer is available or occupied; I think it is but you need to check j after exiting the loop; if j's value is buffer.length-1 then it is available; otherwise might be some problem.
Then manipulate buffer.
The second approach with working with bytes is more involved. It's a question of its own. So try this above.

How do I read an NSInputStream while writing it to an NSOutputStream in iOS?

I am porting an Android app to iPhone (more like improving the iPhone app based on the Android version) and I need to split and combine large uncompressed audio files.
Currently, I load all the files into memory and split them and combine them in separate functions. It crashes with 100MB+ files.
This is the new process needed to do it:
I have two recordings (file1 and file2) and a split position where I want file2 to be inserted inside file1.
-create the input streams for file1 and file2 and the output stream for the outputfile.
-rewrite the new CAF header
-read the data from inputStream1 until it reaches the split point and I write all that data to the output file.
and write it to the output stream.
-read all data from inputStream2 and write it to output file.
-read remaining data from inputStream1 and write it to output file.
Here is my Android code for the process:
File file1File = new File(file1);
File file2File = new File(file2);
long file1Length = file1File.length();
long file2Length = file2File.length();
FileInputStream file1ByteStream = new FileInputStream(file1);
FileInputStream file2ByteStream = new FileInputStream(file2);
FileOutputStream outputFileByteStream = new FileOutputStream(outputFile);
// time = fileLength / (Sample Rate * Channels * Bits per sample / 8)
// convert position to number of bytes for this function
long sampleRate = eRecorder.RECORDER_SAMPLERATE;
int channels = 1;
long bitsPerSample = eRecorder.RECORDER_BPP;
long bytePositionLength = (position * (sampleRate * channels * bitsPerSample / 8)) / 1000;
//calculate total data size
int dataSize = 0;
dataSize = (int)(file1Length + file2Length);
WriteWaveFileHeaderForMerge(outputFileByteStream, dataSize,
dataSize + 36,
eRecorder.RECORDER_SAMPLERATE, 1,
2 * eRecorder.RECORDER_SAMPLERATE);
long bytesWritten = 0;
int length = 0;
//set limit for bytes read, and write file1 bytes to outputfile until split position reached
int limit = (int)bytePositionLength;
//read bytes to limit
writeBytesToLimit(file1ByteStream, outputFileByteStream, limit);
file1ByteStream.close();
file2ByteStream.skip(44);//skip wav file header
writeBytesToLimit(file2ByteStream, outputFileByteStream, (int)file2Length);
file2ByteStream.close();
//calculate length of remaining file1 bytes to be written
long file1offset = bytePositionLength;
//reinitialize file1 input stream
file1ByteStream = new FileInputStream(file1);
file1ByteStream.skip(file1offset);
writeBytesToLimit(file1ByteStream, outputFileByteStream, (int)file1Length);
file1ByteStream.close();
outputFileByteStream.close();
And this is my writeBytesToLimit function:
private void writeBytesToLimit(FileInputStream inputStream, FileOutputStream outputStream, int byteLimit) throws IOException
{
int bytesRead = 0;
int chunkSize = 65536;
int length = 0;
byte[] buffer = new byte[chunkSize];
while((length = inputStream.read(buffer)) != -1)
{
bytesRead += length;
if(bytesRead >= byteLimit)
{
int leftoverBytes = byteLimit % chunkSize;
byte[] smallBuffer = new byte[leftoverBytes];
System.arraycopy(buffer, 0, smallBuffer, 0, leftoverBytes);
outputStream.write(smallBuffer);
break;
}
if(length == chunkSize)
outputStream.write(buffer);
else
{
byte[] smallBuffer = new byte[length];
System.arraycopy(buffer, 0, smallBuffer, 0, length);
outputStream.write(smallBuffer);
}
}
}
How do I do this in iOS? Using the same delegate for two NSInputStreams and an NSOutputStream looks like it will get very messy.
Has anyone seen an example of how to do this (and do it clean)?

I ended up using NSFileHandle. For example, this is the first part of what I am doing.
NSData *readData = [[NSData alloc] init];
NSFileHandle *reader1 = [NSFileHandle fileHandleForReadingAtPath:file1Path];
NSFileHandle *writer = [NSFileHandle fileHandleForWritingAtPath:outputFilePath];
//start reading data from file1 to split point and writing it to file
long bytesRead = 0;
while(bytesRead < splitPointInBytes)
{
//read a chunk of data
readData = [reader1 readDataOfLength:chunkSize];
if(readData.length == 0)break;
//trim data if too much was read
if(bytesRead + readData.length > splitPointInBytes)
{
//get difference of read bytes and byte limit
long difference = bytesRead + readData.length - splitPointInBytes;
//trim data
NSMutableData *readDataMutable = [NSMutableData dataWithData:readData];
[readDataMutable setLength:readDataMutable.length - difference];
readData = [NSData dataWithData:readDataMutable];
NSLog(#"Too much data read, trimming");
}
//write data to output file
[writer writeData:readData];
//update byte counter
bytesRead += readData.length;
}
long file1BytesWritten = bytesRead;

(ask) XOR calculation issue on file in java

I would like to execute XOR operation in my code. However I have strange behavior on the output. Sometimes the result is right but sometime it's not.
Here's the situation:
I have file which I already split into two parts and then I created one parity file using xor operation on both file (source files). So now I have three files. Then I deleted one of the source file. I would like to retrieve the missing file within xor operation between parity file and the remaining source file regarding the missing file. I am using hash function to check whether the output is correct or not. If the function is called only one time, everything is fine, but whenever I have many operations to retrieve the missing file on other files, sometimes my function generates the wrong result.
When they generate the wrong results, it's always generating the same file. BUT if I put thread.sleep for 1 second, they always generate the correct result even if I have more than 1000 operations.
Could somebody help me to spot which part of my code is broke?
private byte[] parityByte(byte[] firstByte, byte[] secondByte) {
int size1;
size1 = firstByte.length;
int size2;
size2 = secondByte.length;
byte[] parity;
parity = new byte[size1];
for (int i = 0; i < size2; i++) {
parity[i] = (byte) (firstByte[i] ^ secondByte[i]);
}
for (int i = size2; i < size1; i++) {
parity[i] = firstByte[i];
}
return parity;
}
/**
* get original chunks
*/
public Chunk getOriginal(Chunk parity, Chunk compare, String orig) throws FileNotFoundException, IOException {
File par = new File(parity.getHash());
InputStream parity = new BufferedInputStream(new FileInputStream(parity.getHash()));
InputStream source = new BufferedInputStream(new FileInputStream(compare.getHash()));
int size = (int) par.length();
int bufferSize = size;
byte[] firstBuffer = new byte[size];
byte[] secondBuffer = new byte[size];
long remainSize;
byte[] destByte = new byte[1];
parity.read(destByte, 0, 1);
Integer dest = new Integer(destByte[0]);
remainSize = size - 1 - dest;
OutputStream originalChunk;
originalChunk = new FileOutputStream(orig);
while (remainSize > 0) {
if (remainSize > bufferSize) {
remainSize -= bufferSize;
} else {
bufferSize = (int) remainSize;
firstBuffer = new byte[bufferSize];
secondBuffer = new byte[bufferSize];
remainSize = 0;
}
parity.read(firstBuffer, 0, bufferSize);
source.read(secondBuffer, 0, bufferSize);
originalChunk.write(parityByte(firstBuffer, secondBuffer));
}
originalChunk.flush();
parity.close();
source.close();
originalChunk.close();
Chunk tempChunk = Chunk.newChunk(orig);
return tempChunk;
}
Thank you
sorry for my bad english.

You are assuming that all the reads fill the buffer. Check the Javadoc. The read(byte[] ...) method returns a value, and it is for a reason.
Have a look at DataInputStream.readFully() for a simple solution.

Parsing a string and converting embedded values to host-byte-order in Java

I have a java client and a C server. I server wants to send a data packet to the client containing some information in a specific order as shown below:
char *buf = NULL;
if(!(buf = malloc(sizeof(char) * pkt_len)))
{
printf("Could not malloc\n");
return -1;
}
memcpy(buf, &pkt_type, 2);
memcpy(buf + 2, &pkt_len, 4);
memcpy(buf + 6, &more_to_come, 1);
memcpy(buf + 7, &fb_id, 8);
memcpy(buf + 15, &match, 4);
memcpy(buf + 19, el->name, name_len);
memcpy(buf + 19 + name_len, "\n\r", 2);
if(send(clientSock, buf, pkt_len, 0) < 0)
{
printf("Can not write to socket %d\n", clientSock);
return -1;
}
Ofcourse I have convereted all the shorts, integers and long integers to network bytes order before writing them to the buffer. The data is received as a string by the Java client. My problem is how to parse this string. For example, I would to know a way to read off the 2 bytes that indicate the pkt length and cast it to a short in host-byte-order. I am aware that Java provides a method to convert a string to an array of bytes. But what do I do after I have obtained the bytes array. Some code to perform this task would be appreciated

You mean something like this?:
char[] charArray = new char[2];
charArray[0] = "a".charAt(0);
charArray[1] = "b".charAt(0);
String string = new String(charArray);

I assume a char is one byte in length here.
You could use a DataInputStream. Depending on your data types, something like the following might get you started. Note the example uses ASCII as the character encoding and doesn't try to be efficient in any way.
package grimbo.test.bytes;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.IOException;
public class BytesTest {
public static void main(String[] args) throws IOException {
// start setup test data
byte[] msgStart = {
/*pkt_type*/0, 1,
/*pkt_len*/0, 0, 0, 1,
/*more_to_come*/1,
/*fb_id*/1, 2, 3, 4, 5, 6, 7, 8,
/*match*/2, 2, 2, 2 };
String name = "John Smith\n\r";
byte[] nameBytes = name.getBytes("ASCII");
byte[] msg = new byte[msgStart.length + nameBytes.length];
System.arraycopy(msgStart, 0, msg, 0, msgStart.length);
System.arraycopy(nameBytes, 0, msg, msgStart.length, nameBytes.length);
// end setup test data
DataInputStream in = new DataInputStream(new ByteArrayInputStream(msg));
new BytesTest().read(in);
}
void read(DataInputStream in) throws IOException {
// assuming pkt_type is an unsigned 2-byte value
int pkt_type = in.readUnsignedShort();
print(pkt_type);
// assuming pkt_len is an unsigned 4-byte value
// Java doesn't have those, so read a signed int and mask to a long
long pkt_len = in.readInt() & 0xFFFFFFFFL;
print(pkt_len);
// assuming vanilla byte is ok for this, but Java bytes are signed, not unsigned
byte more_to_come = in.readByte();
print(more_to_come);
// don't know the format of this, so left as bytes
byte[] fb_id = new byte[8];
in.readFully(fb_id);
print(fb_id);
// don't know the format of this, so left as bytes
byte[] match = new byte[4];
in.readFully(match);
print(match);
char[] nr = { '\n', '\r' };
byte[] name = readUntil(in, nr);
print(name);
System.out.println(">" + new String(name, "ASCII") + "<");
}
private byte[] readUntil(DataInputStream in, /* stop reading when these chars are found */char[] terminate)
throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int terminateIdx = 0;
int input = -1;
while ((input = in.read()) > -1) {
if (input == terminate[terminateIdx]) {
if (terminateIdx == (terminate.length - 1)) {
// we've found the termination sequence
byte[] buf = baos.toByteArray();
// - terminateIdx because we don't include the termination sequence
byte[] result = new byte[buf.length - terminateIdx];
System.arraycopy(buf, 0, result, 0, result.length);
return result;
}
terminateIdx++;
} else {
// no match, reset count
terminateIdx = 0;
}
baos.write(input);
}
return baos.toByteArray();
}
private void print(long l) {
System.out.println(l);
}
void print(byte[] bytes) {
for (int i = 0; i < bytes.length; i++) {
if (i > 0) {
System.out.print(",");
}
System.out.print(bytes[i]);
}
System.out.println();
}
}
And the output is:
1
1
1
1,2,3,4,5,6,7,8
2,2,2,2
74,111,104,110,32,83,109,105,116,104
>John Smith<

How can I write a sequence of strings and then a byte array to a file?

I want to write first a sequence of strings and then a sequence of bytes into a file, using Java. I started by using FileOutputStream because of the array of bytes. After searching the API, I realised that FileOutputStream cannot write Strings, only ints and bytes, so I switched to DataOutputStream. When I run the program, I get an exception. Why?
Here's a portion of my code:
try {
// Create the file
FileOutputStream fos;
DataOutputStream dos; // = new DataOutputStream("compressedfile.ecs_h");
File file= new File("C:\\MyFile.txt");
fos = new FileOutputStream(file);
dos=new DataOutputStream(fos);
/* saves the characters as a dictionary into the file before the binary seq*/
for (int i = 0; i < al.size(); i++) {
String name= al.get(i).name; //gets the string from a global arraylist, don't pay attention to this!
dos.writeChars(name); //saving the name in the file
}
System.out.println("\nIS SUCCESFULLY WRITTEN INTO FILE! ");
dos.writeChars("><");
String strseq;
/*write all elements from the arraylist into a string variable*/
strseq= seq.toString();
System.out.println("sTringSeq: " + strseq);
/*transpose the sequence string into a byte array*/
byte[] data = new byte[strseq.length() / 8];
for (int i = 0; i < data.length; i++) {
data[i] = (byte) Integer.parseInt(strseq.substring(i * 8, (i + 1) * 8), 2);
dos.write(data[i]);
}
dos.flush();
//Close the output stream
dos.close();
} catch(Exception e){}

The problem with your code is that the last for loop was counting over the wrong number of bytes. The code below fixes your problem writing your test data to a file. This works on my machine.
public static void main(String[] args) {
ArrayList<String> al = new ArrayList<String>();
al.add("String1");
al.add("String2");
try {
// Create the file
FileOutputStream fos = new FileOutputStream("MyFile.txt");
DataOutputStream dos = new DataOutputStream(fos);
/* saves the characters as a dictionary into the file before the binary seq */
for (String str : al) {
dos.writeChars(str);
}
System.out.println("\nIS SUCCESFULLY WRITTEN INTO FILE! ");
dos.writeChars("><");
String strseq = "001100111100101000101010111010100100111000000000";
// Ensure that you have a string of the correct size
if (strseq.length() % 8 != 0) {
throw new IllegalStateException(
"Input String is cannot be converted to bytes - wrong size: "
+ strseq.length());
}
int numBytes = strseq.length() / 8;
for (int i = 0; i < numBytes; i++) {
int start = i * 8;
int end = (i + 1) * 8;
byte output = (byte) Integer.parseInt(strseq.substring(start, end), 2);
dos.write(output);
}
dos.writeChars("> Enf of File");
dos.flush();
// Close the output stream
dos.close();
} catch (Exception e) {
e.printStackTrace();
}
}
The approach of writing bytes directly to a test file does have a few problems (I assume that it's a text file in that your test file name ends with .txt), the most obvious one being that some text editors don't handle/display null characters very well (your last test byte was: 00000000 or null). If you want to see the bytes as readable bytes then you could investigate encoding them using Base64 encoding.

Line:
data[i] = (byte) Integer.parseInt(strseq.substring(i * 8, (i + 1) * 8), 2);
looks very suspiciously...
can you provide move details about strseq and its value?

What about this code ?
this code :
byte[] data = new byte[strseq.length() / 8];
for (int i = 0; i < data.length; i++) {
data[i] = (byte) Integer.parseInt(strseq.substring(i * 8, (i + 1) * 8), 2);
dos.write(data[i]);
}
becomes
byte[] data = strseq.getBytes();

With the FileWriter class you have a nice abstraction of a file writing operation.
May this class can help you to write your file...
You can substitute the other OutputStreams by only this class. It have all the methods of you want for write a string and a byte array in a file.