Develop Reference

My DeflaterOutputStream/InputStream code corrupting data

I've got a simple test case that fails to compress a stream of data. I generate a byte[] of some random bytes, compress it via DeflaterOutputStream, flush() the stream, then reverse those operations to retrieve the original array. At byte 505 the reconstructed stream starts to consist entirely of 0x00 bytes, and I don't understand why:
//
// create some random bytes
//
Random rng = new Random();
int len = 5000;
byte[] data = new byte[len];
for (int i = 0; i < len; ++i)
data[i] = (byte) rng.nextInt(0xff);
//
// write to byte[] via a deflater stream
//
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DeflaterOutputStream os = new DeflaterOutputStream(baos, true);
os.write(data);
os.flush();
//
// read back into byte[] via an inflater stream
//
ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
InflaterInputStream is = new InflaterInputStream(bais);
byte[] readbytes = new byte[len];
is.read(readbytes);
//
// check they match (they don't, at byte 505)
//
for (int i = 0; i < len; ++i)
if (data[i] != readbytes[i])
throw new RuntimeException("Mismatch at position " + i);
It doesn't seem to matter what's in the source array, it's always at position 505 it fails.
Here's what the two byte[] arrays look like around the region they differ:
?\m·g··gWNLErZ···,··-··=·;n=··F?···13·{·rw·······\`3···f····{/····t·1·WK$·······WZ······x
?\m·g··gWNLErZ···,··-····································································
^byte 505
All those unprintable chars are 0x00 from that point on. Why is this happening? I feel like I must be misunderstanding something fundamental about how the Deflate/Inflate streams work. The real-world use case here is a stream over a network that I thought I could easily improve the performance of by inserting Deflate/Inflate streams into

When I test this, is.read(readBytes) returns 505, the length of bytes read. The other single-argument-array stream methods return void and guarantee that the entire array is read or written, but is.read() is a different API and requires that you check the amount of bytes actually read.
ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
System.err.println( "bais size = " + bais.available() );
InflaterInputStream is = new InflaterInputStream(bais);
byte[] readbytes = new byte[len];
System.err.println( "read = " + is.read(readbytes) ); // 505
This runs without throwing an error for me:
ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
System.err.println( "bais size = " + bais.available() );
InflaterInputStream is = new InflaterInputStream(bais);
byte[] readbytes = new byte[len];
for( int total = 0, result = 0; (result = is.read(readbytes, total, len-total )) != -1; )
{
total += result;
System.err.println( "reading : " + total );
if( total == len ) break;
}

partial range request with last byte failure

In trying to fulfill a partial range request using chrome as the video playback tool, the video playback start playing back but on reaching halfway, it freezes as if the client is still waiting on more data. At this point the server has already sent the entire request. Please observe the following requests and response and the code used to send the range requested:
Range:bytes=0-
Accept-Ranges:bytes
Content-Length:546827
Content-Range:bytes 0-546827/546828
Content-Type:video/webm
Accept-Ranges:bytes
Content-Length:6155
Content-Range:bytes 540672-546827/546828
Content-Type:video/webm
Accept-Ranges:bytes
Content-Length:1
Content-Range:bytes 546827-546827/546828
Content-Type:video/webm
Is the second response handled correctly? Cause it freezes on this request and starts making multiple requests again until the request times out.
Code handling the request:
private static void copyStream (PullBufferStream inputStream, OutputStream outputStream, BytesStreamObject bytesWrittenObj, int from, int to) throws IOException, CodecNotAvailableException {
//export media stream
BufferedOutputStream bout = new BufferedOutputStream (outputStream);
Buffer b = new Buffer();
long skipped =0;
int byteLength = 0;
do {
inputStream.read(b);
int len = b.getLength();
byte[] data = (byte[]) b.getData();
int offset = b.getOffset();
if (len > 0) {
if(skipped < from) {
// skip bytes until reach from position in inputstream
if(skipped + len <= from) {
// skip all bytes just read
skipped += len;
} else {
// skip only some of bytes read
offset += (from - skipped);
len -= (from - skipped);
skipped = from;
}
}
if(to >= 0) {
if(from + byteLength + len <= to) {
// use the full buffer
} else {
// trim len to needed bytes to be read to prevent exceeding the "to" input parameter
len = (to + 1) - (from + byteLength);
}
}
byteLength+= len;
bytesWrittenObj.bytesWritten = byteLength;
bout.write(data, offset, len);
//bout.write(data, from, end);
}
} while (!b.isEOM());
//fileProperties.setBytesLength(byteLength);
//bout.close();
}

I needed to ensure I flush the outputstream but also made some changes on the actual file size rather than using stream.available().

Mixing wave files

I can not mix two audio extension files wav. My work:
byte[] bufData1 = null;
byte[] bufData2 = null;
ArrayList<Byte> bufData3 = new ArrayList<Byte>();
Creating two arrays with raw audio data
public void bootloadInputData(String p1, String p2) throws IOException {
bufData1 = bootloadReadFileByte(p1);
bufData2 = bootloadReadFileByte(p2);
System.arraycopy(bufData1, 44, bufData1, 0, (bufData1.length - 44));
System.arraycopy(bufData2, 44, bufData2, 0, (bufData2.length - 44));
}
public byte[] bootloadReadFileByte(String path) throws IOException{
ByteArrayOutputStream out = null;
InputStream input = null;
try{
out = new ByteArrayOutputStream();
input = new BufferedInputStream(new FileInputStream(path));
int data = 0;
while((data = input.read()) != -1){
out.write(data);
}
}
finally{
if(null != input){
input.close();
}
if(null != out){
out.close();
}
}
return out.toByteArray();
}
Mixing the bytes of raw audio data
public void bootloadOutputData() throws IOException {
for(int i = 0; i < ((bufData1.length + bufData2.length) / 4); i += 4) {
if(i < bufData1.length){
bufData3.add(bufData1[i]);
bufData3.add(bufData1[i+1]);
bufData3.add(bufData1[i+2]);
bufData3.add(bufData1[i+3]);
}
if(i < bufData2.length){
bufData3.add(bufData2[i]);
bufData3.add(bufData2[i+1]);
bufData3.add(bufData2[i+2]);
bufData3.add(bufData2[i+3]);
}
}
}
Create a new file, fill in the header and raw audio data.
private void bootloadCreateWaveMix(String p1, String p2, String p3) throws IOException {
int size1 = 0;
int size2 = 0;
FileInputStream fis1 = null;
FileInputStream fis2 = null;
try {
fis1 = new FileInputStream(p1);
fis2 = new FileInputStream(p2);
size1 = fis1.available();
size2 = fis2.available();
} finally {
if(fis1 != null){
fis1.close();
}
if(fis2 != null){
fis2.close();
}
}
int mNumBytes = (size1 + size2);
DataOutputStream out = null;
try {
out = new DataOutputStream(new FileOutputStream(p3));
writeId(out, "RIFF");
writeInt(out, 36 + mNumBytes);
writeId(out, "WAVE");
writeId(out, "fmt ");
writeInt(out, 16);
writeShort(out, (short) 1);
writeShort(out, (short) 4);
writeInt(out, (int) 44100);
writeInt(out, 2 * 44100 * 16 / 8);
writeShort(out, (short)(2 * 16 / 8));
writeShort(out, (short) 16);
writeId(out, "data");
writeInt(out, mNumBytes);
out.write(toByteArray(bufData3));
} finally {
if(out != null){
out.close();
}
}
}
private static void writeId(OutputStream out, String id) throws IOException {
for (int i = 0; i < id.length(); i++) out.write(id.charAt(i));
}
private static void writeInt(OutputStream out, int val) throws IOException {
out.write(val >> 0);
out.write(val >> 8);
out.write(val >> 16);
out.write(val >> 24);
}
private static void writeShort(OutputStream out, short val) throws IOException {
out.write(val >> 0);
out.write(val >> 8);
}
public static byte[] toByteArray(ArrayList<Byte> in) {
byte[] data = new byte[in.size()];
for (int i = 0; i < data.length; i++) {
data[i] = (byte) in.get(i);
}
return data;
}
Question:
This code does not correctly create a file that the computer can not
play, but the device can. Reproduction is bad, there is some kind of
interference at the end of the merged files. Also, playback ends when
the first file ends, even if the second file is larger than the first
one. Another problem with the channels on the idea is two stereo
files, and in the title I indicate 4 life even though 2. The files
will always be 44100/16 bit / stereo

If I understand correctly, you want to do the following:
Given 2 input WAV files, mix them together to a single WAV file.
The contents of the output will be the input files played at the same time, not one after the other.
The length of the new file will be the length of the longest of the input files.
All files, input and output, are 16 bit, stereo 44100Hz.
If that's the case, here are (some of) your mistakes:
You need to parse the incoming files so that you don't read their headers as audio data (Do not skip this step just because you already know the format of the audio. You need to read the headers to confirm the data format and accurately determine the number of samples in your input. Also, note that 2/16/44100 WAV files can have different size headers because they can contain various chunks, so you can't just skip over X bytes and then read the file -- you must parse the header!).
If the WAV files are all 16-bit, you need to convert the incoming data from bytes to shorts (note, this is not a simple typecasting -- you must pack 2 bytes into each short. I believe you can use a DataInputStream for this, but be sure to take endianness into account -- WAV files are little-endian and Java is big-endian). Once you've got the shorts representing your samples, average the shorts from the separate files to do the mixing. Your averaged values must then be converted back to bytes (DataOutputStream) to save the resulting file. When you've run out of data from one file, substitute zero.
Your calculation of numBytes is incorrect -- it is not the sum of raw bytes in both files, but a somewhat more complex calculation. In your case, you want it to be equal to something like this:
n1 = number of samples in file 1
n2 = number of samples in file 2
n = MAX( n1 + n2 )
numBytes = n * (number of channels) * (number of bytes per channel) = n * 2 * 2
I strongly urge you to consider using a library like JMF to tackle 1 & 2.

Android: how to have random access from an InputStream?

I have an InputStream, and the relative file name and size.
I need to access/read some random (increasing) positions in the InputStream. This positions are stored in an integer array (named offsets).
InputStream inputStream = ...
String fileName = ...
int fileSize = (int) ...
int[] offsets = new int[]{...}; // the random (increasing) offsets array
Now, given an InputStream, I've found only two possible solutions to jump to random (increasing) positions of the file.
The first one is to use the skip() method of the InputStream (note that I actually use BufferedInputStream, since I will need to mark() and reset() the file pointer).
//Open a BufferInputStream:
BufferedInputStream bufferedInputStream = new BufferedInputStream(inputStream);
byte[] bytes = new byte[1];
int curFilePointer = 0;
long numBytesSkipped = 0;
long numBytesToSkip = 0;
int numBytesRead = 0;
//Check the file size:
if ( fileSize < offsets[offsets.length-1] ) { // the last (bigger) offset is bigger then the file size...
//Debug:
Log.d(TAG, "The file is too small!\n");
return;
}
for (int i=0, k=0; i < offsets.length; i++, k=0) { // for each offset I have to jump...
try {
//Jump to the offset [i]:
while( (curFilePointer < offsets[i]) && (k < 10) ) { // until the correct offset is reached (at most 10 tries)
numBytesToSkip = offsets[i] - curFilePointer;
numBytesSkipped = bufferedInputStream.skip(numBytesToSkip);
curFilePointer += numBytesSkipped; // move the file pointer forward
//Debug:
Log.d(TAG, "FP: " + curFilePointer + "\n");
k++;
}
if ( curFilePointer != offsets[i] ) { // it did NOT jump properly... (what's going on?!)
//Debug:
Log.d(TAG, "InputStream.skip() DID NOT JUMP PROPERLY!!!\n");
break;
}
//Read the content of the file at the offset [i]:
numBytesRead = bufferedInputStream.read(bytes, 0, bytes.length);
curFilePointer += numBytesRead; // move the file pointer forward
//Debug:
Log.d(TAG, "READ [" + curFilePointer + "]: " + bytes[0] + "\n");
}
catch ( IOException e ) {
e.printStackTrace();
break;
}
catch ( IndexOutOfBoundsException e ) {
e.printStackTrace();
break;
}
}
//Close the BufferInputStream:
bufferedInputStream.close()
The problem is that, during my tests, for some (usually big) offsets, it has cycled 5 or more times before skipping the correct number of bytes. Is it normal? And, above all, can/should I thrust skip()? (That is: Are 10 cycles enough to be SURE it will ALWAYS arrive to the correct offset?)
The only alternative way I've found is the one of creating a RandomAccessFile from the InputStream, through File.createTempFile(prefix, suffix, directory) and the following function.
public static RandomAccessFile toRandomAccessFile(InputStream inputStream, File tempFile, int fileSize) throws IOException {
RandomAccessFile randomAccessFile = new RandomAccessFile(tempFile, "rw");
byte[] buffer = new byte[fileSize];
int numBytesRead = 0;
while ( (numBytesRead = inputStream.read(buffer)) != -1 ) {
randomAccessFile.write(buffer, 0, numBytesRead);
}
randomAccessFile.seek(0);
return randomAccessFile;
}
Having a RandomAccessFile is actually a much better solution, but the performance are exponentially worse (above all because I will have more than a single file).
EDIT: Using byte[] buffer = new byte[fileSize] speeds up (and a lot) the RandomAccessFile creation!
//Create a temporary RandomAccessFile:
File tempFile = File.createTempFile(fileName, null, context.getCacheDir());
RandomAccessFile randomAccessFile = toRandomAccessFile(inputStream, tempFile, fileSize);
byte[] bytes = new byte[1];
int numBytesRead = 0;
//Check the file size:
if ( fileSize < offsets[offsets.length-1] ) { // the last (bigger) offset is bigger then the file size...
//Debug:
Log.d(TAG, "The file is too small!\n");
return;
}
for (int i=0, k=0; i < offsets.length; i++, k=0) { // for each offset I have to jump...
try {
//Jump to the offset [i]:
randomAccessFile.seek(offsets[i]);
//Read the content of the file at the offset [i]:
numBytesRead = randomAccessFile.read(bytes, 0, bytes.length);
//Debug:
Log.d(TAG, "READ [" + (randomAccessFile.getFilePointer()-4) + "]: " + bytes[0] + "\n");
}
catch ( IOException e ) {
e.printStackTrace();
break;
}
catch ( IndexOutOfBoundsException e ) {
e.printStackTrace();
break;
}
}
//Delete the temporary RandomAccessFile:
randomAccessFile.close();
tempFile.delete();
Now, is there a better (or more elegant) solution to have a "random" access from an InputStream?

It's a bit unfortunate you have an InputStream to begin with, but in this situation buffering the stream in a file is of no use iff you are always skipping forward. But you don't have to count the number of times you have called skip, that's not really of interest.
What you do have to check if the stream has ended already, to prevent an infinite loop. Checking the source of the default skip implementation, I'd say you'll have to keep calling skip until it returns 0. This will indicate the end of stream has been reached. The JavaDoc was a bit unclear about this for my taste.

You can't. An InputStream is a stream, that is to say a sequential construct. Your question embodies a contradiction in terms.

(ask) XOR calculation issue on file in java

I would like to execute XOR operation in my code. However I have strange behavior on the output. Sometimes the result is right but sometime it's not.
Here's the situation:
I have file which I already split into two parts and then I created one parity file using xor operation on both file (source files). So now I have three files. Then I deleted one of the source file. I would like to retrieve the missing file within xor operation between parity file and the remaining source file regarding the missing file. I am using hash function to check whether the output is correct or not. If the function is called only one time, everything is fine, but whenever I have many operations to retrieve the missing file on other files, sometimes my function generates the wrong result.
When they generate the wrong results, it's always generating the same file. BUT if I put thread.sleep for 1 second, they always generate the correct result even if I have more than 1000 operations.
Could somebody help me to spot which part of my code is broke?
private byte[] parityByte(byte[] firstByte, byte[] secondByte) {
int size1;
size1 = firstByte.length;
int size2;
size2 = secondByte.length;
byte[] parity;
parity = new byte[size1];
for (int i = 0; i < size2; i++) {
parity[i] = (byte) (firstByte[i] ^ secondByte[i]);
}
for (int i = size2; i < size1; i++) {
parity[i] = firstByte[i];
}
return parity;
}
/**
* get original chunks
*/
public Chunk getOriginal(Chunk parity, Chunk compare, String orig) throws FileNotFoundException, IOException {
File par = new File(parity.getHash());
InputStream parity = new BufferedInputStream(new FileInputStream(parity.getHash()));
InputStream source = new BufferedInputStream(new FileInputStream(compare.getHash()));
int size = (int) par.length();
int bufferSize = size;
byte[] firstBuffer = new byte[size];
byte[] secondBuffer = new byte[size];
long remainSize;
byte[] destByte = new byte[1];
parity.read(destByte, 0, 1);
Integer dest = new Integer(destByte[0]);
remainSize = size - 1 - dest;
OutputStream originalChunk;
originalChunk = new FileOutputStream(orig);
while (remainSize > 0) {
if (remainSize > bufferSize) {
remainSize -= bufferSize;
} else {
bufferSize = (int) remainSize;
firstBuffer = new byte[bufferSize];
secondBuffer = new byte[bufferSize];
remainSize = 0;
}
parity.read(firstBuffer, 0, bufferSize);
source.read(secondBuffer, 0, bufferSize);
originalChunk.write(parityByte(firstBuffer, secondBuffer));
}
originalChunk.flush();
parity.close();
source.close();
originalChunk.close();
Chunk tempChunk = Chunk.newChunk(orig);
return tempChunk;
}
Thank you
sorry for my bad english.

You are assuming that all the reads fill the buffer. Check the Javadoc. The read(byte[] ...) method returns a value, and it is for a reason.
Have a look at DataInputStream.readFully() for a simple solution.