Develop Reference

How can I create a stream of bits from a byte array? [duplicate]

How can i iterate bits in a byte array?

You'd have to write your own implementation of Iterable<Boolean> which took an array of bytes, and then created Iterator<Boolean> values which remembered the current index into the byte array and the current index within the current byte. Then a utility method like this would come in handy:
private static Boolean isBitSet(byte b, int bit)
{
return (b & (1 << bit)) != 0;
}
(where bit ranges from 0 to 7). Each time next() was called you'd have to increment your bit index within the current byte, and increment the byte index within byte array if you reached "the 9th bit".
It's not really hard - but a bit of a pain. Let me know if you'd like a sample implementation...

public class ByteArrayBitIterable implements Iterable<Boolean> {
private final byte[] array;
public ByteArrayBitIterable(byte[] array) {
this.array = array;
}
public Iterator<Boolean> iterator() {
return new Iterator<Boolean>() {
private int bitIndex = 0;
private int arrayIndex = 0;
public boolean hasNext() {
return (arrayIndex < array.length) && (bitIndex < 8);
}
public Boolean next() {
Boolean val = (array[arrayIndex] >> (7 - bitIndex) & 1) == 1;
bitIndex++;
if (bitIndex == 8) {
bitIndex = 0;
arrayIndex++;
}
return val;
}
public void remove() {
throw new UnsupportedOperationException();
}
};
}
public static void main(String[] a) {
ByteArrayBitIterable test = new ByteArrayBitIterable(
new byte[]{(byte)0xAA, (byte)0xAA});
for (boolean b : test)
System.out.println(b);
}
}

Original:
for (int i = 0; i < byteArray.Length; i++)
{
byte b = byteArray[i];
byte mask = 0x01;
for (int j = 0; j < 8; j++)
{
bool value = b & mask;
mask << 1;
}
}
Or using Java idioms
for (byte b : byteArray ) {
for ( int mask = 0x01; mask != 0x100; mask <<= 1 ) {
boolean value = ( b & mask ) != 0;
}
}

An alternative would be to use a BitInputStream like the one you can find here and write code like this:
BitInputStream bin = new BitInputStream(new ByteArrayInputStream(bytes));
while(true){
int bit = bin.readBit();
// do something
}
bin.close();
(Note: Code doesn't contain EOFException or IOException handling for brevity.)
But I'd go with Jon Skeets variant and do it on my own.

I needed some bit streaming in my application. Here you can find my BitArray implementation. It is not a real iterator pattern but you can ask for 1-32 bits from the array in a streaming way. There is also an alternate implementation called BitReader later in the file.

I know, probably not the "coolest" way to do it, but you can extract each bit with the following code.
int n = 156;
String bin = Integer.toBinaryString(n);
System.out.println(bin);
char arr[] = bin.toCharArray();
for(int i = 0; i < arr.length; ++i) {
System.out.println("Bit number " + (i + 1) + " = " + arr[i]);
}
10011100
Bit number 1 = 1
Bit number 2 = 0
Bit number 3 = 0
Bit number 4 = 1
Bit number 5 = 1
Bit number 6 = 1
Bit number 7 = 0
Bit number 8 = 0

You can iterate through the byte array, and for each byte use the bitwise operators to iterate though its bits.

Alternatively, you can use BitSet for this:
byte[] bytes=...;
BitSet bitSet=BitSet.valueOf(bytes);
for(int i=0;i<bitSet.length();i++){
boolean bit=bitSet.get(i);
//use your bit
}

Replace all inside bitset with a differently sized bitset

I'm currently dealing with a binary file that will later on be written into a different binary file. This is very important and is the reason I'm hesitant to use ArrayLists and other lists, as they tend to not play nicely with me trying to write it into a file directly.
I've retrieved the bytes out of this binary and separated them into bits using BitSet. I think I've figured out how to find the bitset I want to replace. Currently this looks kinda like this:
try {
InputStream inputStream = new FileInputStream(filepath);
OutputStream outputStream = new FileOutputStream("output.bin");
byte[] buffer = new byte[4096];
BitSet bitSet = new BitSet(4096 * 8);
BitSet bitString = new BitSet(search.length());
BitSet bitReplace = new BitSet(replace.length());
// Search String to bitset
for (int i = search.length() - 1; i >= 0; i--) {
if (search.charAt(i) == '1') {
bitString.set(search.length() - i - 1);
}
}
// Replace String to bitset
for (int i = replace.length() - 1; i >= 0; i--) {
if (replace.charAt(i) == '1') {
bitReplace.set(replace.length() - i - 1);
}
}
while (inputStream.read(buffer) != -1) {
bitSet = BitSet.valueOf(buffer);
bufferCount++;
// GET 4096 BYTES AT THE SAME TIME
// TURN THEM INTO BITS, WE END UP WITH 4096*8 bits
// COMPARE EVERY SEARCHSIZE BITS
for (int i = 0; i < bitSet.length(); i++) {
if (bitSet.get(i, i + bitString.length()).equals(bitString)) {
//TODO: Replace bitset with a different bit set
}
}
}
inputStream.close();
outputStream.close();
} catch (IOException e) {
System.out.println("IOException");
System.exit(1);
}
What I'm missing is how to set an existing bitsets once the pattern of bits have been found with a different bitset(could be differently sized).
So to illustrate:
Find: 01010 replace with: 001111
Would turn this sequence of bits:
00|01010|01000000000000010
into:
00|001111|010000000000000010
Abstractly I've thought of a solution, to be like this:
1. Find the pattern that matches the SEARCHed pattern
2. Replace a bitset with a completely different bitset(this is what I'm struggling with, I was thinking about just appending everything to the end of the file, but that would not be very efficient in terms of read/write
3. Shift the other bits to the left or to the right based on the difference between the sizes of the searched pattern and the pattern we're replacing with.
4. Write into file.

You could define a function setBitsFromIndex(int i, BitSet source, BitSet dest):
private static void setBitsFromIndex(int i, BitSet source, BitSet dest) {
for (int j = 0; j < source.length(); j++) {
dest.set(i+j, source.get(j));
}
}
Then, in your code:
for (int i = 0; i < bitSet.length() - bitString.length(); i++) {
if (bitSet.get(i, i + bitString.length()).equals(bitString)) {
//Replace bitset with a different bit set
BitSet tempBitSet = bitSet.get(i + bitString.length(), bitSet.length());
setBitsFromIndex(i, bitReplace, bitSet);
setBitsFromIndex(i + bitReplace.length(), tempBitSet, bitSet);
// if bitReplace is shorter than bitString, we may need to clear trailing bits
if (bitReplace.length() < bitString.length()) {
bitSet.clear(i + bitReplace.length() + tempBitSet.length(), bitSet.length());
}
break;
}
}
BE WARNED: The length of a BitSet is NOT it's capacity, or even the length it was prior to the last time you set a bit. It is the index + 1 of the HIGHEST SET (1) BIT, so your bitReplace, bitString, and bitSet BitSets might not be the length you think they are if they have 0s in the most-significant bits. If you want to include leading zeros, you have to track the desired size of your bitReplace and bitString BitSets independently.

Generating all possible binary combinations

I am trying to generate all of the possible binary combinations for two bytes ex.
00000000 00000001
00000000 00000010
00000000 00000011
I have the class that I'm working on but its clearly not working at all. I cant get the method to return the output as it is generated.
I got the code below to work properly, but only for calculation 1 byte. How would I change this to calculate all of the possible outcomes for 2 bytes?
package referenceCode;
public class BinaryGenerator {
private int val = 0;
private int[] values = new int[]{0,1};
//This method converts the Binary Pattern output into a char[] so that it can be printed out to a file
public int[] binaryPatternToString() {
int numBits = 8;
values[0] = 0;
values[1] = 1;
int[] returned = null;
for (int i = 1; i < numBits; i++) {
returned = binaryGenerator(i);
for (int j = 1; j < numBits; j++) {
}
}
return returned;
}
private int[] binaryGenerator(int iVal) {
int[] moreValues = new int[values.length * 2];
int start = (int)Math.pow(2, iVal);
for (int j = 0; j < values.length; j++) {
moreValues[j * 2] = values[j] << 1;
moreValues[j * 2 + 1] = values[j] << 1 | 1;
}
values = moreValues;
for (int value : values) {
System.out.println(Integer.toBinaryString(value));
}
return moreValues;
}}
Would it be a better idea or more efficient to make it a recursive method instead of a method with a for loop?

As you may know all java Integers are based on binary numbers. So for 2 bytes, the maximum number is 2^16 = 65536. Simply loop through all numbers and get their binary values, zero-pad them if necessary and finally store them in a list. This will return all possible 2-byte binary numbers. For more bytes, simply increment the byte-variable.
Implementation:
int bytes = 2;
int nBits = bytes * 8;
int maxNumber = 1 << nBits; //this equals 2^nBits or in java: Math.pow(2,nbits)
ArrayList<String> binaries = new ArrayList<>();
for (int i = 0; i < maxNumber; i++) {
String binary = Integer.toBinaryString(i);
while (binary.length() != nBits) {
binary = "0" + binary;
}
binaries.add(binary);
}
System.out.println(binaries);
The bytes and nBits variables are included simply for clarity.
You can also use a recursive method. Start with an empty String and recursively add a 0 or a 1 to the start of the string and continue until you've reached the number of bits you wanted:
public static ArrayList<String> getBinaries(int bits, String current) {
ArrayList<String> binaries = new ArrayList<>();
if (current.length() == bits) {
binaries.add(current);
return binaries;
}
//pad a 0 and 1 in front of current;
binaries.addAll(getBinaries(bits, "0" + current));
binaries.addAll(getBinaries(bits, "1" + current));
return binaries;
}
You can call this function with: getBinaries(16,"") for 2 bytes.

I took the liberty of writing my own version so you can see a simpler way to produce these numbers.
The hardest part here is incrementing the list of booleans. In general it's just like adding 1. You increment the one's slot, and if it was already a 1, you move on to the 10s slot, and so on. Otherwise, you just loop through all the posobilities, printing each one out.
import java.util.ArrayList;
import java.util.List;
public class Sandbox {
// list of booleans to represent each bit
private static List<Boolean> bytes = new ArrayList<>();
public static void main(String[] args) {
// initialize the list to all false
for(int i = 0; i < 16; i++) {
bytes.add(false);
}
// calculate the number of permutations
int numPermutations = (int)Math.pow(2, 16);
// print the first permutation
print();
// loop through all permutations
for(int i = 0; i < numPermutations; i++) {
// increment the 2 bytes
increment();
// print the current permutation
print();
}
}
/**
* Prints out the current permutation
*/
private static void print() {
// loop through the bytes
for(Boolean bool : bytes) {
// print 1 or 0
if(bool)
System.out.print(1);
else
System.out.print(0);
}
// end the line
System.out.println();
}
/**
* Increment the bytes
*/
private static void increment() {
// set increment position to the end of the list
int position = bytes.size() - 1;
// loop through changing next digit if necessary, stopping
// if the front of the list is reached.
do {
bytes.set(position, !bytes.get(position));
} while(!bytes.get(position--) && position >= 0);
}
}

Write "compressed" Array to increase IO performance?

I have an int and float array each of length 220 million (fixed). Now, I want to store/upload those arrays to/from memory and disk. Currently, I am using Java NIO's FileChannel and MappedByteBuffer to solve this. It works fine, but it takes near about 5 seconds (Wall Clock Time) for storing/uploading array to/from memory to disk. Now, I want to make it faster.
Here, I should mention most of those array elements are 0 ( nearly 52 %).
like:
int arr1 [] = { 0 , 0 , 6 , 7 , 1, 0 , 0 ...}
Can anybody help me, is there any nice way to improve speed by not storing or loading those 0's. This can compensated by using Arrays.fill (array , 0).

The following approach requires n / 8 + nz * 4 bytes on disk, where n is the size of the array, and nz the number of non-zero entries. For 52% zero entries, you'd reduce storage size by 52% - 3% = 49%.
You could do:
void write(int[] array) {
BitSet zeroes = new BitSet();
for (int i = 0; i < array.length; i++)
zeroes.set(i, array[i] == 0);
write(zeroes); // one bit per index
for (int i = 0; i < array.length; i++)
if (array[i] != 0)
write(array[y]);
}
int[] read() {
BitSet zeroes = readBitSet();
array = new int[zeroes.length];
for (int i = 0; i < zeroes.length; i++) {
if (zeroes.get(i)) {
// nothing to do (array[i] was initialized to 0)
} else {
array[i] = readInt();
}
}
}
Edit: That you say this is slightly slower implies that the disk is not the bottleneck. You could tune the above approach by writing the bitset as you construct it, so you don't have to write the bitset to memory before writing it to disk. Also, by writing the bitset word by word interspersed with the actual data we can do only a single pass over the array, reducing cache misses:
void write(int[] array) {
writeInt(array.length);
int ni;
for (int i = 0; i < array.length; i = ni) {
ni = i + 32;
int zeroesMap = 0;
for (j = i + 31; j >= i; j--) {
zeroesMap <<= 1;
if (array[j] == 0) {
zeroesMap |= 1;
}
}
writeInt(zeroesMap);
for (j = i; j < ni; j++)
if (array[j] != 0) {
writeInt(array[j]);
}
}
}
}
int[] read() {
int[] array = new int[readInt()];
int ni;
for (int i = 0; i < array.length; i = ni) {
ni = i + 32;
zeroesMap = readInt();
for (j = i; j < ni; j++) {
if (zeroesMap & 1 == 1) {
// nothing to do (array[i] was initialized to 0)
} else {
array[j] = readInt();
}
zeroesMap >>= 1;
}
}
return array;
}
(The preceeding code assumes array.length is a multiple of 32. If not, write the last slice of the array in whatever way you like)
If that doesn't reduce proceccing time either, compression is not the way to go (I don't think any general purpose compression algorithm will be faster than the above).

Depending upon the distribution, consider Run-length Encoding:
Run-length encoding (RLE) is a very simple form of data compression in which runs of data (that is, sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run. This is most useful on data that contains many such runs.
It is simple ... which is good, and possibly bad, here ;-)

In case you are willing to write the serialization-desirialization code yourself, instead of storing all the zeroes you can store a series of ranges that indicate where those zeros are(with a special marker), together with the actual non-zero data.
So the array in your example: { 0 , 0 , 6 , 7 , 1, 0 , 0 ...}
can be stored as:
%0-1, 6, 7, 1 %5-6
when reading this data, if you hit % it means you have a range in from of you, you read the start and the end and fill an zeroes. Then you go on and see a non #, this means you hit an actual value.
In a sparse array that has large sequences of consecutive values this will yield great compression.

There is a standard compression utils in java: java.util.zip - it's general purpose library but due to sheer availability is an ok solution. Specialized compressions, encoding should be researched, if need arises and I rarely recommend zip as the soultion of choise.
Here is a sample how to handle zip via Deflater/Inflater.
Most people know ZipInput/Output Stream (and esp. Gzip). All of them have downsdes in handling the copy from mem->zlib and esp. GZip which is a total disaster as having CRC32 calling the native code (calling native code removes the ability to optimize and introduces some more performance hits).
Few important notes: do not boost zip compression high, that will kill any performance whatsoever - of course one can experiment and fit their best ratio between CPU and disk activity.
The code also demonstrates one of the real shortcomings of java.util.zip - it doesn't support direct buffers. The support is beyond trivial, yet no one bother to do it. Direct buffers will save few memory copies and reduces the memory footprint.
Last note: there is java version of (j)zlib and it beats the native impl. on compression quite nicely.
package t1;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.util.Random;
import java.util.zip.DataFormatException;
import java.util.zip.Deflater;
import java.util.zip.Inflater;
public class ZInt {
private static final int bucketSize = 1<<17;//in real world should not be const, but we bored horribly
static final int zipLevel = 2;//feel free to experiement, higher compression (5+)is likely to be total waste
static void write(int[] a, File file, boolean sync) throws IOException{
byte[] bucket = new byte[Math.min(bucketSize, Math.max(1<<13, Integer.highestOneBit(a.length >>3)))];//128KB bucket
byte[] zipOut = new byte[bucket.length];
final FileOutputStream fout = new FileOutputStream(file);
FileChannel channel = fout.getChannel();
try{
ByteBuffer buf = ByteBuffer.wrap(bucket);
//unfortunately java.util.zip doesn't support Direct Buffer - that would be the perfect fit
ByteBuffer out = ByteBuffer.wrap(zipOut);
out.putInt(a.length);//write length aka header
if (a.length==0){
doWrite(channel, out, 0);
return;
}
Deflater deflater = new Deflater(zipLevel, false);
try{
for (int i=0;i<a.length;){
i = put(a, buf, i);
buf.flip();
deflater.setInput(bucket, buf.position(), buf.limit());
if (i==a.length)
deflater.finish();
//hacking and using bucket here is tempting since it's copied twice but well
for (int n; (n= deflater.deflate(zipOut, out.position(), out.remaining()))>0;){
doWrite(channel, out, n);
}
buf.clear();
}
}finally{
deflater.end();
}
}finally{
if (sync)
fout.getFD().sync();
channel.close();
}
}
static int[] read(File file) throws IOException, DataFormatException{
FileChannel channel = new FileInputStream(file).getChannel();
try{
byte[] in = new byte[(int)Math.min(bucketSize, channel.size())];
ByteBuffer buf = ByteBuffer.wrap(in);
channel.read(buf);
buf.flip();
int[] a = new int[buf.getInt()];
if (a.length==0)
return a;
int i=0;
byte[] inflated = new byte[Math.min(1<<17, a.length*4)];
ByteBuffer intBuffer = ByteBuffer.wrap(inflated);
Inflater inflater = new Inflater(false);
try{
do{
if (!buf.hasRemaining()){
buf.clear();
channel.read(buf);
buf.flip();
}
inflater.setInput(in, buf.position(), buf.remaining());
buf.position(buf.position()+buf.remaining());//simulate all read
for (;;){
int n = inflater.inflate(inflated,intBuffer.position(), intBuffer.remaining());
if (n==0)
break;
intBuffer.position(intBuffer.position()+n).flip();
for (;intBuffer.remaining()>3 && i<a.length;i++){//need at least 4 bytes to form an int
a[i] = intBuffer.getInt();
}
intBuffer.compact();
}
}while (channel.position()<channel.size() && i<a.length);
}finally{
inflater.end();
}
// System.out.printf("read ints: %d - channel.position:%d %n", i, channel.position());
return a;
}finally{
channel.close();
}
}
private static void doWrite(FileChannel channel, ByteBuffer out, int n) throws IOException {
out.position(out.position()+n).flip();
while (out.hasRemaining())
channel.write(out);
out.clear();
}
private static int put(int[] a, ByteBuffer buf, int i) {
for (;buf.hasRemaining() && i<a.length;){
buf.putInt(a[i++]);
}
return i;
}
private static int[] generateRandom(int len){
Random r = new Random(17);
int[] n = new int[len];
for (int i=0;i<len;i++){
n[i]= r.nextBoolean()?0: r.nextInt(1<<23);//limit bounds to have any sensible compression
}
return n;
}
public static void main(String[] args) throws Throwable{
File file = new File("xxx.xxx");
int[] n = generateRandom(3000000); //{0,2,4,1,2,3};
long start = System.nanoTime();
write(n, file, false);
long elapsed = System.nanoTime() - start;//elapsed will be fairer if the sync is true
System.out.printf("File length: %d, for %d ints, ratio %.2f in %.2fms %n", file.length(), n.length, ((double)file.length())/4/n.length, java.math.BigDecimal.valueOf(elapsed, 6) );
int[] m = read(file);
//compare, Arrays.equals doesn't return position, so it sucks/kinda
for (int i=0; i<n.length; i++){
if (m[i]!=n[i]){
System.err.printf("Failed at %d%n",i);
break;
}
}
System.out.printf("All done!");
};
}
Please note, the code is not a proper benchmark!
The delayed replies comes from the fact it was quite boring to code, yet another zip example, sorry

Recursion and Immutability

I'm having the hardest time trying to get my class to work properly. It's a natural number class with methods like increase and decrease. I'm trying to make it immutable, and I'm stuck. If I increment a number such that it's least significant digit isn't 9, it works fine. But once I get to the boundary case, it fails.
IE. I have a number that's 69999, I increment it and it's 7.
private SlowBigNatural(int[] natural, int nSize){
this.nSize = nSize - 1;
this.natural = new int[this.nSize];
for (int i = 0; i < this.nSize; i++) {
this.natural[i] = natural[i];
}
}
#Override
public BigNatural increment() {
int[] nClone = natural.clone();
if (nSize == 1 || nClone[nSize - 1] != HIGHEST) {
nClone[nSize - 1]++;
String nString = "";
for(int i = 0; i < nSize; i++){
nString += String.valueOf(nClone[i]);
}
BigNatural nInc = new SlowBigNatural(nString);
return nInc;
}
else {
nClone[nSize - 1] = 0;
BigNatural temp = new SlowBigNatural(nClone, nSize);
return temp.increment();
}
}

I'm a little confused as to the endianness of your BigNatural. You don't explicitly mention how you're storing your number. You can either store with the most significant digit first, or least significant digit first.
That is the number 12,345 can be stored as {1,2,3,4,5} (Big Endian) or {5,4,3,2,1} (Little Endian).
If your increment is correct then getting 7 from adding 1 to 69,999 is likely a problem with endianness.
That is {7,0,0,0,0} is 70,000 if Big Endian or 7 if Little Endian. Check the string constructor. See what endianness it expects.

The problem is in the else. You have to compose the number after increment:
else {
nClone[nSize - 1] = 0;
BigNatural temp = new SlowBigNatural(nClone, nSize);
BigNatural incremented = temp.increment();
return new SlowBigNatural(addElement(incremented.natural, 0), nSize + 1);
}
private int[] addElement(int[] arr, int element) {
int[] copy = new int[arr.length+1];
for (int index = 0; index < arr.length+1; index++){
copy[index] = arr[index-1];
}
copy[arr.length] = element;
return copy;
}