Develop Reference

What are the assumption under default Long.hashCode implementation in Java? [duplicate]

I was wondering if someone could explain in detail what
(int)(l ^ (l >>> 32));
does in the following hashcode implementation (generated by eclipse, but the same as Effective Java):
private int i;
private char c;
private boolean b;
private short s;
private long l;
private double d;
private float f;
#Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + i;
result = prime * result + s;
result = prime * result + (b ? 1231 : 1237);
result = prime * result + c;
long t = Double.doubleToLongBits(d);
result = prime * result + (int) (t ^ (t >>> 32));
result = prime * result + Float.floatToIntBits(f);
result = prime * result + (int) (l ^ (l >>> 32));
return result;
}
Thanks!

Basically it XORs the top 32 bits of a long with the bottom 32 bits. Here's an exploded version:
// Unsigned shift by 32 bits, so top 32 bits of topBits will be 0,
// bottom 32 bits of topBits will be the top 32 bits of l
long topBits = l >>> 32;
// XOR topBits with l; the top 32 bits will effectively be left
// alone, but that doesn't matter because of the next step. The
// bottom 32 bits will be the XOR of the top and bottom 32 bits of l
long xor = l ^ topBits;
// Convert the long to an int - this basically ditches the top 32 bits
int hash = (int) xor;
To answer your comment: you have a long value which has to be converted into an int to be part of the hash (the result has to only be 32 bits). How are you going to do that? You could just take the bottom 32 bits - but then that means changes in only the top 32 bits would be ignored, which wouldn't make it a very good hash. This way, a change in a single bit of input always results in a change of a single bit of the hash. Admittedly you can still get collisions easily - change both bits 7 and 39, for example, or any other pair of bits 32 positions apart - but that's bound to be the case, given that you're going from 264 possible values to 232.

It takes a 64 bit number, splits it half, and xors the two halves together (essentially).

It takes a (64-bit) long l, exclusive-or's the top and bottom halves (of 32 bits each) into the bottom 32 bits of a 64-bit results, then takes only the bottom 32 bits with the (int) cast.

How to represent bit fields (and send them) in Java

In my current project I need to send over the network a parsed structure that contains some n-bit fields. for instance:
protocol version: 1 byte
messageId: 1 byte
creationTime: 6 bytes
traceId: 3 bits
reliability: 7 bits
Etc...
Thus, I created a simple POJO class to represent this for parsing & unparsing, but I have some doubts on what type to use for those fields, since this decision can make parsing & unparsing easy or a bit of a nightmare. I must say that the Message to be sent over the network has a very specific size constraint: it cannot surpass the sum of all fields.
I first tought in using bytes for everything and then have a Message.getBytes() method that will convert the message and for those fields that are less than a byte, use bitwise operations to discard the unnecessary bits.
Am I going in the right direction or there is another much simpler way to do this? I just feel I'm re-inventing the wheel here, this feels kinda boilerplate code...
Thanks!
EDIT: If anybody else stumbles here, I'll post how I solved this (thanks to a mate at work that helped me with these), so just keep reading:
Luckily enough, my protocol socket size rounds up to a fixed number of bytes (49) and those fields that are less than byte size, sum up a byte at then end, resulting in that I can combine both fields in a single byte prior parsing/unparsing.
That said, imagine I have two fields, say field1 and field2, first in 7 bits and the other just a bit. To combine those, I just do this trick:
byte resultingByte = short2Byte((short) ((field1 % 128) * 2 + (field2 ? 1 : 0)));
Note that both field1 and field2 are short type. I found this the most convenient way to work at the bit level. Thus, I mod first field, making sure I'm only getting 7 bits, move the bits to the left divining by 2 since only one bit is needed to be moved. Lastly I add the field 2 short, which can be 1 or 0. Then I have a short with the required values in the 8 Less Signigicant Bits.
I created commodity methods to convert from short2Byte, Long, and some others:
private byte [] to2Bytes(int in) {
ByteBuffer ret = ByteBuffer.allocate(2);
int val = in % 65536;
short s1 = (short) (val / 256);
short s0 = (short) (val % 256);
ret.put(short2Byte(s1));
ret.put(short2Byte(s0));
return ret.array();
}
private byte [] to4Bytes(long in) {
ByteBuffer ret = ByteBuffer.allocate(4);
long div = 4294967296L;
long val = in % div;
int rem = 0;
short s3 = (short) (val / 16777216L);
rem = (int) (val % 16777216L);
short s2 = (short) (rem / 65536);
rem = rem % 65536;
short s1 = (short) (rem / 256);
short s0 = (short) (rem % 256);
ret.put(short2Byte(s3));
ret.put(short2Byte(s2));
ret.put(short2Byte(s1));
ret.put(short2Byte(s0));
return ret.array();
}
private byte [] time2Bytes(Long time) {
ByteBuffer ret = ByteBuffer.allocate(6);
String hex = Long.toHexString(time).toUpperCase();
while (hex.length() < 12) {
hex = "0" + hex;
}
while (hex.length() > 12) {
hex = hex.substring(1);
}
try {
for (int i = 0; i < 6; i++) {
String strByte = "" + hex.charAt(i*2) + hex.charAt(i*2 + 1);
short b = Short.parseShort(strByte, 16);
if (b > 127) {
b -= 256;
}
ret.put((byte) b);
}
}
catch (NumberFormatException e) {
// Exception captured for correctness
e.printStackTrace();
}
return ret.array();
}
private long bytes2time(byte b5, byte b4, byte b3, byte b2, byte b1, byte b0) {
long l5, l4, l3, l2, l1, l0;
l5 = byte2short(b5) * 1099511627776L;
l4 = byte2short(b4) * 4294967296L;
l3 = byte2short(b3) * 16777216L;
l2 = byte2short(b2) * 65536L;
l1 = byte2short(b1) * 256L;
l0 = byte2short(b0) * 1L;
return l5 + l4 + l3 + l2 + l1 + l0;
}
private long bytes2long(byte b3, byte b2, byte b1, byte b0) {
long l3, l2, l1, l0;
l3 = byte2short(b3) * 16777216L;
l2 = byte2short(b2) * 65536L;
l1 = byte2short(b1) * 256L;
l0 = byte2short(b0) * 1L;
return l3 + l2 + l1 + l0;
}
private int bytes2int(byte b1, byte b0) {
return (int)byte2short(b1) * 256 + (int)byte2short(b0);
}
private short byte2short(byte b) {
if (b < 0) {
return (short) (b+256);
}
return (short)b;
}
private byte short2Byte(short s) {
if (s < 128) {
return (byte) s;
}
else {
return (byte) (s-256);
}
}
At the end I'm sending a byte array with 49 bytes. Unparsing is very similar process, obviously. There must be a proper way to do this, but well, it works...Hope this helps someone!

You can to use ByteBuffer and BitSet class to write and read messages, but it can still become a nightmare even worse then pure bit manipulation (and it will affect performance)

Inverse function of Java's Random function

Java's Random function takes a seed and produces the a sequence of 'psuedo-random' numbers.
(It is implemented based on some algorithm discussed in Donald Knuth, The Art of Computer Programming, Volume 3, Section 3.2.1.), but the article is too technical for me to understand)
Is there an inverse function of it?
That is, given a sequence of numbers, would it be possible to mathematically determine what the seed would be?
(, which means, brute-forcing doesn't count as a valid method)
[Edit]
There seems to be quite a number of comments here... I thought I'd clarify what I am looking for.
So for instance, the function y = f(x) = 3x has an inverse function, which is y = g(x) = x/3.
But the function z = f(x, y) = x * y does not have an inverse function, because (I could give a full mathematical proof here, but I don't want to sidetrack my main question), intuitively speaking, there are more than one pair of (x, y) such that (x * y) == z.
Now back to my question, if you say the function is not inversible, please explain why.
(And I am hoping to get answers from those who have really read to article and understand it. Answers like "It's just not possible" aren't really helping)

If we're talking about the Oracle (née Sun) implementation of java.util.Random, then yes, it is possible once you know enough bits.
Random uses a 48-bit seed and a linear congruential generator. These are not cryptographically safe generators, because of the tiny state size (bruteforceable!) and the fact that the output just isn't that random (many generators will exhibit small cycle length in certain bits, meaning that those bits can be easily predicted even if the other bits seem random).
Random's seed update is as follows:
nextseed = (seed * 0x5DEECE66DL + 0xBL) & ((1L << 48) - 1)
This is a very simple function, and it can be inverted if you know all the bits of the seed by calculating
seed = ((nextseed - 0xBL) * 0xdfe05bcb1365L) & ((1L << 48) - 1)
since 0x5DEECE66DL * 0xdfe05bcb1365L = 1 mod 248. With this, a single seed value at any point in time suffices to recover all past and future seeds.
Random has no functions that reveal the whole seed, though, so we'll have to be a bit clever.
Now, obviously, with a 48-bit seed, you have to observe at least 48 bits of output or you clearly don't have an injective (and thus invertible) function to work with. We're in luck: nextLong returns ((long)(next(32)) << 32) + next(32);, so it produces 64 bits of output (more than we need). Indeed, we could probably make do with nextDouble (which produces 53 bits), or just repeated calls of any other function. Note that these functions cannot output more than 248 unique values because of the seed's limited size (hence, for example, there are 264-248 longs that nextLong will never produce).
Let's specifically look at nextLong. It returns a number (a << 32) + b where a and b are both 32-bit quantities. Let s be the seed before nextLong is called. Then, let t = s * 0x5DEECE66DL + 0xBL, so that a is the high 32 bits of t, and let u = t * 0x5DEECE66DL + 0xBL so that b is the high 32 bits of u. Let c and d be the low 16 bits of t and u respectively.
Note that since c and d are 16-bit quantities, we can just bruteforce them (since we only need one) and be done with it. That's pretty cheap, since 216 is only 65536 -- tiny for a computer. But let's be a bit more clever and see if there's a faster way.
We have (b << 16) + d = ((a << 16) + c) * 0x5DEECE66DL + 11. Thus, doing some algebra, we obtain (b << 16) - 11 - (a << 16)*0x5DEECE66DL = c*0x5DEECE66DL - d, mod 248. Since c and d are both 16-bit quantities, c*0x5DEECE66DL has at most 51 bits. This usefully means that
(b << 16) - 11 - (a << 16)*0x5DEECE66DL + (k<<48)
is equal to c*0x5DEECE66DL - d for some k at most 6. (There are more sophisticated ways to compute c and d, but because the bound on k is so tiny, it's easier to just bruteforce).
We can just test all the possible values for k until we get a value whos negated remainder mod 0x5DEECE66DL is 16 bits (mod 248 again), so that we recover the lower 16 bits of both t and u. At that point, we have a full seed, so we can either find future seeds using the first equation, or past seeds using the second equation.
Code demonstrating the approach:
import java.util.Random;
public class randhack {
public static long calcSeed(long nextLong) {
final long x = 0x5DEECE66DL;
final long xinv = 0xdfe05bcb1365L;
final long y = 0xBL;
final long mask = ((1L << 48)-1);
long a = nextLong >>> 32;
long b = nextLong & ((1L<<32)-1);
if((b & 0x80000000) != 0)
a++; // b had a sign bit, so we need to restore a
long q = ((b << 16) - y - (a << 16)*x) & mask;
for(long k=0; k<=5; k++) {
long rem = (x - (q + (k<<48))) % x;
long d = (rem + x)%x; // force positive
if(d < 65536) {
long c = ((q + d) * xinv) & mask;
if(c < 65536) {
return ((((a << 16) + c) - y) * xinv) & mask;
}
}
}
throw new RuntimeException("Failed!!");
}
public static void main(String[] args) {
Random r = new Random();
long next = r.nextLong();
System.out.println("Next long value: " + next);
long seed = calcSeed(next);
System.out.println("Seed " + seed);
// setSeed mangles the input, so demangle it here to get the right output
Random r2 = new Random((seed ^ 0x5DEECE66DL) & ((1L << 48)-1));
System.out.println("Next long value from seed: " + r2.nextLong());
}
}

I normally wouldn't just link articles... But I found a site where someone looks into this in some depth and thought it was worth posting. http://jazzy.id.au/default/2010/09/20/cracking_random_number_generators_part_1.html
It seems that you can calculate a seed this way:
seed = (seed * multiplier + addend) mod (2 ^ precision)
where multiplier is 25214903917, addend is 11, and precision is 48 (bits). You can't calculate what the seed was with only 1 number, but you can with 2.
EDIT: As nhahtdh said there's a part 2 where he delves into more of the math behind the seeds.

I would like to present an implementation to reverse a sequence of integers generated by nextInt().
The program will brute force on the lower 16-bit discarded by nextInt(), use the algorithm provided in the blog by James Roper to find previous seed, then check that upper 32 bit of the 48-bit seed are the same as the previous number. We need at least 2 integers to derive the previous seed. Otherwise, there will be 216 possibilities for the previous seed, and all of them are equally valid until we have at least one more number.
It can be extended for nextLong() easily, and 1 long number is enough to find the seed, since we have 2 pieces of upper 32-bit of the seed in one long, due to the way it is generated.
Note that there are cases where the result is not the same as what you set as secret seed in the SEED variable. If the number you set as secret seed occupies more than 48-bit (which is the number of bits used for generating random numbers internally), then the upper 16 bits of 64 bit of long will be removed in the setSeed() method. In such cases, the result returned will not be the same as what you have set initially, it is likely that the lower 48-bit will be the same.
I would like to give most the credit to James Roper, the author of this blog article which makes the sample code below possible:
import java.util.Random;
import java.util.Arrays;
class TestRandomReverse {
// The secret seed that we want to find
private static long SEED = 782634283105L;
// Number of random numbers to be generated
private static int NUM_GEN = 5;
private static int[] genNum(long seed) {
Random rand = new Random(seed);
int arr[] = new int[NUM_GEN];
for (int i = 0; i < arr.length; i++) {
arr[i] = rand.nextInt();
}
return arr;
}
public static void main(String args[]) {
int arr[] = genNum(SEED);
System.out.println(Arrays.toString(arr));
Long result = reverse(arr);
if (result != null) {
System.out.println(Arrays.toString(genNum(result)));
} else {
System.out.println("Seed not found");
}
}
private static long combine(int rand, int suffix) {
return (unsignedIntToLong(rand) << 16) | (suffix & ((1L << 16) - 1));
}
private static long unsignedIntToLong(int num) {
return num & ((1L << 32) - 1);
}
// This function finds the seed of a sequence of integer,
// generated by nextInt()
// Can be easily modified to find the seed of a sequence
// of long, generated by nextLong()
private static Long reverse(int arr[]) {
// Need at least 2 numbers.
assert (arr.length > 1);
int end = arr.length - 1;
// Brute force lower 16 bits, then compare
// upper 32 bit of the previous seed generated
// to the previous number.
for (int i = 0; i < (1 << 16); i++) {
long candidateSeed = combine(arr[end], i);
long previousSeed = getPreviousSeed(candidateSeed);
if ((previousSeed >>> 16) == unsignedIntToLong(arr[end - 1])) {
System.out.println("Testing seed: " +
previousSeed + " --> " + candidateSeed);
for (int j = end - 1; j >= 0; j--) {
candidateSeed = previousSeed;
previousSeed = getPreviousSeed(candidateSeed);
if (j > 0 &&
(previousSeed >>> 16) == unsignedIntToLong(arr[j - 1])) {
System.out.println("Verifying: " +
previousSeed + " --> " + candidateSeed);
} else if (j == 0) {
// The XOR is done when the seed is set, need to reverse it
System.out.println("Seed found: " + (previousSeed ^ MULTIPLIER));
return previousSeed ^ MULTIPLIER;
} else {
System.out.println("Failed");
break;
}
}
}
}
return null;
}
private static long ADDEND = 0xBL;
private static long MULTIPLIER = 0x5DEECE66DL;
// Credit to James Roper
// http://jazzy.id.au/default/2010/09/21/cracking_random_number_generators_part_2.html
private static long getPreviousSeed(long currentSeed) {
long seed = currentSeed;
// reverse the addend from the seed
seed -= ADDEND; // reverse the addend
long result = 0;
// iterate through the seeds bits
for (int i = 0; i < 48; i++)
{
long mask = 1L << i;
// find the next bit
long bit = seed & mask;
// add it to the result
result |= bit;
if (bit == mask)
{
// if the bit was 1, subtract its effects from the seed
seed -= MULTIPLIER << i;
}
}
return result & ((1L << 48) - 1);
}
}

How to count the number of 1's a number will have in binary? [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Best algorithm to count the number of set bits in a 32-bit integer?
How do I count the number of 1's a number will have in binary?
So let's say I have the number 45, which is equal to 101101 in binary and has 4 1's in it. What's the most efficient way to write an algorithm to do this?

Instead of writing an algorithm to do this its best to use the built in function. Integer.bitCount()
What makes this especially efficient is that the JVM can treat this as an intrinsic. i.e. recognise and replace the whole thing with a single machine code instruction on a platform which supports it e.g. Intel/AMD
To demonstrate how effective this optimisation is
public static void main(String... args) {
perfTestIntrinsic();
perfTestACopy();
}
private static void perfTestIntrinsic() {
long start = System.nanoTime();
long countBits = 0;
for (int i = 0; i < Integer.MAX_VALUE; i++)
countBits += Integer.bitCount(i);
long time = System.nanoTime() - start;
System.out.printf("Intrinsic: Each bit count took %.1f ns, countBits=%d%n", (double) time / Integer.MAX_VALUE, countBits);
}
private static void perfTestACopy() {
long start2 = System.nanoTime();
long countBits2 = 0;
for (int i = 0; i < Integer.MAX_VALUE; i++)
countBits2 += myBitCount(i);
long time2 = System.nanoTime() - start2;
System.out.printf("Copy of same code: Each bit count took %.1f ns, countBits=%d%n", (double) time2 / Integer.MAX_VALUE, countBits2);
}
// Copied from Integer.bitCount()
public static int myBitCount(int i) {
// HD, Figure 5-2
i = i - ((i >>> 1) & 0x55555555);
i = (i & 0x33333333) + ((i >>> 2) & 0x33333333);
i = (i + (i >>> 4)) & 0x0f0f0f0f;
i = i + (i >>> 8);
i = i + (i >>> 16);
return i & 0x3f;
}
prints
Intrinsic: Each bit count took 0.4 ns, countBits=33285996513
Copy of same code: Each bit count took 2.4 ns, countBits=33285996513
Each bit count using the intrinsic version and loop takes just 0.4 nano-second on average. Using a copy of the same code takes 6x longer (gets the same result)

The most efficient way to count the number of 1's in a 32-bit variable v I know of is:
v = v - ((v >> 1) & 0x55555555);
v = (v & 0x33333333) + ((v >> 2) & 0x33333333);
c = ((v + (v >> 4) & 0xF0F0F0F) * 0x1010101) >> 24; // c is the result
Updated: I want to make clear that it's not my code, actually it's older than me. According to Donald Knuth (The Art of Computer Programming Vol IV, p 11), the code first appeared in the first textbook on programming, The Preparation of Programs for an Electronic Digital Computer by Wilkes, Wheeler and Gill (2nd Ed 1957, reprinted 1984). Pages 191–193 of the 2nd edition of the book presented Nifty Parallel Count by D B Gillies and J C P Miller.

See Bit Twidling Hacks and study all the 'counting bits set' algorithms. In particular, Brian Kernighan's way is simple and quite fast if you expect a small answer. If you expect an evenly distributed answer, lookup table might be better.

This is called Hamming weight. It is also called the population count, popcount or sideways sum.

The following is either from "Bit Twiddling Hacks" page or Knuth's books (I don't remember). It is adapted to unsigned 64 bit integers and works on C#. I don't know if the lack of unsigned values in Java creates a problem.
By the way, I write the code only for reference; the best answer is using Integer.bitCount() as #Lawrey said; since there is a specific machine code operation for this operation in some (but not all) CPUs.
const UInt64 m1 = 0x5555555555555555;
const UInt64 m2 = 0x3333333333333333;
const UInt64 m4 = 0x0f0f0f0f0f0f0f0f;
const UInt64 h01 = 0x0101010101010101;
public int Count(UInt64 x)
{
x -= (x >> 1) & m1;
x = (x & m2) + ((x >> 2) & m2);
x = (x + (x >> 4)) & m4;
return (int) ((x * h01) >> 56);
}

public int f(int n)
{
int result = 0;
for(;n > 0; n = n >> 1)
result += ((n & 1) == 1 ? 1 : 0);
return result;
}

The following Ruby code works for positive numbers.
count = 0
while num > 1
count = (num % 2 == 1) ? count + 1 : count
num = num >> 1
end
count += 1
return count

The fastest I have used and also seen in a practical implementation (in the open source Sphinx Search Engine) is the MIT HAKMEM algorithm. It runs superfast over a very large stream of 1's and 0's.

counting number of ones in binary representation of a number [duplicate]

This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Best algorithm to count the number of set bits in a 32-bit integer?
I want to find out how many 1s are there in binary representation of a number.I have 2 logic .
int count =0;
int no = 4;
while(no!=0){
int d = no%2;
if(d==1)
count++;
no = no/2;
str = str+ d;
}
Now second logic is to keep on masking number iteratively with 1,2,4,8,32 and check if result is 1,2,4, 8..... Am not geting what should be ending condition for this loop.

Use Java API(java 5 or above).
Integer.bitCount(int);
Long.bitCount(long);
NOTE: The above java methods are based on hacker's delight

faster than any of the earlier answers:
(proportional to number of 1 bits rather than total bits)
public class Foo {
public static void main(String[] argv) throws Exception {
int no = 12345;
int count;
for (count = 0; no > 0; ++count) {
no &= no - 1;
}
System.out.println(count);
}
}

Looks like c/c++/c#, if so you have shifting.. just loop to N-1 bits from 0 and use sum+=(value>>i)&1
Ie: you always check the last/right most bit but move the binary representation of the number to the right for every iteration until you have no more bits to check.
Also, think about signed/unsigned and any integer format. But you dont state how that should be handled in the question.

We can make use of overflow for your loop:
int count = 0;
int number = 37;
int mask = 1;
while(mask!=0)
{
int d = number & mask;
if(d != 0)
count++;
/* Double mask until we overflow, which will result in mask = 0... */
mask = mask << 1;
str = str+ d;
}

One idea that's commonly employed for counting ones is to build a lookup table containing the answers for each individual byte, then to split apart your number into four bytes and sum the totals up. This requires four lookups and is quite fast. You can build this table by writing a program that manually computes the answer (perhaps using your above program), and then can write a function like this:
private static final int[] BYTE_TOTALS = /* ... generate this ... */;
public static int countOneBits(int value) {
return BYTE_TOTALS[value & 0xFF] +
BYTE_TOTALS[value >>> 8 & 0xFF] +
BYTE_TOTALS[value >>> 16 & 0xFF] +
BYTE_TOTALS[value >>> 24 & 0xFF];
}
Hope this helps!

There are various ways to do this very fast.
MIT HAKMEM Count
int no =1234;
int tmp =0;
tmp = no - ((no >> 1) & 033333333333) - ((no >> 2) & 011111111111);
System.out.println( ((tmp + (tmp >> 3)) & 030707070707) % 63);

Your end condition should be keeping track of the magnitude of the bit you are at; if it is larger than the original number you are done (will get only 0s from now on).
Oh, and since you didn't specify a language, here's a Ruby solution :)
class Integer
def count_binary_ones
to_s(2).scan('1').length
end
end
42.count_binary_ones #=> 3

How about using the BigInteger class.
public void function(int checkedNumber) {
BigInteger val = new BigInteger(String.valueOf(checkedNumber));
val = val.abs();
int count = val.bitCount();
String binaryString = val.toString(2);
System.out.println("count = " + count);
System.out.println("bin = " + binaryString);
}
The result of function(42); is following.
count = 3
bin = 101010