Develop Reference

Why random of 0 to 4 is 1 most of times?

I am using a simple random calculations for a small range with 4 elements.
indexOfNext = new Random().nextInt(4); //randomize 0 to 3
When I attach the debugger I see that for the first 2-3 times every time the result is 1.
Is this the wrong method for a small range or I should implement another logic (detecting previous random result)?
NOTE: If this is just a coincidence, then using this method is the wrong way to go, right? If yes, can you suggest alternative method? Shuffle maybe? Ideally, the alternative method would have a way to check that next result is not the same as the previous result.

Don't create a new Random() each time, create one object and call it many times.
This is because you need one random generator and many numbers from its
random sequence, as opposed to having many random generators and getting
just the 1st numbers of the random sequences they generate.

You're abusing the random number generator by creating a new instance repeatedly. (This is due to the implementation setting a starting random number value that's a very deterministic function of your system clock time). This ruins the statistical properties of the generator.
You should create one instance, then call nextInt as you need numbers.

One thing you can do is hold onto the Random instance. It has to seed itself each time you instantiate it and if that seed is the same then it will generate the same random sequence each time.
Other options are converting to SecureRandom, but you definitely want to hold onto that random instance to get the best random number performance. You really only need SecureRandom is you are randomly generating things that have security implications. Like implementing crypto algorithms or working around such things.

"Random" doesn't mean "non-repeating", and you cannot judge randomness by short sequences. For example, imagine that you have 2 sequences of 1 and 0:
101010101010101010101010101010101
and
110100100001110100100011111010001
Which looks more random? Second one, of course. But when you take any short sequence of 3-4-5 numbers from it, such sequence will look less random than taken from the first one. This is well-known and researched paradox.

A pseudo-random number generator requires a seed to work. The seed is a bit array which size depends on the implementation. The initial seed for a Random can either be specified manually or automatically assigned.
Now there are several common ways of assigning a random seed. One of them is ++currentSeed, the other is current timestamp.
It is possible that java uses System.currentTimeMillis() to get the timestamp and initialize the seed with it. Since the resolution of the timestamp is at most a millisecond (it differs on some machines, WinXP AFAIK had 3ms) all Random instances instantiated in the same millisecond-resolution window will have the same seed.
Another feature of pseudo-random number generators is that if they have the same seed, they return the same numbers in the same order.
So if you get the first pseudo-random number returned by several Randoms initialized with the same seed you are bound to get the same number for a few times. And I suspect that's what's happening in your case.

It seems that the number 1 has a 30% chance of showing its self which is more than other numbers. You can read this link for more information on the subject.
You can also read up on Benford's law.

I'd like to understand Java's SecureRandom object

While doing a beginner's crypto course I'm trying to get to grips with Java's SecureRandom object. What I think I understand is that:
a) No matter how long a sequence of random numbers you know, there is no way of predicting the next random number in the sequence.
b) No matter how long a sequence of random numbers you know, there is no way of knowing which seed was used to start them off, other than brute force guesswork.
c) You can request secure random numbers of various sizes.
d) You can seed a newly-created SRNG with various different-sized values. Every newly-created SRNG you create and seed with the same value will produce the same sequence of random numbers.
I should add that I'm assuming that this code is used on Windows:
Random sr = SecureRandom.getInstance("SHA1PRNG", "SUN");
Is my basic understanding correct? Thanks in advance.
I have some further questions for anyone who's reasonably expert in crypto. They relate to seeding a SRNG as opposed to letting it seed itself on first use.
e) What difference, if any, does it make to the random numbers generated, if you seed a SRNG with a long integer as opposed to an array of 8 bytes?
f) If I seed a SRNG with, say, 256 bytes is there any other seed that can produce the same sequence of random numbers?
g) Is there some kind of optimum seed size? It seems to me that this might be a meaningless question.
h) If I encrypt a plaintext by seeding a SRNG with, say, 256 bytes then getting it to generate random bytes to XOR with the bytes in the plaintext, how easy would it be for an eavesdropper to decrypt the resulting ciphertext? How long might it take? Am I right in thinking that the eavesdropper would have to know, guess, or calculate the 256-byte seed?
I've looked at previous questions about SecureRandom and none seem to answer my particular concerns.
If any of these questions seem overly stupid, I'd like to reiterate that I'm very much a beginner in studying this field. I'd be very grateful for any input as I want to understand how Java SecureRandom objects might be used in cryptography.

d) This is true for a PRNG. It is not always true for a CSRNG. Read the Javadoc for SecureRandom.setSeed(): "The given seed supplements, rather than replaces, the existing seed. Thus, repeated calls are guaranteed never to reduce randomness."
Any reasonable CSRNG will have "invisible" sources of entropy that you cannot explicitly control, often various internal parameters taken from the Operating System level. Hence there is more seeding than any number you explicitly pass to the RNG.

OK, in order:
a) correct
b) correct
c) correct, you can even request a number in a range [0, n) using nextInt(n)
d) as good as correct: the implementation of SHA1PRNG is not publicly defined by any algorithm and there are indications that the implementation has changed in time, so this is only true for the Sun provider, and then only for a specific runtime configuration
e) as the API clearly indicates that all the bytes within the long are used ("using the eight bytes contained in the given long seed") there should not be any difference regarding the amount of entropy added to the state
Note that a quick check shows that setSeed(long) behaves entirely different from setSeed(byte[]) with the main difference that the seed of the long value is always mixed in with randomness retrieved from the system, even if it is the first call after the SecureRandom instance is constructed.
f) yes - an infinite number of seeds generate the same stream; since a hash function is used, it will be impossible to find one though
g) if you mix in additional entropy, then the more entropy the better, but there is no minimum; if you use it as the only seed then you should not start off with less than 20 bytes of seed, that is: if you want to keep the seed to the same security constraints as the inner state of the PRNG
And I would add that if you use less than 64 bytes of entropy you are in the danger zone for sure. Note that 1 bit of entropy does not always mean 1 bit in a byte. A byte array of size 8 may have 64 bits of entropy or less.
h) that's basically a hash based stream cipher; it's secure, so an attacker has little to no chance (given you don't reuse the seed) but it is a horribly unreliable (see answer d) and slow stream cipher, so please never ever do this - use a Cipher with "AES/CTR/NoPadding" or "AES/GCM/NoPadding" instead

e) I don't think that it makes a difference. Assuming that the long and the 8-byte array contain the same data.
f) In principle, yes. If your seed is larger than the internal state of the RNG, then there may exist some other seed that will result in the same internal state. If the seed is smaller than the state, then there shouldn't be. I don't know what SecureRandom's internal state looks like.
g) It's not the size of the seed that matters; it's the amount of entropy in it. You need there to be at least as much entropy in your seed as the security you expect out of the RNG; I'm not quite sure what best practices are here.
h) I'm not sure how easy it would be to break the RNG-based stream cipher that you propose. But I would recommend against using it in practice, because it's not a standard cryptographic construction that has been reviewed by experts and has reasonable security proofs. Remember the Rules of Crypto:
Never design your own crypto.
Never implement your own crypto.
Anyone can design crypto that they can't break themselves.

How to really shuffle a deck of cards

When I need to shuffle a deck of poker cards in Java/Android, I use Collections.shuffle(List<?> list), of course. I've ever been doing this and the results seemed acceptable. But they aren't.
As outlined in this paper, there are 52! possible unique shuffles of a 52 card poker deck. That amounts to about 2^226.
But Collections.shuffle(List<?> list) uses new Random() by default which uses a 48-bit seed and can therefore only create 2^48 unique shuffles - which is only 3.49*10^(-52) percent of all possible shuffles!
So how do I shuffle cards the right way?
I've started using SecureRandom, but is that enough, finally?
List<Card> cards = new ArrayList<Card>();
...
SecureRandom secureRandom;
try {
secureRandom = SecureRandom.getInstance("SHA1PRNG");
}
catch (NoSuchAlgorithmException e) {
secureRandom = new SecureRandom();
}
secureRandom.nextBytes(new byte[20]); // force SecureRandom to seed itself
Collections.shuffle(cards, secureRandom);

You may only be able to get 248 different hands from a specific starting arrangement but there's no requirement that you start at the same arrangement each time.
Presumably, after the deck is finished (poker hands, blackjack and so on), it will be in an indeterminate order, and any one of those rearrangements will be suitable.
And, if you're worried about the fact that you start from a fixed arrangement each time you start your program, just persist the order when exiting and reload it next time.
In any case, 248 is still a huge number of possibilities (some 280,000,000,000,000), more than adequate for a card game, more so when you come to a realisation that it's limiting shuffles rather than arrangements. Unless you're a serious statistician or cryptographer, what you have should be fine.

Although you are using a SecureRandom, is still has a limited state. As long as that input seed has a smaller range than 52! it can not be completely random.
In fact, SHA1PRNG is 160 bit seeded, which means it is still not random enough. Follow this link, it has a solution years ago by using a third party library called UnCommons Math.

If you want real randomness, you could just skip pseudo random generators and go for something better like random numbers generated from athmospheric noise.
random.org offers an API to integrate random numbers generated that way into your own software.

Stealing an answer from the article you link:
START WITH FRESH DECK
GET RANDOM SEED
FOR CT = 1, WHILE CT <= 52, DO
X = RANDOM NUMBER BETWEEN CT AND 52 INCLUSIVE
SWAP DECK[CT] WITH DECK[X]
The random number generator should be good and use a 64 bit seed that you pick unpredictably, preferably using hardware.

How to really shuffle a deck?
There are several shuffling techniques.
Either (Stripping/Overhand):
Cut the deck in two
Add a small (pseudorandom) amount of one half to the front of the front of the other
Add a small (pseudorandom) amount of one half to the front of the back of the other
Do this until one hand is empty
Repeat
Or (Riffle):
Cut the deck in two
Set down a small (pseudorandom) portion of one half
Set down a small (pseudorandom) portion of the other
Do this until both hands are empty, and you have a new deck
Repeat
And there are more on top of this, as detailed in my link above.
Regardless, there are so many combinations that even the perfect shuffling algorithm would take a machine exploring 2*10^50 unique permutations per second to finish exploring every permutation in the time the universe has existed. Modern computers are only predicted to hit 1 ExaFLOPs (1*10^18 floating point operations per second) by 2019.
No human shuffler will explore that range of possibilities either, and you are, I believe (at the most basic level) simulating a human shuffling, correct? Would you find it likely that a croupier could shuffle an incrementally ordered deck into decreasing order in one shuffle? To split the deck with even ranks before odd, in one shuffle?
I don't find it unacceptable to limit yourself to a (albeit extremely) small subsection of that phase space (2^48 possible random numbers) in each shuffle, so long as you don't continuously seed the same way etc.
There are exactly 52 factorial (expressed in shorthand as 52!) possible orderings of the cards in a 52-card deck. This is approximately 8×1067 possible orderings or specifically: 80,658,175,170,943,878,571,660,636,856,403,766,975,289,505,440,883,277,824,000,000,000,000.
The magnitude of this number means that it is exceedingly improbable that two randomly selected, truly randomized decks, will ever, even in the history of the Universe, be the same. However, while the exact sequence of all cards in a randomized deck is unpredictable, it may be possible to make some probabilistic predictions about a deck that is not sufficiently randomized.
~Wikipedia
Also, it's worth noting that Bayer & Diaconis in 1992 proved it only takes 7 good shuffles to properly randomize a deck, here is the section on it from wikipedia which has plenty links to papers discussing this.

Random Number Generation without repetition in Java

I would like to get clarifications on Pseudo Random Number generation.
My questions are:
Is there any chance for getting repeated numbers in Pseudo Random Number Generation?
When i googled i found true random number generation. Can i get some algorithms for true random number generation, so that i can use it with
SecureRandom.getInstance(String algorithm)
Please give guidance with priority given to security.

1) Yes, you can generally have repeated numbers in a PRNG. Actually, if you apply the pigeon hole principle, the proof is quite straightforward (ie, suppose you have a PRNG on the set of 32-bit unsigned integers; if you generate more than 2^32 pseudo random numbers, you will certainly have at least one number generated at least 2 times; in practice, that would happen way faster; usually the algorithms for PRNGs will cycle through a sequence, and you have a way to calculate or estimate the size of that cycle, at the end of which every single number will start repeating, and the image of the algorithm is usually way, way smaller than the set from which you take your numbers).
If you need non-repeated numbers (since security seems to be a concern for you, note that this is less secure than a sequence of (pseudo) random numbers in which you allow repeated numbers!!!), you can do as follows:
class NonRepeatedPRNG {
private final Random rnd = new Random();
private final Set<Integer> set = new HashSet<>();
public int nextInt() {
for (;;) {
final int r = rnd.nextInt();
if (set.add(r)) return r;
}
}
}
Note that the nextInt method defined above may never return! Use with caution.
2) No, there's no such thing as an "algorithm for true random number generation", since an algorithm is something known, that you control and can predict (ie, just run it and you have the output; you know exactly its output the next time you run it with the same initial conditions), while a true RNG is completely unpredictable by definition.
For most common non security-related applications (ie, scientific calculations, games, etc), a PRNG will suffice. If security is a concern (ie, you want random numbers for crypto), then a CSPRNG (cryptographycally secure PRNG) will suffice.
If you have an application that cannot work without true randomness, I'm really curious to know more about it.

Yes, any random number generator can repeat. There are three general solutions to the non-duplicate random number problem:
If you want a few numbers from a large range then pick one and reject
it if it is a duplicate. If the range is large, then this won't cause
too many repeated attempts.
If you want a lot of numbers from a small range, then set out all the numbers in an
array and shuffle the array. The Fisher-Yates algorithm is standard for array
shuffling. Take the random numbers in sequence from the shuffled array.
If you want a lot of numbers from a large range then use an appropriately sized
encryption algorithm. E.g. for 64 bit numbers use DES and encrypt 0, 1, 2, 3, ...
in sequence. The output is guaranteed unique because encryption is reversible.

Pseudo RNGs can repeat themselves, but True RNGs can also repeat themselves - if they never repeated themselves they wouldn't be random.
A good PRNG once seeded with some (~128 bit) real entropy is practically indistinguishable from a true RNG. You certainly won't get noticeably more collisions or repetitions than with a true RNG.
Therefore you are unlikely to ever need a true random number generator, but if you do check out the HTTP API at random.org. Their API is backed by a true random source. The randomness comes from atmospheric noise.

If a PRNG or RNG never repeated numbers, it would be... really predictable, actually! Imagine a PRNG over the numbers 1 to 8. You see it print out 2, 5, 7, 3, 8, 4, 6. If the PRNG tried its hardest not to repeat itself, now you know the next number is going to be 1 - that's not random at all anymore!
So PRNGs and RNGs produce random output with repetition by default. If you don't want repetition, you should use a shuffling algorithm like the Fisher-Yates Shuffle ( http://en.wikipedia.org/wiki/Fisher%E2%80%93Yates_shuffle ) to randomly shuffle an array of the numbers you want, in random order.
Also, if you need a source of random number generation for cryptographic purposes, seek out a provider of cryptographic PRNGs for your language. As long as it's cryptographically strong it should be fine - A true RNG is a lot more expensive (or demands latency, such as using random.org) and not usually needed.

Reimplementing mkpasswd

On Linux I am used to using mkpasswd to generate random passwords to use, on OS X however I don't have this command. Instead of sshing in to my vps every time, I wanted to re implement it using Java. What I have done is pick at random 4 lower case letters, 2 upper case letters, 2 symbols (/ . , etc) and 2 numbers. Then I create a vector and shuffle that too.
Do you think this is good enough randomization?

If you use java.security.SecureRandom instead of java.util.Random then it's probably secure. SecureRandom provides a "cryptographically strong pseudo-random number generator (PRNG)". I.e. it ensures that the seed cannot easily be guessed and that the numbers generated have high entropy.

yes, it is. If you are using java.util.Random:
An instance of this class is used to generate a stream of pseudorandom numbers. The class uses a 48-bit seed, which is modified using a linear congruential formula. (See Donald Knuth, The Art of Computer Programming, Volume 2, Section 3.2.1.)
The algorithms implemented by class Random use a protected utility method that on each invocation can supply up to 32 pseudorandomly generated bits.
EDIT
in response to a comment:
/**
* Creates a new random number generator. This constructor sets
* the seed of the random number generator to a value very likely
* to be distinct from any other invocation of this constructor.
*/
public Random() {
this(++seedUniquifier + System.nanoTime());
}
private static volatile long seedUniquifier = 8682522807148012L;

There is a similar pwgen command available in the Mac Ports.

Depends on where your entropy comes from. Using rand() or similar functions that your particular language comes with may not be secure.
On OSX you can use /dev/random I think.

It might be OK, but you should allow some randomization in password lengths perhaps.
If your program became popular it would become a weakness that the password length was public knowledge. Also randomize the exact ratio of lowercase:uppercase:symbols:numbers a little.

Why not just compile mkpasswd on your OS X host?