I have a function which is expected to randomly return one value from a collection of values. Is there a good way to test this random behavior with unit-test tools like JUnit etc.?
In situations that called for a lot of unit testing of code that normally behaves randomly, I've sometimes wrapped a stream of results from a java.util.Random in an Iterable<Integer>. The advantage is that, during unit testing, I can call the same method with an ArrayList<Integer> and get completely predictable behavior.
No. By definition, the result is (or should be) indeterminate, so the usual concept of "expected result" doesn't apply.
However, you could write a fairly simple test using a statistical approach, perhaps that when called n times, the set (unique) of values returned is at least .75 n, or something similar.
Another approach may be to defer the "randomness" to a trusted and sufficient implementation, such as an algorithm based on Math.random(), meaning your unit test would not have to test the randomness, rather just the functionality.
java.util.Random can probably be relied upon to be 'sufficiently random' for your purposes, so I assume what you're trying to test is that you get a proper random distribution among the items in your collection. It's true that there are a number of ways you could select an item out of a collection based on a random number and still end up biasing the results. For instance, iterating across the collection and using a random check at each stage will bias towards earlier items in the list.
If you want to test that your function actually produces random results you're going to need to find a statistical analysis toolkit which can do that. I'd suggest you fill up collections of various sizes with integer sequences, and then run tests of your random fetching code against those collections. You can feed the fetched values into the statistical analysis to determine if they're random or biased, and since they are linear sequences, the result should imply the same property for your fetching code as a whole.
The standard way of testing random values is to generate a few thousand of them, enumerate how many of each you get, calculate the chi-square statistic of the data set, then the incomplete gamma function will give you the probability of that distribution occurring at random. If that probability is too close to 0, it is likely that your RNG is biased.
The classic example of this is the "dieharder" test suite. You might also check out the test code in my http://github.com/lcrocker/ojrandlib.
Related
How can a method be tested with random input in a systematic way? What I mean by this is, if the input data for the units being tested changes each run, how can you effectively find out which values caused the failure and retest those same values after the code has been change? For example the function int[] foo(int[] data) and it can be checked if the return value is correct or not, after writing test cases for "hard coded inputs" (e.g. {1,2,3}) how can random input be tested? If it's random each time, any errors wont be reproducible. I could print the random input to the screen, but that would become messy for each function. Any suggestions?
Also, is it still unit testing if the whole program is being tested? For example calling the constructor and all methods of class in one #Test? The class only has one public method to begin with so this can't really be avoided, but I'm wondering if JUnit is the best tool for it.
How can a method be tested with random input in a systematic way?
In the general case and in the most strict sense, you cannot.
By "strict sense" I mean: validating the correctness of the output no matter which input is presented.
Assuming it would be possible, 'strictness' imply that your test case can compute the result of the function on each and every (random) input. Which means you will need to write a piece of code that replicates the method to be tested - theoretically possible, but leading to a paradoxical situation:
assume you find multiple bugs in the method under test. What is the cheapest way to correct it? Of course, substituting the method code with the testing code
and now the tester (the author of the currently implemented method) needs to... what?... write another "incarnation" of the function in order to test her own implementation?
However, "fuzzing" is still a valid method: except that it is never to be taken in the strict sense; the tests expects the results to expose certain traits/invariants/etc, something that can be defined/checked not matter what the input is. For example, "the method never throws", or "the returned array has the same length as the input array (or double the size, or all elements are odd, or whatever" or "the result is alway a proper HTML page which passes the W3C markup validator".
tested with random input in a systematic way?
You almost have an oxymoron here, mate, like "honest politician" or "aeroplane-safe 2016-made Galaxy 7 Note". If testing is "systematic" it means "there is a (rule) system that governs the way the tests are conducted" - almost quite opposite to "random input".
The trick to reconcile the two: you still have a (rule-based) system to categorize your input (e.g. equivalence partitioning ), except that instead of taking a certain (constant) representative for your categories, you pick the representatives at random. I'm going to repeat this: inside each categories, you pick your representative at random; (as opposed to "pick a random input and see which category it belongs to").
How's this useful? Not much added value, because according to the system/equivalence, picking a representative is as good as picking any other.
Sorry my QA mate, you don't get off-the-hook in regards with your responsibility as a tester to design and plan for the tests (no matter if you use random techniques in generating your input).
If it's random each time, any errors wont be reproducible. I could print the random input to the screen, but that would become messy for each function. Any suggestions?
This is a frivolous reason to avoid random input if random input is deemed necessary: just use some tools to visually organize the testing logs if the simple flowing text format is so hard to read.
E.g. output your test log formatted as JSON with a certain structure and use/write a visualisation tool to represent/explore/fold/unfold it in such a way that human exploration is not a pain in the nether back part of the body.
If your job is to automate the testing, it is supposed you are able to code, isn't it?
Also, is it still unit testing if the whole program is being tested?
"Whole program is being tested" exactly how? What is the specific goal of this "whole system testing"?
There is a distinction between "unit testing" (even a more-than-comprehensive 10000% coverage unit testing) and "functional testing" or "integration testing" or "performance testing" or "usability testing" or "localization testing" (this including the gra'ma' of the UI/error messages) - all the latter belonging to "whole program testing" without being unit testing.
Hint: in defining the type of testing, the specific goal one has in mind when designing/executing test takes precedence over the means used in testing; I've seen testing performed manually using GUI test harnesses, in which the testers where manually writing values to unit test the SDK underneath.
On the other side, there are categories of non-unit testing which may make use of unit testing techniques (e.g. a Web or REST Service - wrap and present it as a proxy-function API and then you can write your tests using JUnit/TestBG or whatever unit test framework you fancy. And yet, you are doing functional or integration testing).
Property-based testing could be a solution for you. Basically, the idea is to have a framework generating all sorts of input data (random, edge cases) which is then fed into your implementation.
It's true that with random data you may end up with test cases behaving differently with every run. But at least the test framework usually would show you which input is used when tests are failing and you can have a further look into the reasons for failing. It's not a guarantee that your method will work in 100% of the cases, but at least you get some coverage and it's still better than nothing.
Typically, such a framework also allows you to restrict the generated data to a set which makes sense for your implementation. Or you can implement your own generator providing data for your tests.
For Java there is e.g. JUnit-Quickcheck which integrates with JUnit: http://pholser.github.io/junit-quickcheck/site/0.6.1/
A lot has already been written about the differences between unit tests/integration tests etc. Maybe have a look here: What is the difference between integration and unit tests?
How does one go about determining the different types of sorting algorithms without being able to look at code?
The scenario is that our lecturer showed a .jar file with 4 sorting methods in it (sortA, sortB...) that we can call to sort a vector.
So, as far as I know, we can only measure and compare the times for the sorts. What's the best method to determine the sorts?
The main issue is that the times for the sorts don't take very long (differing by ~1000ms) to begin with, so comparing them by that isn't really on option, and so far all the data sets I've used (ascending, descending, nearly sorted) haven't really been giving much variation in the sort time.
I would create a data structure and do the following:
Sort the data on paper using known sorting algorithms and document the expected behavior on paper. Be very specific on what happens to your data after each pass.
Run a test on debug mode and step through the sorting process.
Observe how the elements are being sorted and compare it with your predictions (notes) obtained in step 1.
I think this should work. Basically, you are using the Scientific Method to determine which algorithm is being used in a given method. This way, you don't have to resort to "cheating" by decompiling code or rely in imprecise methodologies like using execution time. The three step process I outlined rely in solid, empirical data to arrive to your conclusions.
I'm implementing a genetic algorithm using Java programming language. As you know, there are some random events in the algorithm like roullete selection, crossover, mutation, etc. In order to generate a better probability distribution among these events, which approach should be better, to use a unique Random object or create a separate Random object for each event?
Use a single object. Random number generators are designed to have long periods -- using the same seeded instance, you get a good sequence of random digits out. If you're constantly creating and destroying them, you're only getting however much randomness there is in the seeding process, which may even be none. Imagine what happens if your RNG is seeded from the system clock, and you're doing it thousands of times per second, for instance.
Use a single Random stored globally, and refer to it everywhere you need randomness. What's more, initialize it with a known seed, and write this seed to a file along with the results of the genetic algorithm.
In addition to the benefits mentioned by deong, this lets you rerun the whole program exactly if you find some interesting outputs. It can be extremely frustrating with genetic algorithms to see an interesting result and then be unable to reproduce it, because it was a rare outcome. If you have the seed, you can just rerun the program deterministically.
If you want each run to use a new seed you can do it like this:
long seed = new Random().nextLong();
log("Seed for the current run is: " + seed);
Global.setRandom(new Random(seed));
That way you get a new random seed everytime, but you can still reconstruct a given run if you need to.
Note that the Random object should not be shared between two different runs. At the start of each run, you should create a new random object and make a note of the seed.
So I have some C code which calculate some results based on the number generated by srand(). If I use the same seed number, the result will always be the same.
Now I have an Android app load these C code via JNI. However, the results become different although the same seed number is being used. I have double checked the seed number to make sure it is the same. However, since both the Android program and the native code are pretty complicated, I am having a hard time to figure out what is causing this problem.
What I am sure is, we did not use function in the java program to generate random numbers. So presumably srand() is not called with a different seed number every time. Can other functions in Java or C change the random number generated by srand()?
Thanks!
Update:
I guess my question was a little confusing. To clarify, the results I am comparing are from the same platform, but different runs. The c code use rand() to get a number calculate a result based on that. So if the seed number of srand() is always the same, the number get by rand() should be the same and hence the results should be the same. but somehow even I use the same seed for srand(), the rand() give me different number... Any thought on that?
There are many different types of random number generators, and they are not all guaranteed to be the same from platform to platform. If having a cross platform 100% predictable solution is necessary for your project, you'll probably have to write your own.
It's really not as bad as it may sound...
I'd recommend looking up random number generation such as the Mersenne Twister algorithm (which is what I use in my projects), and write a small block of code that you can share amongst all your projects. This also gives you the benefit of being able to have multiple generators with varying seeds, which comes in really useful for something like a puzzle game, where you might want a predictably random set based on a specific seed to generate your puzzle, but another clock seeded generator for randomizing special FX or other game elements.
The pseudo-random algorithm implemented by rand() is determined by the C library, and there is no standard algorithm for it. You are absolutely not guaranteed to get the same sequence of numbers from one implementation to the next, and it sounds like the Android implementation differs from your development environment. If you need a predictable sequence across platforms, you should implement your own random number generator.
I'm relatively new to JUnit, and I was writing a few of my first tests today. For some particular method, I wanted to pass random values (all of which are within the correct range). If the method fails for any reason, I want to know which value caused it to fail. So what's the recommended way of doing this?
(Or is it bad to use random values in JUnit tests?)
If you really want to use random values then simply place the value in use in the textual part of the assert methods. Then if an assert is blown the input value will be present and you can investigate why it had a problem.
This is Fuzz Testing and is a powerful technique but is most useful when you don't have the available source code or when testing a system with complex internal state and multiple interactions.
A more useful type of testing for you may be white box testing where test inputs are deliberately selected to cover the various classes of input you might get. JTest appears to be an automated tool for this in java. MS Research supplies PEX for c#).
Simply using a coverage tool and validating that you cover the relevant paths is normally sufficient if doing it by hand, though the boundary cases supplied by the automated tools are often instructive.
You can try to use: http://www.openfuture.de/Log4Unit/ for logging, but I would recommend against random values for unit tests as they are supposed to be repeated. If you want to test a lot of values just use a for loop and some modifications to the index value, which is easily repeated.
If you think about it there is really no situation where it would be more beneficial to use random values than "hard coded" ones. If you want a good spread over a value range you can use a function or use random values with a fixed seed (to get the same numbers).
If a test fails you want to be able to fix it and run the test again. That's the problem with random numbers in unit tests.
Just report the actual and expected values in the "diagnostic message" parameter of your assertions. That's the common practice for JUnit tests, and the "helper" assert methods tend to do this by default, e.g. assertEquals will say something like "expected 6 and got 7".
Random values are OK, as long as you can ensure that the random range constitutes an equivalence class for the code under test.
Have you tried using the 'assertThat' methods and Hamcrest Matchers that are part of JUnit 4.4+? Check out the README [1] and search for 'assertThat'.
I've become quite fond of the much more semantic look of the code and the failure messages are much more informative.
[1] http://junit.sourceforge.net/README.html
I would propose parameterized test cases. so you can use random values (in the Data method) and it's "logged" in your runner, if any method will fail.
If your unit test fails for any reason, you will see a red traffic light in your test runner. The test runner will also show you which test method failed and the test runner's log reports more detail (the dumped stack trace, for example). Investigate that error, correct it, and your test should never fail again, unless you break the code.
Thus, I don't see the necessity for logging exceptions. You should fix any error (red traffic light) immediately.
Using random values can be quite dangerous if you cannot guarantee that your generation of these values is free of errors. Checking the boundary conditions might be more useful.
You can have repeatable random values by providing a constant seed to the random number generator. In Java create a java.util.Random with a fixed seen and pass to the class as a constructor parameter (dependency injection). Like this:
new ClassUnderTest(new Random(123L));
Depending on what you are testing, you might also separate the generating of the random values from their use. If you have class X which takes random values from range 1 to 10, then you can test it by passing it the edge values 1 and 10, and some value from the middle such as 4. Then you need another test for the producer of those random values. For example give it a java.util.Random with a fixed seed and generate 100 values, and check that all of them are within the allowed range.