While unit testing some methods, there can be some scenarios where value of some parameters do not matter and can be any value.
For example in this piece of code:
public void method(String arg1, String arg2, int arg3){
if(arg1 == null) throw new NullPointerException("arg1 is null");
//some other code
}
unit testing the behavior that when arg1 is null then NPE must be thrown, the values of other arguments do not matter, they can be any value or be null.
So I wanted to document the fact that the values do not matter for the method under test.
I thought of following options:
Option 1: Define constants of ANY_XXX
I thought of explicitly creating constants ANY_STRING and ANY_INT, which contain a fixed value which documents that it can be any value and the method under test does not care about the actual value.
I can put all these constants in a single class called Any and reuse them across all test classes.
Option 2: Random values for ANY_XXX
This option seems a bit hacky to me as I have read somewhere that randomness should not be brought into test cases. But in this scenario this randomness will not be visible as the parameters will not create any side effect.
Which approach would be more suitable for better, readable tests?
UPDATE:
While I can use ANY_XXX approach by defining constants in Any class, but I am also thinking of generating ANY_XXX values with some constraints such as
Any.anyInteger().nonnegative();
Any.anyInteger().negative();
Any.anyString().thatStartsWith("ab");
I am thinking that maybe Hamcrest Matchers can be used for creating this chaining. But I am not sure if this approach is a good one. Similar methods for anyObject() are already provided by Mockito but those only work on Mocks and spies and not on normal objects. I want to achieve the same for normal objects for more readable tests.
Why I want to do this?
Suppose I have a class
class MyObject{
public MyObject(int param1, Object param2){
if(param1 < 0) throw new IllegalArgumentException();
if(param2 == null) throw new NullPointerException();
}
}
And now while writing tests for constructor
class MyObjectTest{
#Test(expected=NullPointerException.class)
public void testConstructor_ShouldThrowNullpointer_IfSecondParamIsNull(){
//emphasizing the fact that value of first parameter has no relationship with result, for better test readability
new MyObject(Any.anyInteger().nonnegative(), null);
}
}
I see both og them quite a lot
Personally I disagree that randomness should not be brought into tests. Using randomness to some degree should make your tests more robust, but not necessarily easier to read
If you go for the first approach I would not create a constants class, but rather pass the values (or nulls) directly, since then you see what you pass in without the need to have a look in another class - which should make your tests more readable. You can also easily modify your tests later if you need the other parameters later on
My preference is to build up a utility class of constants along with methods to help with the creation of the constant values for tests, e.g.:
public final class Values {
public static final int ANY_INT = randomInt(Integer.MIN_VALUE, Integer.MAX_VALUE);
public static final int ANY_POSITIVE_INT = randomInt(0, Integer.MAX_VALUE);
public static final String ANY_ISBN = randomIsbn();
// etc...
public static int randomInt(int min, int max) { /* omitted */ }
public static String randomIsbn() { /* omitted */ }
// etc...
}
Then I would use static imports to pull the constants and methods I needed for a particular test class.
I use the ANY_ constants only in situations where I do not care about the value, I find that they can make the intent of the test clearer, for example:
// when
service.fooBar(ANY_INT, ANY_INT, ANY_INT, ANY_INT, 5);
It's clear that the value 5 is of some significance - although it would be better as a local variable.
The utility methods can be used for adhoc generation of values when setting up tests, e.g.:
// given
final String isbn1 = randomIsbn();
final String isbn2 = randomIsbn();
final Book[] books = { new Book(isbn1), new Book(isbn2) };
// when
bookRepository.store(books);
Again, this can help to keep the test classes concerned about the tests themselves and less about data set up.
In addition to this I have also used a similar approach from domain objects. When you combine the two approaches it can be quite powerful. e.g.:
public final class Domain {
public static Book book() {
return new Book(randomIsbn());
}
// etc...
}
I've faced the same problem when i've started to write unit tests for my project and had to deal with numerous of arrays, lists, integer inputs, strings etc.
So I decided to use QuickCheck and create a generator util class.
Using Generators in this library, you can generate primitive data types and String easily.
For example, when you want to Generate an integer; simply use IntegerGenerator class.You can define maximum and minimum values in the constructor of the generator.You can also use CombinedGeneratorSamples class to generate data structures like lists, maps and arrays.
Another feature of this library is implementing Generator interface for custom class generators.
You're overthinking and creating unnecessary barriers for your project :
if you want to document your method, do it with words! that's why the Javadoc is here for
if you want to test your method with "any positive int" then just call it with a couple different positive ints. In your case, ANY does not mean testing every possible integer value
if you want to test your method with "a string that starts with ab", call it with "abcd", then "abefgh" and just add a comment on the test method !
Sometimes we are so caught with frameworks and good practices that it takes common sense away.
In the end : most readable = simplest
How about using a caller method for the actual method.
//This is the actual method that needs to be tested
public void theMethod(String arg1, String arg2, int arg3, float arg4 ){
}
Create a caller method that calls the method with the required parameters and default(or null) values for the rest of the params and run your test case on this caller method
//The caller method
#Test
public void invokeTheMethod(String param1){
theMethod(param1, "", 0, 0.0F); //Pass in some default values or even null
}
Although you will have to be pretty sure that passing default values on theMethod(...) for the other parameters wont cause any NPE.
i see 3 options:
never pass nulls, forbid your team passing nulls. nulls are evil. passing null should be an exception, not a rule
simply use annotation in production code: #NotNull or sth like that. if u use lombok, this annotation will also do the actual validation
and if u really have to do it in tests then simply create a test with proper name:
static final String ANY_STRING = "whatever";
#Test
public void should_throw_NPE_when_first_parameter_is_null() {
object.method(null, ANY_STRING, ANY_STRING); //use catch-exception or junit's expected
}
If you're willing to give JUnitParams' framework a go, you could parametrize your tests specifying meaningful names to your parameters:
#Test
#Parameters({
"17, M",
"2212312, M" })
public void shouldCreateMalePerson(int ageIsNotRelevant, String sex) throws Exception {
assertTrue(new Person(ageIsNotRelevant, sex).isMale());
}
I'm always in favor of the constants approach. The reason is that I believe it gets more readable than chaining several matchers.
Instead of your example:
class MyObjectTest{
#Test(expected=NullPointerException.class)
public void testConstructor_ShouldThrowNullpointer_IfSecondParamIsNull(){
new MyObject(Any.anyInteger().nonnegative(), null);
}
}
I would d:
class MyObjectTest{
private static final int SOME_NON_NEGATIVE_INTEGER = 5;
#Test(expected=NullPointerException.class)
public void testConstructor_ShouldThrowNullpointer_IfSecondParamIsNull(){
new MyObject(SOME_NON_NEGATIVE_INTEGER, null);
}
}
Also, I prefer the use of 'SOME' over 'ANY', but that's also a matter of personal taste.
If you're considering testing the constructor with a number of different variants as you mentioned (nonNegative(), negative(), thatStartsWith(), etc.), I would that instead you write parameterized tests. I recommend JUnitParams for that, here's how I'd do it:
#RunWith(JUnitParamRunner.class)
class MyObjectTest {
#Test(expected = NullPointerException.class)
#Parameters({"-4000", "-1", "0", "1", "5", "10000"})
public void testConstructor_ShouldThrowNullpointer_IfSecondParamIsNull(int i){
new MyObject(i, null);
}
...
}
I suggest you go with constant values for those parameters which may be arbitrary. Adding randomness makes your test runs not repeatable. Even if parameter values "don't matter" here, actually the only "interesting" case is when a test fails, and with random behavior added in, you might not be able to reproduce the error easily. Also, simpler solutions are often better, and easier to maintain: using a constant is certainly simpler than using random numbers.
Of course if you go with constant values, you could put these values in static final fields, but you could also put them in methods, with names such as arbitraryInt() (returning e.g. 0) and so on. I find the syntax with methods cleaner than with constants, as it resembles Mockito's any() matchers. It also allows you to replace the behavior more easily in case you need to add more complexity later on.
In case you want to indicate that a parameter doesn't matter and the parameter is an object (not primitive type), you can also pass empty mocks, like so: someMethod(null, mock(MyClass.class)). This conveys to a person reading the code that the second parameter can be "anything", since a newly created mock has only very basic behavior. It also doesn't force you to create your own methods for returning "arbitrary" values. The downside is it doesn't work for primitive types or for classes which can't be mocked, e.g. final classes like String.
Ok.... I see a big Problem with you approach!
The other value doesn't matter? Who guarantees this? The Writer of the Test, the writer of the Code? What if you have a Method, which throws some unrelated Exception if the first Parameter is exactly 1000000 even if the second parameter is NULL ?
You have to formulate your Test-Cases: What is the Test-Specification... What do you want to proof? Is it:
In some cases if the first parameter is some arbitrary value and the second is null, this method should throw a NullPointerException
For any possible first Input value, if the second value is NULL the method should always throw a NullPointerException
If you want to test the first case - your approach is ok. Use a constant, a random value, a Builder... whatever you like.
But if your specification actually requires the 2nd condition all of your presented solutions are not up for the task, since they only test some arbitrary value. A good test should still be valid if the programmer changes some code in the method. This means the right way to test this method would be a whole series of Testcases, testing all corner-cases as with all other methods. So each critical value which can lead to a different execution-path should be checked - or you need a testsuite which checks for code-path completeness...
Otherwise your test is just bogus and there to look pretty...
Related
This might be a trivial question, but I need some clarification...
There is a book called Clean Code that says that our methods should be small, preferably up to 5-10 lines long. In order to achieve that we need to split our methods into smaller ones.
For instance, we may have someMethod() shown below. Let's say, modification of 'Example' takes 5 lines and I decide to move it into a separate method, modify 'Example' there and return it back to someMethod(). By doing this, someMethod() becomes smaller and easier to read. That's good, but there is a thing called "side effects" which says that we shouldn't pass an object to another method and modify it there. At least, I was told that it's a bad idea ) But I haven't seen anything prohibiting me from doing so in Clean Code.
public Example someMethod() {
// ... different lines here
Example example = new Example();
example = doSomethingHere(example, param1, param2, ...);
// ... different lines here
return example;
}
private Example doSomethingHere(Example example, 'some additional params here') {
// ... modify example's fields here ...
return example;
}
So, am I allowed to split the methods this way or such a side effect is prohibited and instead I should deal with a rather long-line method that definitely breaks Clean Code's rules talking about short methods?
UPDATED (more specific name for the sub-method)
public Example someMethod() {
// ... different lines here
Example example = new Example();
example = setExampleFields(example, param1, param2, ...);
// ... different lines here
return example;
}
private Example setExampleFields(Example example, 'some additional params here') {
// ... modify example's fields here ...
return example;
}
As JB Nizet commented, it's not actually a side effect if it's the only effect, so any blanket statement that "all side effects are bad" doesn't apply here.
Still, the main question stands: Is this (side) effect okay?
Talking about the principles first, side effects are, in general, dangerous for two reasons:
they make concurrency more difficult
they obscure/hide information
In your example, there is some information that is hidden. You could call this a potential side effect, and it can be exposed with a question: "Does this doSomethingHere method create a new object or modify the one I pass in?"
The answer is important, and even more so if it's a public method.
The answer should be trivial to find by reading the doSomethingHere method, especially if you're keeping your methods 'clean', but the information is nonetheless hidden/obscured.
In this specific case, I would make doSomethingHere return void. That way there's no potential for people to think that you've created a new object.
This is just a personal approach - I'm sure that plenty of developers say you should return the object you modify.
Alternatively, you can pick a 'good' method name. "modifyExampleInPlace" or "changeSomeFieldsInPlace" are pretty safe names for your specific example, imo.
we shouldn't pass an object to another method and modify it there.
Who says that? That is actually a good practice in order to split your function in a way that forms a "recipe" and have specific functions that know exactly how to populate your object properly.
What is not recommended (and probably the source where you got your recommendation misunderstood this rule) is defining a public API and modify the arguments. Users appreciate not having their arguments modified as it leads to less surprises. An example of that is passing arrays as arguments to methods.
When you define an object and pass it to an other method, method itself can modify the content of the object therein which may be unwanted in some cases. This is because you pass the reference(shallow copy) of the object to that method and method can modify that object.For example when you pass an Array, Arrays are objects, to a method, method can change the content of the Array which may not be what the caller method expects.
public static void main(String[] args){
int[] arr= {1,2,3,4};
y(arr);
//After the method arr is changed
}
public void y(int[] comingArray){
comingArray[0] = 10;
}
To make sure the values of Array cannot be changed, deep copy of the Array should be sent to method which is another story
However this is not the case when you use primite types(int, float etc.)
public static void main(String[] args){
int a= 1
y(a);
//After the method a is not changed
}
public void y(int comingInt){
comingInt = 5;
}
Due to the nature of the Objects, you should be carefulTo learn more about shallow copy and deep copy https://www.cs.utexas.edu/~scottm/cs307/handouts/deepCopying.htm
I have to deal with a legacy application that has no tests. So before I begin refactoring I want to make sure everything works as it is.
Now imagine the following situation:
public SomeObject doSomething(final OtherObject x, final String something) {
if(x == null) throw new RuntimeException("x may not be null!");
...
}
Now I want to test that null check, so to be sure it works and I don't lose it once I refactor.
So I did this
#Test(expected = RuntimeException.class)
public void ifOtherObjectIsNullExpectRuntimeException() {
myTestObject.doSomething(null, "testString");
}
Now, this works of course.
But instead of "testString" I'd like to pass in a random String.
So I tried with:
#Test(expected = RuntimeException.class)
public void ifOtherObjectIsNullExpectRuntimeException() {
myTestObject.doSomething(null, Mockito.anyString());
}
But this is not allowed., as I get org.mockito.exceptions.misusing.InvalidUseOfMatchersException:
... You cannot use argument matchers outside of verifications or stubbing
I do understand the meaning of this, but I wonder whether I can still manage to do what I want without parameterizing my test or the like.
The only libraries I may use are Junit, AssertJ, Mockito and Powermock.
Any ideas?
Tests should be deterministic. Using random values in a test makes it difficult to reproduce behavior when debuging a failed test. I suggest that you just create a String constant for the test such as "abcdefg".
Well, like Mockito is trying to tell you via that exception, that's not really how you'd use anyString. Such methods are only to be used by mocks.
So, why not try testing with an actual random string? My personal favorite in such a scenario: java.util.UUID.randomUUID().toString(). This will virtually always generate a brand new string that has never been used for your test before.
I'd also like to add that if you are writing tests for your SomeObject class that you should avoid mocking SomeObject's behavior. From your example, you weren't exactly doing that, but it looked like you might be going down that route. Mock the dependencies of the implementation you're trying to test, not the implementation itself! This is very important; otherwise you aren't actually testing anything.
You are mixing up concepts here.
All those "mocking" helpers like anyString() are meant to be used when configuring a mock object.
But when you check your testing code:
#Test(expected = RuntimeException.class)
public void ifOtherObjectIsNullExpectRuntimeException() {
myTestObject.doSomething(null, "testString");
}
you will find: there is absolutely no mocking involved for this test. You simply can't use those Mockito calls in that place; because "there is no Mockito" in that place.
And just for the record - no need to go overboard here anyway. Your logic is very clear here: when the first argument is null, then you throw that exception. Thus it really doesn't matter at all what comes in as second argument. So thinking for an hour how to test null with any second argument is, well, in my eyes: waste of your time.
Final hint: there is java.lang.Objects
And that class has a nice check for null, so my production code only looks like
public SomeObject doSomething(final OtherObject x, final String something) {
Objects.requireNonNull(otherObject, "otherObject must not be null");
Objects.requireNonNull(something, "something must not be null");
Only difference there: requires... throws NullPointerExceptions
Final finally: some people suggest to put final on every parameter; but I wouldn't do that. It adds no value in 99% of all cases. It just means that you have more code to read; for no good reasons. But that is a question of style.
EDIT on the comment about having a test to check for potential future changes: you shouldn't do that:
To a certain degree, how your input is verified is an implementation detail. You don't test for implementation details. In other words:
Your method has a certain contract (that you, for example specify informally by writing a javadoc that says "throws NPE on null input"). Your tests should verify exactly that current contract. And the contract is: throws if first argument is null.
And maybe another point of view; as I still think you are wasting your time here! You should make sure that all your interfaces are clear, easy to understand, and easy to use. That they allow users of your code to do the right thing easily; and prevent him from doing wrong things. That is what you should focus on - the quality of your interfaces as a whole!
So instead of worrying how you could write a test for potential future changes; just make sure that your code base is overall consistent.
Well i do not have much knowledge of mockito but you can always create your own random string generator. maybe that can work and u can modify more types of inputs in it
I'm still coming to terms with unit testing certain things and I have a question about my JUnit test for a relatively simple class:
public class Kite {
private List<String> strings = new ArrayList<String>();
public void addString(String string) { //... }
public void removeString(String string) { //... }
public List<String> getStrings() { //... }
public int getNumStrings() { //... }
}
In this case I would like to test all four methods. However the tests, now that I've written them, are all very similar. They (excluding the remove) all follow the basic structure of add a String to a Kite and then checking the number of Strings in the Kite object.
Is there a better way to test these "CRUD" types of methods?
Do I need to test them?
It's better to be more specific in your testing. For addString(), you want to test that:
The string you added is present in the collection.
No other strings were added to the collection as a side effect.
If you pass null to addString(), an IllegalArgumentException (or whatever the behavior should be) is thrown
If you pass the same string in twice, the behavior is what you want (could be an IllegalArgumentException, could be a no-op)
get the idea? You want to add tests for edge cases, as well as the normal behavior (sometimes called the "happy path"). Think about your tests in terms of possible inputs, possible outputs, and code paths that can be taken.
I would think about writing one test that models a transaction:
Check the pre-condition.
Perform add operation.
Check post-condition.
Rollback add operation with remove.
Make sure that Kite is in original state.
All those methods will be tested that way.
I would like to test an API, which received one argument and returns a set. The test invokes the API with an argument and checks if the returned set contains expected values.
Suppose, I have to test the API with arguments arg1, arg2, and arg3 and check if values a, b, c appear in the returned set. That is, my test case looks as follows:
invoke the API with arg1 and check if a,
b, c appear in the returned set.
invoke the API with arg2 and check if a,
b, c appear in the returned set.
invoke the API with arg3 and check if a,
b, c appear in the returned set.
How to develop this test case with Junit 4 ? What if I have to add arg4 ? What if I have to check if value d appear in the returned set ? Can I read the list of arguments and expected values from the configuration?
Fluent assertions
First of all, use FEST-Assertions library to introduce pleasantly looking assertions with meaningful error messages:
assertThat(method(arg1)).containsExactly(a, b, c);
assertThat(method(arg2)).containsExactly(a, b, c);
assertThat(method(arg3)).containsExactly(a, b, c);
The BDD way
But I understand your question is not about the syntax, but about methodology: what should you do if arg4 needs to be tested? Well, if arg1 through arg4 have a different semantic meaning, I would advice you to have a separate test for each argument. Very verbose, but also extremely readable (pseudocode):
#Test
public void shouldReturnAbcWhenSomeArgumentUsed() {
//given
Object arg = arg1;
//when
Set<Object> result = method(arg);
//then
assertThat(result).containsExactly(a, b, c);
}
..and repeat this for every test. The key part is: method name should represent the meaning of an argument, what does this method actually test, what do you expect, what is the scenario?
Consider testing isEven method. I would recommend the following tests:
shouldReturnTrueForZero
shouldReturnTrueForSmallPositiveEvenNumber
shouldReturnTrueForLargePositiveEvenNumber
shouldReturnFalseForSmallPositiveOddNumber
shouldReturnFalseForLargePositiveOddNumber
... and mirror the tests for negative numbers
Each test represent a slightly different, well defined scenario. On the other hand you might generate literally thousands of shouldReturnFalseWhen227, but what is the value of such a test suite, except it's large? Test semantically different arguments and corner cases, defining precisesly what case is being tested.
Parameterized way
If you really want to have just a single generic test method, Parameterized runner is the way to go. I think the example is self-explanatory. NB: you can parameterize expected values as well.
#RunWith(value = Parameterized.class)
public class JunitTest6 {
private Object arg;
public JunitTest6(Object arg) {
this.arg = arg;
}
#Parameterized.Parameters
public static Collection<Object[]> data() {
return Arrays.asList(
new Object[][]{
{arg1},
{arg2},
{arg3}
});
}
#Test
public void testMethod() {
assertThat(method(arg)).containsExcatly(a, b, c);
}
}
Based on this.
I would usually turn to Hamcrest for something like this -- it's a library for declaratively writing "matchers", and it plays very nicely with JUnit.
However, this question on SO points out that although this can be done with Hamcrest, a simpler way is just to use the containsAll method from java.util.Collection:
ArrayList<Integer> expected = new ArrayList<Integer>();
expected.add(1); expected.add(2); expected.add(3);
assertTrue(actual.containsAll(expected));
In terms of methodology:
The "agile" way
Tests need ongoing development and refactoring, just like production code. As well, principles like YAGNI ("you ain't gonna need it") apply too. If right now you only need to test a, b and c then I would start with an ordinary hard-coded unit test. If later your test cases start becoming repetitive then by all means consider how to refactor them.
Or maybe you are already at that point now, but to me the question doesn't seem to provide enough information to give a more specific suggestion about how to refactor the unit tests. Read tests from XML? Generate combinatorial test data? Parameterized runner (as per #Tomasz)? Maybe I just haven't understood the question well enough, but the problem as stated seems too abstract still.
assuming you want to test the method "method1" that take one parameter and return a set you would write:
Set result = method1(arg1);
assertTrue(result.contains(a));
assertTrue(result.contains(b));
assertTrue(result.contains(c));
But maybe the best would be to compare the set directly with it expected value:
Set expected = new HashSet();
expected.add(a);
expected.add(b);
expected.add(c);
assertEquals(expected, method1(arg1));
assertEquals(expected, method1(arg2));
assertEquals(expected, method1(arg3));
And of course don't hesiste to use loops to be more generic if needed.
Functions (side-effect free ones) are such a fundamental building block, but I don't know of a satisfying way of testing them in Java.
I'm looking for pointers to tricks that make testing them easier. Here's an example of what I want:
public void setUp() {
myObj = new MyObject(...);
}
// This is sooo 2009 and not what I want to write:
public void testThatSomeInputGivesExpectedOutput () {
assertEquals(expectedOutput, myObj.myFunction(someInput);
assertEquals(expectedOtherOutput, myObj.myFunction(someOtherInput);
// I don't want to repeat/write the following checks to see
// that myFunction is behaving functionally.
assertEquals(expectedOutput, myObj.myFunction(someInput);
assertEquals(expectedOtherOutput, myObj.myFunction(someOtherInput);
}
// The following two tests are more in spirit of what I'd like
// to write, but they don't test that myFunction is functional:
public void testThatSomeInputGivesExpectedOutput () {
assertEquals(expectedOutput, myObj.myFunction(someInput);
}
public void testThatSomeOtherInputGivesExpectedOutput () {
assertEquals(expectedOtherOutput, myObj.myFunction(someOtherInput);
}
I'm looking for some annotation I can put on the test(s), MyObject or myFunction to make the test framework automatically repeat invocations to myFunction in all possible permutations for the given input/output combinations I've given, or some subset of the possible permutations in order to prove that the function is functional.
For example, above the (only) two possible permutations are:
myObj = new MyObject();
myObj.myFunction(someInput);
myObj.myFunction(someOtherInput);
and:
myObj = new MyObject();
myObj.myFunction(someOtherInput);
myObj.myFunction(someInput);
I should be able to only provide the input/output pairs (someInput, expectedOutput), and (someOtherInput, someOtherOutput), and the framework should do the rest.
I haven't used QuickCheck, but it seems like a non-solution. It is documented as a generator. I'm not looking for a way to generate inputs to my function, but rather a framework that lets me declaratively specify what part of my object is side-effect free and invoke my input/output specification using some permutation based on that declaration.
Update: I'm not looking to verify that nothing changes in the object, a memoizing function is a typical use-case for this kind of testing, and a memoizer actually changes its internal state. However, the output given some input always stays the same.
If you are trying to test that the functions are side-effect free, then calling with random arguments isn't really going to cut it. The same applies for a random sequence of calls with known arguments. Or pseudo-random, with random or fixed seeds. There's a good chance are that a (harmful) side-effect will only occur with any of the sequence of calls that your randomizer selects.
There is also a chance that the side-effects won't actually be visible in the outputs of any of the calls that you are making ... no matter what the inputs are. They side-effects could be on some other related objects that you didn't think to examine.
If you want to test this kind of thing, you really need to implement a "white-box" test where you look at the code and try and figure out what might cause (unwanted) side-effects and create test cases based on that knowledge. But I think that a better approach is careful manual code inspection, or using an automated static code analyser ... if you can find one that would do the job for you.
OTOH, if you already know that the functions are side-effect free, implementing randomized tests "just in case" is a bit of a waste of time, IMO.
I'm not quite sure I understand what you are asking, but it seems like Junit Theories (http://junit.sourceforge.net/doc/ReleaseNotes4.4.html#theories) could be an answer.
In this example, you could create a Map of key/value pairs (input/output) and call the method under test several times with values picked from the map. This will not prove, that the method is functional, but will increase the probability - which might be sufficient.
Here's a quick example of such an additional probably-functional test:
#Test public probablyFunctionalTestForMethodX() {
Map<Object, Object> inputOutputMap = initMap(); // this loads the input/output values
for (int i = 0; i < maxIterations; i++) {
Map.Entry test = pickAtRandom(inputOutputMap); // this picks a map enty randomly
assertEquals(test.getValue(), myObj.myFunction(test.getKey());
}
}
Problems with a higher complexity could be solved based on the Command pattern: You could wrap the test methods in command objects, add the command object to a list, shuffle the list and execute the commands (= the embedded tests) according to that list.
It sounds like you're attempting to test that invoking a particular method on a class doesn't modify any of its fields. This is a somewhat odd test case, but it's entirely possible to write a clear test for it. For other "side effects", like invoking other external methods, it's a bit harder. You could replace local references with test stubs and verify that they weren't invoked, but you still won't catch static method calls this way. Still, it's trivial to verify by inspection that you're not doing anything like that in your code, and sometimes that has to be good enough.
Here's one way to test that there are no side effects in a call:
public void test_MyFunction_hasNoSideEffects() {
MyClass systemUnderTest = makeMyClass();
MyClass copyOfOriginalState = systemUnderTest.clone();
systemUnderTest.myFunction();
assertEquals(systemUnderTest, copyOfOriginalState); //Test equals() method elsewhere
}
It's somewhat unusual to try to prove that a method is truly side effect free. Unit tests generally attempt to prove that a method behaves correctly and according to contract, but they're not meant to replace examining the code. It's generally a pretty easy exercise to check whether a method has any possible side effects. If your method never sets a field's value and never calls any non-functional methods, then it's functional.
Testing this at runtime is tricky. What might be more useful would be some sort of static analysis. Perhaps you could create a #Functional annotation, then write a program that would examine the classes of your program for such methods and check that they only invoke other #Functional methods and never assign to fields.
Randomly googling around, I found somebody's master's thesis on exactly this topic. Perhaps he has working code available.
Still, I will repeat that it is my advice that you focus your attention elsewhere. While you CAN mostly prove that a method has no side effects at all, it may be better in many cases to quickly verify this by visual inspection and focus the remainder of your time on other, more basic tests.
have a look at http://fitnesse.org/: it is used often for Acceptance Test but I found it is a easy way to run the same tests against huge amount of data
In junit you can write your own test runner. This code is not tested (I'm not sure if methods which get arguments will be recognized as test methods, maybe some more runner setup is needed?):
public class MyRunner extends BlockJUnit4ClassRunner {
#Override
protected Statement methodInvoker(final FrameworkMethod method, final Object test) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
Iterable<Object[]> permutations = getPermutations();
for (Object[] permutation : permutations) {
method.invokeExplosively(test, permutation[0], permutation[1]);
}
}
};
}
}
It should be only a matter of providing getPermutations() implementation. For example it can take data from some List<Object[]> field annotated with some custom annotation and produce all the permutations.
I think the term you're missing is "Parametrized Tests". However it seems to be more tedious in jUnit that in the .Net flavor. In NUnit, the following test executes 6 times with all combinations.
[Test]
public void MyTest(
[Values(1,2,3)] int x,
[Values("A","B")] string s)
{
...
}
For Java, your options seem to be:
JUnit supports this with version 4. However it's a lot of code (it seems, jUnit is adamant about test methods not taking parameters). This is the least invasive.
DDSteps, a jUnit plugin. See this video that takes values from appropriately named excel spreadsheet. You also need to write a mapper/fixture class that maps values from the spreadsheet into members of the fixture class, that are then used to invoke the SUT.
Finally, you have Fit/Fitnesse. It's as good as DDSteps, except for the fact that the input data is in HTML/Wiki form. You can paste from an excel sheet into Fitnesse and it formats it correctly at the push of a button. You need to write a fixture class here too.
Im afraid that I dont find the link anymore, but Junit 4 has some help functions to generate testdata. Its like:
public void testData() {
data = {2, 3, 4};
data = {3,4,5 };
...
return data;
}
Junit will then thest your methods will this data. But as I said, I cant' find the link anymore (forgot the keywords) for a detailed (and correct) example.