Develop Reference

Verify a temporary objects are created

I'm in a java context and am using Mockito (but I'm not bound to it) for basic mocking needs.
I have code like this
public class AuditInfoSerializer {
[..]
public Map<String, Object> doStuff(Object a) {
doOtherStuff("hello", new TempClass(someField, <someParams>));
doOtherStuff("world", new TempClass(someField, <otherParams>));
return getResult();
}
}
and in a test I want to verify that there are two instances of TempClass created with the correct set of parameters when I call the doStuff method.
Is this possible somehow?

You don't want to verify temporary data on the object under test. You want to mock dependencies and assert the object under test behavior : that is with this input you have this output.
Mock verifying is a trade off for methods to mock that return nothing but only produces side effect.
So use it only as you don't have the choice.
In your unit test, what you want is asserting what the method to test returns that is getResult().
Do that with Assert.assertEquals(...) not with Mockito.verify(...).

For the most part I agree with #davidxxx's point about the mock verifying tradeoff. If you have a setup that allows you to make assertions about an outcome like a map that is created as a result, go for it!
From an API perspective doStuff is a simple straight-forward method: Throw something at it, get something back. The information you are interested in will be contained in the map (this would be your assertion).
There is a lot going on under the hood before doStuff returns something. Many people tend to want to break up encapsulation when testing stuff. They are constantly looking for ways to uncover what is going on behind the curtains. I believe that's totally natural. But of course, it's also an anti pattern. It doesn't matter what tool you (mis)use to break natural boundaries (mocking frameworks, custom reflection, "back doors" in your code base, etc). It is always wrong. As #Michael already pointed out, the call to doOtherStuff is indeed an implementation detail. Take the perspective of client code that makes a call to doStuff. Is it interested in how the map is created? I doubt it. This should also be your testing perspective.
One last thing about using verification in tests. I would like to mitigate the trade off statement. I really don't like the generalization here. Verification is not always the less attractive choice compared to real assertions:
// Valid test without any verifaction
#Test
void method_foo_returns_gibberish (#Mock SomeInput someInput) {
// Maybe this is just to prevent an NPE ...
when(someInput.readStuff()).thenReturn("bla");
assertEquals("gibberish", Foo.foo(someInput));
}
// Test made possible by verification
#Test
void method_foo_is_readonly (#Mock SomeInput someInput) {
Foo.foo(someInput);
verify(someInput.readStuff());
verifyNoMoreInteractions(mockedList);
}
This is just the most obvious example that I could think of. There is a fraction of BDD geniuses who strive to build their whole architecture around verification driven tests! Here is an excellent article by Martin Fowler
When talking about testing, most of the time, there is no black and white. Using mocks and verification means writing different tests.
As always, it's about picking the right tool.

Creating tests with many mock objects?

I'm having doubts about if I should create tests that have many mock objects or not.
I recently read When should I mock? and I'm feeling confused.
Let's take a look at a method I have (it is just to illustrate the problem)
#Override
protected void validate() throws WTException {
Either<ImportError, RootFinderResult> rootPart = getDataValidator().getRootPart();
if (rootPart.isLeft()) {
addValidationMessage(ROOT_PART_NOT_FOUND);
} else if (rootPart.isRight()) {
getObjectsToValidate().forEach(Lambda.uncheckedBiConsumer((part, epmDocuments) -> {
LocalizableMessage rootRevision = getRevision(part);
Optional<EPMDocument> wrongRevisionEPM = epmDocuments.stream()
.filter(epmDocument -> !isSameRevision(rootRevision, epmDocument))
.findAny();
wrongRevisionEPM.ifPresent(epmDocument -> addValidationMessage("blabla"));
}));
}
}
All of the following methods need to have a connection to a server in order to work, otherwise they will throw errors
getDataValidator().getRootPart();
getRevision(part)
!isSameRevision(rootRevision, epmDocument))
In addition I can't create 'real' objects of part or epm documents. This also requires to have a connection to a server.
So at this point, what I really want to test is actually logic of this part of code
Optional<EPMDocument> wrongRevisionEPM = epmDocuments.stream()
.filter(epmDocument -> !isSameRevision(rootRevision, epmDocument))
.findAny();
wrongRevisionEPM.ifPresent(epmDocument -> addValidationMessage("blabla"));
But to test it I need to mock really many objects
#Spy
#InjectMocks
private SameRevision sameRevision;
#Mock
private WTPartRelatedObjectDataValidator wTPartRelatedObjectDataValidator;
#Mock
private ValidationEntry validationEntry;
#Mock
private WTPart rootPart1, rootPart2;
#Mock
private EPMDocument epmDocument1, epmDocument2, epmDocument3;
#Mock
private Either<ImportError, RootFinderResult> rootPart;
#Mock
private LocalizableMessage rootPartRevisionOne, rootPartRevisionTwo;
so finally I can test the logic:
#Test
#DisplayName("Should contain error message when part -> epms revisions are not the same")
void shoulHaveErrorMessagesWhenDifferentRevisions() throws Exception {
doReturn(getMockObjectsToValidate()).when(sameRevision).getObjectsToValidate();
doReturn(rootPart).when(liebherrWTPartRelatedObjectDataValidator).getRootPart();
doReturn(false).when(rootPart).isLeft();
doReturn(true).when(rootPart).isRight();
doReturn(rootPartRevisionOne).when(sameRevision).getRevision(rootPart1);
doReturn(rootPartRevisionTwo).when(sameRevision).getRevision(rootPart2);
doReturn(true).when(sameRevision).isSameRevision(rootPartRevisionOne, epmDocument1);
doReturn(false).when(sameRevision).isSameRevision(rootPartRevisionOne, epmDocument2);
doReturn(true).when(sameRevision).isSameRevision(rootPartRevisionTwo, epmDocument3);
validationEntry = sameRevision.call();
assertEquals(1, validationEntry.getValidationMessageSet().size());
}
where
doReturn(rootPart).when(liebherrWTPartRelatedObjectDataValidator).getRootPart();
doReturn(false).when(rootPart).isLeft();
doReturn(true).when(rootPart).isRight();
doReturn(rootPartRevisionOne).when(sameRevision).getRevision(rootPart1);
doReturn(rootPartRevisionTwo).when(sameRevision).getRevision(rootPart2);
can be moved to #BeforeEach.
At last, I have my test and it works. It validates what I wanted to be validated but in order to come to this point I had to put a lot of effort to come through the whole API which needs interactions with a server.
What do you guys think, is it worth it to create tests like this? I guess this is a wide-open topic 'cause many newbies that try to come into the 'test world' will have similar a problem, so please do not close the topic because of opinion-based judgement and give your feedback on this topic.

You are right. It is a big effort to mock all those dependencies. Let me get over a few points that may make things clearer:
Treat writing tests like an investment: So yes, sometimes it is more effort to write the test than to write the actual code. However, you will thank yourself later when you introduce a bug and the tests can catch it. Having good tests gives you confidence when modifying your code that you didn't break anything, and if you did, your tests will find the issue. It pays off over time.
Keep your test focused on a specific class. Mock the rest: When you mock everything except the class under test, you can be sure that when a problem occurs that it is from the class under test, and not from one of its dependencies. This makes troubleshooting a lot easier.
Think of testability when writing new code: Sometimes it may not be avoidable to have a complicated piece of code which is hard to test. However, generally, this situation can be avoided by keeping the number of dependencies you have to a minimum and writing testable code. For example, if a method needs 5 or 6 more dependencies to do its job, then probably that method is doing too much and could be broken down. Same thing can be said on a class level, modules, etc..

You should mock other dependencies on which your class which going to be tested relies and set up behavior which you need .
This needs to be done to test your method isolated and not dependent on thrirdparty classes
You can write private void methods which can contain your mock behavior and use them in tests ,
In #BeforeEach annotated method you could mock behavior which wil be same in all tests or mock the same mocking behavior for across all tests
In your method which is void you can have spy objects which can be veereified if they were called like Mockito.verify()

Yes, it is quite time investing when you have to mock so many things.
In my opinion, if you add some value when testing something, it is worth testing, the problem could be of course how much time would you consume.
In your specific case, I would test on different "layers".
For example, the methods:
getDataValidator().getRootPart();
getRevision(part)
!isSameRevision(rootRevision, epmDocument))
They can be independently tested and in your case just mock their result, meaning that you don't really care about the parameters there, you just care what happens in case of a certain return value.
So, on one layer you really test the functionality, on the next layer you just mock the result you need, in order to test the other functionality.
I hope it is more clear now...

JUnit - testing with re-usable functions; validity

What I am trying to do is to test some Lucene code I wrote and would like some information on best practices when using JUnit for testing. Lucene, BTW, is a search engine which you can use to create a flat file to index a bunch of data.
So what I would like to test is the creation of this inverted index, then search through the index to verify that some data is present.
My question is in the code:
public class IndexTest {
#Test
public void testWriteIndexFromDB() {
//run test
assertTrue(something in this test); // some test for this method
// Is using a function like so a proper way of writing a test?
checkSomeDataIsReturned();
}
#Test
public void testWriteIndexFromExcelFile() {
//run test
assertTrue(something in this test); // some test for this method
// Is using a function like so a proper way of writing a test?
checkSomeDataIsReturned();
}
#Test
public void testRefreshIndexWithNewData() {
//run test
assertTrue(something in this test); // some test for this method
// Is using a function like so a proper way of writing a test?
checkSomeDataIsReturned();
}
// this function checks that data is returned after writing an index
public void checkSomeDataIsReturned(){ // not a test but does a check anyways
results = myIndex.searchForStuff(some input);
assertTrue(results.length > 0); // if length is zero, there is no data and something went wrong
}
}
To summarize, I have three options to write an index, I am testing that each of them writes. Is the re-usable function that is not a test the proper way to write a test? Or is there a better practice?

Is off course a good thing write reusable code in test, but more important than that is writing that are easy to understand. In general the asserts are in the test method itself, moving the asserts to helpers methods can make your test difficult to understand.
One way to write reusable code for checking expectations its using Hamcrest (https://code.google.com/p/hamcrest/wiki/Tutorial) and build a matchers (the library also comes with some very useful matchers for collections and stuff like that).
for example, you can write something like that:
public void
test_can_index_from_database() {
// create your index from database
assertThat(myIndex, containsWord('expected_word_in_index'));
}
The matchers "containsWord(String)" its a matcher that you write using hamcrest and you can re-use this logic in all your test. And with hamcrest you can write really easy to understand test.

Well, good practices such as reusable code are to be used in unit tests, too.
However, please consider that if you are in need to repeat code in unit tests, it may (and often does) mean that your tested methods take too many responsibilities.
I don't know if it really is your case but think about refactoring your code (here splitting your tested methods into more smaller methods) so you don't feel the need to repeat the same tests all over.
When each method takes a single responsibility only, and delegate the shared code to another method/class, you test this functionality somewhere else and here just test (using mocking and spying) if your methods calls the corresponding method/object.

How focused JUnit tests in BDD way should be

I'm test-driving some code for practice and spotted strange situation.
There is a ChannelRegistry that contains all communication channels references, and PrimaryConsumer who needs to attach itself to one of those channels choosen in runtime when initialize() called.
So I've done my first test as follows:
#RunWith(MockitoJUnitRunner.class)
public class PrimaryConsumerTest {
private #Mock ChannelsRegistry communicationRegistry;
private PrimaryConsumer consumer;
#Before
public void setup() {
consumer = new PrimaryConsumer(communicationRegistry);
}
#Test
public void shouldAttachToChannel() throws Exception {
consumer.initialize();
verify(communicationRegistry).attachToChannel("channel", consumer);
}
}
I'm checking if attaching method is called. To get it green I put impl like that:
public void initialize() {
communicationRegistry.attachToChannel("channel", this);
}
Now next test: get channel id by name and attach to this specific channel. I want my test to describe class' behavior instead of its internals so I don't want my test to be "shouldGetSpecificChannel". Instead I check if it can attach to channel selected in runtime:
#Test
public void shouldAttachToSpecificChannel() throws Exception {
String channelName = "channel";
when(communicationRegistry.getChannel("channel_name")).thenReturn(channelName);
consumer.initialize();
verify(communicationRegistry).attachToChannel(channelName, consumer);
}
This test passes immediately, but implementation is screwed ("channel" hardcoded).
2 questions here:
is it ok to have 2 tests for such behavior? Maybe I should stub getting channel immediately in first test? If so, how does it map to testing single thing in single test?
how to cope with such situation: tests green, impl "hardcoded"? Should I write another test with different channel's name? If so, should I remove it after correcting impl (as it gets useless?)
UPDATE:
Just some clarifications.
I've hardcoded "channel" here
public void initialize() {
communicationRegistry.attachToChannel("channel", this);
}
just to make first test pass quickly. But then, when running second test it passes immediately. I don't verify if stubbed method was called as I think stubs should not be verified explicitly.
Is this what you Rodney mean saying test are redundant? If yes shoud I make stub at the very beginning in the first test?

More tests is usually preferred to too few, so two tests is fine. A better question is whether the two tests are redundant: is there any situation or combination of inputs that would make one of the tests fail, but not the other? Then the two tests are both needed. If they always fail or succeed together, then you probably need only one of them.
When would you need a different value for channelName? It sounds like this is a configuration setting that is irrelevant to these particular tests. That's fine, perhaps you would test that configuration at a higher level, in your integration tests. A bigger concern I would have is why it's hard-coded in the first place: it should be injected into your class (probably via the constructor). Then you can test different channel names -- or not. Either way, you don't want to be changing your code just for testing if it means changing it back when you're done.

Basically ditto Rodney for the question of multiple tests. I would suggest, based on your update, one or two things.
First off, you have used the same data for both tests. In the Kent Beck book on TDD he mentions the use of "Triangulation". If you used different reference data in the second case then your code would not have passed without any additional work on your part.
On the other hand, he also mentions removing all duplication, and duplication includes duplication between the code and the tests. In this scenario you could have left both of your tests as is, and refactored out the duplication between the string "channel" in the code and the same in the test by replacing the literal in your class under test with the call to your communicationRegistry.getChannel(). After this refactoring you now have the string literal in one and only one place: The test.
Two different approaches, same result. Which one you use comes down to personal preference. In this scenario I would have taken the second approach, but that's just me.
Reminder to check out Rodney's answer to the question of multiple tests or not. I'm guessing you could delete the first one.
Thanks!
Brandon

How can I simplify testing of side-effect free methods in Java?

Functions (side-effect free ones) are such a fundamental building block, but I don't know of a satisfying way of testing them in Java.
I'm looking for pointers to tricks that make testing them easier. Here's an example of what I want:
public void setUp() {
myObj = new MyObject(...);
}
// This is sooo 2009 and not what I want to write:
public void testThatSomeInputGivesExpectedOutput () {
assertEquals(expectedOutput, myObj.myFunction(someInput);
assertEquals(expectedOtherOutput, myObj.myFunction(someOtherInput);
// I don't want to repeat/write the following checks to see
// that myFunction is behaving functionally.
assertEquals(expectedOutput, myObj.myFunction(someInput);
assertEquals(expectedOtherOutput, myObj.myFunction(someOtherInput);
}
// The following two tests are more in spirit of what I'd like
// to write, but they don't test that myFunction is functional:
public void testThatSomeInputGivesExpectedOutput () {
assertEquals(expectedOutput, myObj.myFunction(someInput);
}
public void testThatSomeOtherInputGivesExpectedOutput () {
assertEquals(expectedOtherOutput, myObj.myFunction(someOtherInput);
}
I'm looking for some annotation I can put on the test(s), MyObject or myFunction to make the test framework automatically repeat invocations to myFunction in all possible permutations for the given input/output combinations I've given, or some subset of the possible permutations in order to prove that the function is functional.
For example, above the (only) two possible permutations are:
myObj = new MyObject();
myObj.myFunction(someInput);
myObj.myFunction(someOtherInput);
and:
myObj = new MyObject();
myObj.myFunction(someOtherInput);
myObj.myFunction(someInput);
I should be able to only provide the input/output pairs (someInput, expectedOutput), and (someOtherInput, someOtherOutput), and the framework should do the rest.
I haven't used QuickCheck, but it seems like a non-solution. It is documented as a generator. I'm not looking for a way to generate inputs to my function, but rather a framework that lets me declaratively specify what part of my object is side-effect free and invoke my input/output specification using some permutation based on that declaration.
Update: I'm not looking to verify that nothing changes in the object, a memoizing function is a typical use-case for this kind of testing, and a memoizer actually changes its internal state. However, the output given some input always stays the same.

If you are trying to test that the functions are side-effect free, then calling with random arguments isn't really going to cut it. The same applies for a random sequence of calls with known arguments. Or pseudo-random, with random or fixed seeds. There's a good chance are that a (harmful) side-effect will only occur with any of the sequence of calls that your randomizer selects.
There is also a chance that the side-effects won't actually be visible in the outputs of any of the calls that you are making ... no matter what the inputs are. They side-effects could be on some other related objects that you didn't think to examine.
If you want to test this kind of thing, you really need to implement a "white-box" test where you look at the code and try and figure out what might cause (unwanted) side-effects and create test cases based on that knowledge. But I think that a better approach is careful manual code inspection, or using an automated static code analyser ... if you can find one that would do the job for you.
OTOH, if you already know that the functions are side-effect free, implementing randomized tests "just in case" is a bit of a waste of time, IMO.

I'm not quite sure I understand what you are asking, but it seems like Junit Theories (http://junit.sourceforge.net/doc/ReleaseNotes4.4.html#theories) could be an answer.

In this example, you could create a Map of key/value pairs (input/output) and call the method under test several times with values picked from the map. This will not prove, that the method is functional, but will increase the probability - which might be sufficient.
Here's a quick example of such an additional probably-functional test:
#Test public probablyFunctionalTestForMethodX() {
Map<Object, Object> inputOutputMap = initMap(); // this loads the input/output values
for (int i = 0; i < maxIterations; i++) {
Map.Entry test = pickAtRandom(inputOutputMap); // this picks a map enty randomly
assertEquals(test.getValue(), myObj.myFunction(test.getKey());
}
}
Problems with a higher complexity could be solved based on the Command pattern: You could wrap the test methods in command objects, add the command object to a list, shuffle the list and execute the commands (= the embedded tests) according to that list.

It sounds like you're attempting to test that invoking a particular method on a class doesn't modify any of its fields. This is a somewhat odd test case, but it's entirely possible to write a clear test for it. For other "side effects", like invoking other external methods, it's a bit harder. You could replace local references with test stubs and verify that they weren't invoked, but you still won't catch static method calls this way. Still, it's trivial to verify by inspection that you're not doing anything like that in your code, and sometimes that has to be good enough.
Here's one way to test that there are no side effects in a call:
public void test_MyFunction_hasNoSideEffects() {
MyClass systemUnderTest = makeMyClass();
MyClass copyOfOriginalState = systemUnderTest.clone();
systemUnderTest.myFunction();
assertEquals(systemUnderTest, copyOfOriginalState); //Test equals() method elsewhere
}
It's somewhat unusual to try to prove that a method is truly side effect free. Unit tests generally attempt to prove that a method behaves correctly and according to contract, but they're not meant to replace examining the code. It's generally a pretty easy exercise to check whether a method has any possible side effects. If your method never sets a field's value and never calls any non-functional methods, then it's functional.
Testing this at runtime is tricky. What might be more useful would be some sort of static analysis. Perhaps you could create a #Functional annotation, then write a program that would examine the classes of your program for such methods and check that they only invoke other #Functional methods and never assign to fields.
Randomly googling around, I found somebody's master's thesis on exactly this topic. Perhaps he has working code available.
Still, I will repeat that it is my advice that you focus your attention elsewhere. While you CAN mostly prove that a method has no side effects at all, it may be better in many cases to quickly verify this by visual inspection and focus the remainder of your time on other, more basic tests.

have a look at http://fitnesse.org/: it is used often for Acceptance Test but I found it is a easy way to run the same tests against huge amount of data

In junit you can write your own test runner. This code is not tested (I'm not sure if methods which get arguments will be recognized as test methods, maybe some more runner setup is needed?):
public class MyRunner extends BlockJUnit4ClassRunner {
#Override
protected Statement methodInvoker(final FrameworkMethod method, final Object test) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
Iterable<Object[]> permutations = getPermutations();
for (Object[] permutation : permutations) {
method.invokeExplosively(test, permutation[0], permutation[1]);
}
}
};
}
}
It should be only a matter of providing getPermutations() implementation. For example it can take data from some List<Object[]> field annotated with some custom annotation and produce all the permutations.

I think the term you're missing is "Parametrized Tests". However it seems to be more tedious in jUnit that in the .Net flavor. In NUnit, the following test executes 6 times with all combinations.
[Test]
public void MyTest(
[Values(1,2,3)] int x,
[Values("A","B")] string s)
{
...
}
For Java, your options seem to be:
JUnit supports this with version 4. However it's a lot of code (it seems, jUnit is adamant about test methods not taking parameters). This is the least invasive.
DDSteps, a jUnit plugin. See this video that takes values from appropriately named excel spreadsheet. You also need to write a mapper/fixture class that maps values from the spreadsheet into members of the fixture class, that are then used to invoke the SUT.
Finally, you have Fit/Fitnesse. It's as good as DDSteps, except for the fact that the input data is in HTML/Wiki form. You can paste from an excel sheet into Fitnesse and it formats it correctly at the push of a button. You need to write a fixture class here too.

Im afraid that I dont find the link anymore, but Junit 4 has some help functions to generate testdata. Its like:
public void testData() {
data = {2, 3, 4};
data = {3,4,5 };
...
return data;
}
Junit will then thest your methods will this data. But as I said, I cant' find the link anymore (forgot the keywords) for a detailed (and correct) example.