I found many information about bad practices to use several asserts in one method (one test condition). It's clear. Each method should test one and only one condition. So it should be one assert in most cases.
But I can't find any information about best practices to situation, if my one assert tests several conditions.
Example:
#Test
public void findAllPerformers_returnsPerformersInAlphabeticalOrder() {
List<String> performers = reportRepository.findAllPerformers();
assertThat("Should return performers in alphabetical order",
performers, contains("Bart Simpson", "Homer Simpson",
"Ned Flanders", "Xena Warrior Princess"));
}
First, I had to have 2 more test methods (conditions): findAllPerformers_returnsAllUniquePerformers and findAllPerformers_returnsEachPerformerOnlyOnce. But then I realized that my first method tests this two conditions too!
So, what should I do?
Leave all methods as is.
Remove 2 other test methods and rename my findAllPerformers_returnsPerformersInAlphabeticalOrder method to very long name with all three conditions included (because it really tests all three conditions!)
Just remove 2 other test methods.
Leave all methods but invoke findAllPerformers_returnsPerformersInAlphabeticalOrder method from two other methods.
Copy-paste findAllPerformers_returnsPerformersInAlphabeticalOrder method contents to two other methods.
Anything else...
If this test fails then the person looking at the failed test would initially see findAllPerformers_returnsPerformersInAlphabeticalOrder().
Looking at that alone it appears that the method under test returned a list that wasn't in alphabetical order, not that the list didn't contain 'Bart Simpson'.
Avoid options 4 and 5, they don't really solve anything and will increase duplication/maintenance without any added value (since tests will fail for reasons not mentioned in the test name). Instead try to find a way to actually test what you want with the assert.
Related
Let's imagine having class (written in Java-like pseudocode):
class MyClass {
...
public List<Element> getElementsThatContains(String str) {
return this.getElementsThatContains(new Set<String> { str });
}
public List<Element> getElementsThatContains(Set<String> strs) {
...
}
}
First of all - I have getElementsThatContains(Set<String> strs) properly 100% covered.
How should I cover getElementsThatContains(String str):
Should I copy (almost) all the tests but with call to getElementsThatContains(String str)?
Should I just make one test
method that check if results from first and second methods are same
(with same incoming data)?
Should I refactor my code so I do not have
such a situation? (If yes, how?)
Yes, you should cover both methods. The reason for having unit tests is the safety net, when the code is refactored. For example, Someone might refactor the implementation of 'getElementsThatContains(String str)' and it will always return an empty List. Despite getElementsThatContains(Set strs) has 100% coverage those tests won't catch this.
No, you should not make one test method that check if results from first and second methods are same. This is generally considered a bad practice. Moreover, if there is a bug in one method, your test would just check the other method returns same incorrect result.
No, you should not copy all the tests, because the test cases for each method would be different. The arguments for the methods are different. So you will have different test cases for each, despite that underneath the same method is called.
Yes you should test both methods, and you should use distinct test cases for each method.
But you should care less for your line coverage.
Don't get me wrong here! It is important to keep the line coverage high. But it is more important to have 100% behavior coverage. And if you come across untested lines your question should be: "Is this untested code needed (i.e. what requirement does it implement) or is it obsolete?".
When we write our tests with line coverage in mind we tend to focus on the implementation details of our code under test. In consequence our tests are likely to fail when we change this implementation details (e.g. during refactoring). But our tests should only fail if the tested behavior changes and not when we change the way this behavior is achieved.
I use assertJ and have multiple assertThat assertions in my test case.
When first assertion fails test is finished but I don't want that.
I'd like to have information about all failing assertions after single executing of test case.
Is it any way to do that ?
I have found solution with SoftAssertions here -> http://joel-costigliola.github.io/assertj/assertj-core-features-highlight.html#soft-assertions
but it's ugly to add variable. before each assertThat
A bit of example code would help, but then, this is more of a theoretical problem, as the real answer is: consider not having multiple assertions in one test call!
Meaning: the idea of a failing test is to get you to a problem as quickly as possible. When you combine multiple asserts into a single test, then you make our life harder by default. Because instead of knowing "test X with assertion Y failed, you have to first study logs very carefully to identify which asserts passed, and which one failed.
Therefore the recommend practice is to not put multiple asserts/check into a single test.
If you don't like soft assertions, you can give a try to JUnit 5 assertAll but otherwise I would follow #GhostCat advice and try to assert one thing per test (that usually leads to only a few assertions).
I think that in some cases you may and sometimes even you have to assert multiple things in a single test method if your method perform multiple changes that you should check through different levels/abstractions.
For example as you test a method that adds an element in a object that stores it, you can assert that the number of elements contained in the object were incremented by one but you can also check that the new element were correctly added concerning its values.
You have two levels/abstractions : the object that contains the element that has a "direct/core" state and the elements that it contains that have their own states.
In splitting it in two assertions, it would give a test that looks like :
#Test
public void addElt(){
foo.addElt(new Element("a name", "a role"));
assertThat(foo).extracting(Foo::getSize)
.contains(actualSize+1);
assertThat(foo.getLastElt()).extracting(Element::getName, Element::getRole)
.containsExactly(addedElt.getName(), addedElt.getRole());
}
So now why trying to couple two assertions that checks two different
things ?
Does it really bring a value for debugging your test ?
I don't think so.
Trying to assert the changes on the two level of abstraction in a single assertion makes clearly no sense : complex and useless noises.
If the first assertion fails :
assertThat(foo).extracting(Foo::getSize)
.contains(actualSize+1);
It very probably means that the element was not added.
So in this case, performing the second assertion :
assertThat(foo.getLastElt()).extracting(Element::getName, Element::getRole)
.containsExactly(addedElt.getName(), addedElt.getRole());
makes no sense as it will very probably be also in error.
The developer that handles the failure test needs only to have useful information and not noise that can make its solving harder. So having a feedback about the size that is not which one expected is just what you need.
What I try to explain is right for AssertJ as for any testing framework.
Is there any reason to have more than one verify statement when testing a specific functionality - ie. verify that multiple/ or no dependent methods were called?
For Example:
public void doSomething(int size){
if(size < 50){
return;
}
someService.someMethod();
anotherService.someMethod();
}
To test this method
#Test
public void testDoSomethingWithSmallSize() throws Exception{
testObj.doSomething(5);
verify(someServiceMock, never()).someMethod();
//IS THERE ANY VALUE TO VERFIYING MORE THAN ONE CALL?
//LIKE THIS??
verfiy(anotherServiceMock, never()).someMethod();
}
Is there value to having the second verify statement or is it unnecessary because if the first statement wasn't called the second wasn't either?
You should verify the 2 statements because your code can change.
Unit test is some kind of documentation of your code.
Verifying both statements means that both statements MUST not be called.
Verifying only one means that only the 1st statement MUST not be called.
If someone changes the method to call anotherService.someMethod() inside the if statement, your test will still pass with 1 verify and will fail with 2 verify.
You are right to worry that your test should only be testing one "concept" at a time, and that it is a matter of judgment about what constitutes a test that does "too much". Your original example is a good one:
#Test
public void testDoSomethingWithSmallSize() throws Exception{
testObj.doSomething(5);
verify(someServiceMock, never()).someMethod();
verify(anotherServiceMock, never()).someMethod();
// GOOD: You've called one method-under-test and
// verified two related postconditions. A perfect, small, clear test.
}
But it's easy to take this too far.
#Test
public void testDoSomethingWithAnySmallSize() throws Exception{
testObj.doSomething(1);
testObj.doSomething(3);
testObj.doSomething(5);
verify(someServiceMock, never()).someMethod();
verify(anotherServiceMock, never()).someMethod();
// LESS GOOD: You've called one method-under-test three times, but
// they share postconditions and concepts. Pragmatic, but not the
// "one test method for one test case" ideal of most frameworks.
}
#Test
public void thisShouldBeThreeTests() throws Exception{
testObj.doSomething(7);
verify(someServiceMock).doAThing(7);
verify(anotherService).doAThing(700);
testObj.doSomethingElse(9);
verify(someServiceMock).doAThing(9);
verify(anotherService).doAThing(900);
testObj.doAThirdThing(12);
verify(someServiceMock).doAThing(12);
verify(anotherService).doAThing(1200);
// BAD: Even though these are related, this could easily be three
// unrelated tests for better naming and reporting, and to help you
// identify why one case might be failing versus three. Break this up.
}
So yes, don't be afraid to have more than one verify in the same test, but do be careful not to let your verify statements stray to be unrelated, and be especially careful if you're resetting your test setup with Mockito's reset method.
YES! Part of the function of a unit test is documenting the expected behaviors of code. The test(s) should be written in a way that treats the code under test as a black box. If the code under test will do two things in a specific order (if conditions are met) then the unit test should check that all things were done under positive conditions AND that NONE of the items were done under the negative conditions.
If you only check one condition in your test, then 2 years from now, an intern might update the code, and need to remove the one task that is being checked. Your test will still pass (code removed not executed - check!) but that doesn't mean the code is behaving correctly.
This question already has answers here:
Continuing test execution in junit4 even when one of the asserts fails
(7 answers)
Closed 6 years ago.
I am somewhat new to TDD and JUnit, I know that I can write test cases for methods I am implementing in my code.
And obviously, there are methods in my code which need several corner cases to be tested in order to verify that method implementation is OK. Since generally the good practice is keeping one test method per one method in code, I have to add multiple assertions for that kind of a method as explained in this answer. https://stackoverflow.com/a/762582/5715934
public void testValueOf() {
assertEquals(1, Integer.valueOf("1").intValue());
assertEquals(0, Integer.valueOf("0").intValue());
assertEquals(-1, Integer.valueOf("-1").intValue());
assertEquals(Integer.MAX_VALUE, Integer.valueOf("2147483647").intValue());
assertEquals(Integer.MIN_VALUE, Integer.valueOf("-2147483648").intValue());
....
}
But when I am executing the test case, I am not getting test status (pass/fail) for each assertion inside the test method. Instead, it shows 'red' even if one assertion is failed and green if all are passed.
Isn't it easier to have the track of each assertion to make debugging easier? And is there any formal way/tool/workaround to do that (JUnit 4)?
First of all it is NOT a good idea to have a single test method for single method, quite the opposite. In your code above each of the asserts should actually be a separate test method.
This doesn't mean that you will always have a single assert in test method, this will be the case only for the simplest ones. If a method e.g. returns an object you will have more asserts that validate that this object is good.
But the most important thing is that the method under tests is called usually only once.
The only way to know which assertion failed is by looking at the message/stack trace. But generally if a test fails you know that something is wrong, you open IDE and look at the test.
I am trying to write a unit test for a method which takes a string as a para-
meter and throws an exception if it is malformed (AND NONE if it is okay).
I want to write a parameterized test which feeds in several strings and the
expected exception (INCLUDING the case that none is thrown if the input
string is well-formed!). If trying to use the #Test(expect=SomeException.class)
annotation, I encountered two problems:
expect=null is not allowed.
So how could I test for the expected outcome of NO exception to be thrown
(for well-formed input strings)?
expect= not possible?
I not yet tried it, but I strongly suspect that this is the case after
reading this (could you please state whether this is true?):
http://tech.groups.yahoo.com/group/junit/message/19383
This then seems to be the best solution I found yet. What do you think about
it, especially compared to that:
How do I test exceptions in a parameterized test?
Thank you in advance for any help, I look forward the discussion :)
Create two test case classes:
ValidStringsTest
InvalidStringsTest
Obviously the first one tests all sorts of valid inputs (not throwing an exception), whilst the second one always expects the exception.
Remember: readability of your tests is even more important than readability of production code. Don't use wacky flags, conditions and logic inside JUnit test cases. Simplicity is the king.
Also see my answer here for a hint how to test for exceptions cleanly.
Have two different tests - one for valid inputs and one for invalid ones. I haven't used JUnit 4 so I can't comment on the exact annotation format - but basically you'd have one parameterized test with various different invalid inputs, which says that it does expect an exception, and a separate test with various different valid inputs which doesn't say anything about exceptions. If an exception is thrown when your test doesn't say that it should be, the test will fail.
Splitting the test cases into two test classes is the appropriate approach in many cases - as both Tomasz and Jon already outlined.
But there are other cases where this split is not a good choice just in terms of readability. Let's assume the rows in the tested data set have a natural order and if the rows are sorted by this natural order it may be easy to see whether or not the test data covers all relevant use cases. If one splits the test cases into two test classes, there is no longer an easy way to see whether all relevant test cases are covered. For these cases
How do I test exceptions in a parameterized test?
seeems to provide the best solution indeed.