Testing for optional exception in parameterized JUnit 4+ test - java

I am trying to write a unit test for a method which takes a string as a para-
meter and throws an exception if it is malformed (AND NONE if it is okay).
I want to write a parameterized test which feeds in several strings and the
expected exception (INCLUDING the case that none is thrown if the input
string is well-formed!). If trying to use the #Test(expect=SomeException.class)
annotation, I encountered two problems:
expect=null is not allowed.
So how could I test for the expected outcome of NO exception to be thrown
(for well-formed input strings)?
expect= not possible?
I not yet tried it, but I strongly suspect that this is the case after
reading this (could you please state whether this is true?):
http://tech.groups.yahoo.com/group/junit/message/19383
This then seems to be the best solution I found yet. What do you think about
it, especially compared to that:
How do I test exceptions in a parameterized test?
Thank you in advance for any help, I look forward the discussion :)

Create two test case classes:
ValidStringsTest
InvalidStringsTest
Obviously the first one tests all sorts of valid inputs (not throwing an exception), whilst the second one always expects the exception.
Remember: readability of your tests is even more important than readability of production code. Don't use wacky flags, conditions and logic inside JUnit test cases. Simplicity is the king.
Also see my answer here for a hint how to test for exceptions cleanly.

Have two different tests - one for valid inputs and one for invalid ones. I haven't used JUnit 4 so I can't comment on the exact annotation format - but basically you'd have one parameterized test with various different invalid inputs, which says that it does expect an exception, and a separate test with various different valid inputs which doesn't say anything about exceptions. If an exception is thrown when your test doesn't say that it should be, the test will fail.

Splitting the test cases into two test classes is the appropriate approach in many cases - as both Tomasz and Jon already outlined.
But there are other cases where this split is not a good choice just in terms of readability. Let's assume the rows in the tested data set have a natural order and if the rows are sorted by this natural order it may be easy to see whether or not the test data covers all relevant use cases. If one splits the test cases into two test classes, there is no longer an easy way to see whether all relevant test cases are covered. For these cases
How do I test exceptions in a parameterized test?
seeems to provide the best solution indeed.

Related

I don't want assertJ assertThat ends test when assertion fails

I use assertJ and have multiple assertThat assertions in my test case.
When first assertion fails test is finished but I don't want that.
I'd like to have information about all failing assertions after single executing of test case.
Is it any way to do that ?
I have found solution with SoftAssertions here -> http://joel-costigliola.github.io/assertj/assertj-core-features-highlight.html#soft-assertions
but it's ugly to add variable. before each assertThat
A bit of example code would help, but then, this is more of a theoretical problem, as the real answer is: consider not having multiple assertions in one test call!
Meaning: the idea of a failing test is to get you to a problem as quickly as possible. When you combine multiple asserts into a single test, then you make our life harder by default. Because instead of knowing "test X with assertion Y failed, you have to first study logs very carefully to identify which asserts passed, and which one failed.
Therefore the recommend practice is to not put multiple asserts/check into a single test.
If you don't like soft assertions, you can give a try to JUnit 5 assertAll but otherwise I would follow #GhostCat advice and try to assert one thing per test (that usually leads to only a few assertions).
I think that in some cases you may and sometimes even you have to assert multiple things in a single test method if your method perform multiple changes that you should check through different levels/abstractions.
For example as you test a method that adds an element in a object that stores it, you can assert that the number of elements contained in the object were incremented by one but you can also check that the new element were correctly added concerning its values.
You have two levels/abstractions : the object that contains the element that has a "direct/core" state and the elements that it contains that have their own states.
In splitting it in two assertions, it would give a test that looks like :
#Test
public void addElt(){
foo.addElt(new Element("a name", "a role"));
assertThat(foo).extracting(Foo::getSize)
.contains(actualSize+1);
assertThat(foo.getLastElt()).extracting(Element::getName, Element::getRole)
.containsExactly(addedElt.getName(), addedElt.getRole());
}
So now why trying to couple two assertions that checks two different
things ?
Does it really bring a value for debugging your test ?
I don't think so.
Trying to assert the changes on the two level of abstraction in a single assertion makes clearly no sense : complex and useless noises.
If the first assertion fails :
assertThat(foo).extracting(Foo::getSize)
.contains(actualSize+1);
It very probably means that the element was not added.
So in this case, performing the second assertion :
assertThat(foo.getLastElt()).extracting(Element::getName, Element::getRole)
.containsExactly(addedElt.getName(), addedElt.getRole());
makes no sense as it will very probably be also in error.
The developer that handles the failure test needs only to have useful information and not noise that can make its solving harder. So having a feedback about the size that is not which one expected is just what you need.
What I try to explain is right for AssertJ as for any testing framework.

Assume vs assert in JUnit tests

I have read that assume will not run the test if assumption failed,
but I am not sure regarding the logic of when to place assert vs assume.
For example: any resource loading check should be done with assume?
When should I use assume over assert?
(Note: i am looking for correct design of when to use one over the other)
You would use assume if you have circumstances under which some tests should not run at all. "Not run" means that it cannot fail, because, well, it did not run.
You would use assert to fail a test if something goes wrong.
So, in a hypothetical scenario where:
you have different builds for different customers, and
you have some resource which is only applicable to a particular client, and
there is something testable about that resource, then
you would write a test which:
assumes that the resource is present, (so the test will not run on customers that do not have that resource,) and then
asserts that everything about the resource is okay (so on the customer that does actually have the resource, the test makes sure that the resource is as it should be.)
The Assert class is the workhorse of JUnit and is the class JUnit testers are most familiar with. Most JUnit assert signatures are similar in nature. They consist of an optional message, an expected instance or variable and the actual instance or variable to be compared. Or, in the case of a boolean test like True, False, or Null, there is simply the actual instance to be tested.
The signature with a message simply has an initial parameter with a message string that will be displayed in the event the assert fails:
assert<something>(“Failure Message String”, <condition to be tested>);
Assumptions:
You’ve probably heard that it’s best not to work on assumptions so here is a testing tool JUnit gives you to ensure your tests don’t.
Both Asserts and Assumes stop when a test fails and move on to the next test. The difference is that a failed Assert registers the failure as a failed test while an Assume just moves to the next test. This permits a tester to ensure that conditions, some of which may be external and out of control of the tester, are present as required before a test is run.
There are four varieties of Assumes: one to check a boolean condition, one to check that an exception has not occurred, one to check for null objects, and one that can take a Hamcrest matcher. As seen in the Assert section above, the ability to take a Hamcrest matcher is a gateway to testing flexibility.
You can read more here
https://objectcomputing.com/resources/publications/sett/march-2014-junit-not-just-another-pretty-assert/
In short Assume used to disable tests, for example the following disables a test on Linux: Assume.assumeFalse(System.getProperty("os.name").contains("Linux"));
Assert is used to test the functionality.
The most easiest difference between Assert and Assume is :
Assume will only run when the assumption is true. Will be skipped if it false.
assumeTrue(boolean assumption, String message)
Assert will run normally if true.
In case of false assert, it gives predefined error message.
assertTrue(boolean condition, String message)
Simply check out the javadoc for Assume:
A set of methods useful for stating assumptions about the conditions in which a test is meaningful. A failed assumption does not mean the code is broken, but that the test provides no useful information.
In other words: when an assert fires, you know that your testcase failed. Your production code isn't doing what you expect it to do.
Assume means ... you don't know exactly what happened.

Why doesn't Guava have postconditions? What can I use instead?

So, Guava has simple yet useful Preconditions to check method arguments. But I guess it would be reasonable to have a "Postconditions" class too. Or is it just because java provides assertions?
Since a class like this doesn't exist, what is the "best" (practice) alternative way to check postonditions before a mathod returns?
Testing post conditions would be superfluous .
The way we test post-conditions in java is by unit testing.
With unit testing, we make sure that for a given input we get predictable output. With Preconditions, we can verify that we have valid input, and hence the output is already guaranteed by the tests.
I would use the Java assert keyword within the method itself to encode the postcondition.
Unit Test or Postcondition?
Unit tests and postconditions serve different purposes.
An assertion in a unit test provides a check on the result of a method for one input vector. It is an oracle specifying the expected outcome for one specific case.
An assert in the method itself verifies that for any input the postcondition holds. It is an oracle specifying (properties of) the expected outcome for all possible cases.
Such a postcondition-as-oracle combines well with automated testing techniques in which it is easy to generate inputs, but hard to generate the expected value for each input.
Guava Postconditions?
As to why Guava has a Precondition class, but no Postcondition class, here's my understanding.
Guava Preconditions effectively provides a number of shorthands for common situations in which you'd want to throw a particular kind of exception (Illegal argument, null pointer, index out of bounds, illegal state) based on the method's inputs or the object's state.
For postconditions there are fewer such common cases. Hence there is less need to provide a shorthand throwing specific kinds of exceptions. A failing postcondition is like a HTTP 500 "Internal Server Error" -- all we know something went wrong executing our method.
(Note that Guava's notion of precondition is quite different from that of pure design-by-contract, in which there are no guarantees at all if a precondition is not met -- not even that a reasonable exception is thrown. Guava's Preconditions class provides useful capabilities to make a public API more defensive).
Preconditions and postconditions serve very different purposes.
Preconditions test the input, which is not under the method's control; postconditions test the output, which is. Therefore they make no sense inside the method itself, but only as outside code that tests the method.
However, if you really wanted to put such assertions in your code, the Guava Preconditions would serve pretty well for that, too, even if that is not their intended purpose.

Should every possible branch in a method have a separate junit?

This is more of a design question.
Suppose you have a method like this (as an example):
if (x == 5) {
c = 1;
} else {
if (z != 2) {
b = 6;
} else {
a = 3;
}
Do you think it's best practice to have a junit for each possible branch? Ie, testx5, test xnot5znot2, testxnot5z2, etc, or something like:
void testMethod() {
// x is 5
test/assert code;
// x not 5, z not 2
test/assert code;
// x not 5, z is 2
test/assert code
// etc
}
EDIT: Just to be clear, my goal is complete code coverage. I just want to know opinions on whether I should make a new test for each branch or combine them in one test. Thank you for your input.
The JUnit FAQ seems to indicate that it is better to have more tests with fewer assertions, if only because JUnit will only report the first assertion failure in a test method. With one method, if you broke the x = 5 case, you'd have no way to tell if any of the x != 5 cases were still working.
What you're discussing is called Branch Coverage.
The conventional wisdom is if it's important enough to write code to cover that use case, it's important enough to write a test case to cover that code. So this would seem to say that 100% branch coverage is an excellent goal (and will also imply 100% statement coverage, but not necessarily 100% loop or 100% condition coverage).
However, you also need to balance the effort of writing tests with the value of getting those tests. For example, if the code you're testing has a try/catch to catch a checked exception, but the exception is almost never thrown (or difficult to cause to be thrown in a test case), then writing a test to cover that exception is probably not worth your time.
This is why you see in a lot of places that aiming for a certain % of test coverage is a bad idea, because you end up writing test cases to get coverage, not to find bugs. In your example, yes, each branch deserves it's own test case. In every piece of production code, it's probably not necessary.
In unit testing, your goal is to test behaviors -- not the "code". Think of the method you're testing a black box and you want to see if it works correctly. You don't know how it does it's job internally, but you expect certain results for certain inputs. So you'd want to create tests for different cases of how you'd expect the code to work as if you didn't know anything about how the internals of the method actually does it's job. So you'd write tests like "applysDiscountToShoppingCart" and "addsDeliveryFeeToShoppingCart".
Now, all that being said, it's also useful to create "edge cases" in which you're testing things that are likely to break (like nulls, zeros, negatives, data too big/small, etc) to see if it fails in an expected manner too. Usually to write those, you need to know how the method actually works. If you can design tests that will cover all your code, that's great! 100% test coverage is a definite thing to strive for, but it's not always practical (or useful) depending on the situation.
Especially on build servers it is easier to have many different testcases/functions because it will be easy to identify which test fails. Another downside is that the testcase will halt if the first one fails, and you will not know the result of the other testcases.
For me personally this benefit stops when you have to do a lot of copy pasting to set up/explain the testcase, in that case I will just do several asserts in the same test case.

Java JUnit test case

For example, in a method
public void divide(Integer a){
//.....
}
In a Java test, we need to test a parameter as String, null, 0, Long .... , do we have a tool which will automatically test these all cases ?
Thanks
First, the compiler will not let you pass String nor Long so there's no point in unit-testing these.
Second, while there are some tools like the one you're describing, I wouldn't recommend them. Effective unit testing is not about trying to cover as much of the state space as possible (because the state space is infinitely large). It is about the wise choice of the "significant" cases. The one who decides what is considered to be significant is you, the programmer.
Well you can write this yourself using a JUnit TestCase. I don't know of any free tools which will automatically bombard it with all possible inputs to see how it will react, but I do know of the AgitarOne software product (commercial) which does this kind of testing. It's called agitation and lets you explore how your code behaves with nulls, max values, min values, etc.
Why not use JUnit 4 with a Parameterized test case? JUnit 3 did not support parameterized test cases OOTB, but JUnit 4 has specific API to accept arguments and invoke your test case using those parameters as input. Should that be feasible in your case?

Categories

Resources