Develop Reference

How can I test exception in completable future?

I have been converting some code to be asynchronous. The original unit test used the annotation #Test(expected = MyExcpetion.class) but I don't think this will work because the exception I want to assert on is wrapped in java.util.concurrent.ExcutionException . I did try calling my future like this but my assertion is still failing and I don't love that I had to add in return null
myApiCall.get(123).exceptionally((ex) -> {
assertEquals(ex.getCause(),MyCustomException.class)
return null
}
I also tried this flavor but still not working
myApiCall.get(123).exceptionally((ex) -> {
assertThat(ex.getCause())
.isInstanceOF(MyException.class)
.hasMessage("expected message etc")
return null;
}
My API just throws exception if it can't find id. How should I be properly testing this? Can I use that original annotation in anyway?
my api call reaches out to db when run. In this test I am setting up my future to return an error so it doesn't actually try to communicate with anything. the code under test looks like this
public class myApiCall {
public completableFuture get(final String id){
return myService.getFromDB(id)
.thenApply(
//code here looks at result and if happy path then returns it after
//doing some transformation
//otherwise it throws exception
)
}
}
in the unit test I force myService.getFromDB(id) to return bad data so I can test exception and also keep this a unit test don't reach out to db etc.

Let's assume your API throws if called with 0:
public static CompletableFuture<Integer> apiCall(int id) {
return CompletableFuture.supplyAsync(() -> {
if (id == 0) throw new RuntimeException("Please not 0!!");
else return id;
});
}
You can test that it works as expected with the following code (I'm using TestNG but I suspect it won't be too difficult to translate into a JUnit test):
#Test public void test_ok() throws Exception {
CompletableFuture<Integer> result = apiCall(1);
assertEquals(result.get(), (Integer) 1);
}
#Test(expectedExceptions = ExecutionException.class,
expectedExceptionsMessageRegExp = ".*RuntimeException.*Please not 0!!")
public void test_ex() throws Throwable {
CompletableFuture<Integer> result = apiCall(0);
result.get();
}
Note that the second test uses the fact that the ExecutionException message will contain the original exception type and message and captures the expectation with a regex. If you can't do that with JUnit, you can call result.get() in a try/catch block and call throw e.getCause(); in the catch block. In other words, something like this:
#Test(expectedExceptions = RuntimeException.class,
expectedExceptionsMessageRegExp = "Please not 0!!")
public void test_ex() throws Throwable {
CompletableFuture<Integer> result = apiCall(0);
try {
result.get();
} catch (ExecutionException e) {
throw e.getCause();
}
}

You can try also alternative option:
import org.hamcrest.core.IsInstanceOf;
import org.junit.rules.ExpectedException;
public class Test() {
#Rule
public ExpectedException thrown = ExpectedException.none();
#Test
public void myApiCallTest() {
thrown.expect(ExcutionException.class);
thrown.expectCause(IsInstanceOf.instanceOf(MyException.class));
thrown.expectMessage("the message you expected");
myApiCall.get("");
}
}
Assuming that:
public class myApiCall {
public completableFuture get(final String id) {
// ...
throw new ExcutionException(new MyException("the message you expected"))
}
}

Assume that you have a class and you want to test a method which returns a completable future:
public class A {
private final Api api;
public A(Api api) { this.api = api;}
public CompletableFuture<Void> execute(Integer input) {
final CompletableFuture<Void> future = api.execute(input)
.thenApplyAsync(result -> doSomething())
.exceptionally(ex -> doFailure());
return future;
}
}
To test the execution of "doSomething()" then you may use mockito and do the following:
// prepare test
final Api api = mock(Api.class)
final A a = new A(api);
when(api.execute(any(Integer.class)))
.thenReturn(CompletableFuture.completedFuture(null));
// execute
final CompletableFuture<Void> result = a.execute(input);
// validate
...
To test "doFailure" do the following:
when(api.execute(any(Integer.class))).thenAnswer(answer -> {
CompletableFuture<Void> future = new CompletableFuture<>();
future.completeExceptionally(new RuntimeException());
return future;
});
// execute
final CompletableFuture<Void> result = a.execute(input);
// validate
assertTrue(result.isCompletedExceptionally());

that is easy thing doing in junit-4. Are you remember the #RunWith annotation? Yes, write your own TestRunner to intercept the exception before the junit expected exception processor is invoked, for example:
public class ConcurrentRunner extends BlockJUnit4ClassRunner {
public ConcurrentRunner(Class<?> klass) throws InitializationError {
super(klass);
}
#Override
protected Statement possiblyExpectingExceptions(FrameworkMethod method,
Object test,
Statement next) {
return super.possiblyExpectingExceptions(
method, test, throwingActualException(next)
);
}
private Statement throwingActualException(Statement next) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
try {
next.evaluate();
} catch (ExecutionException | CompletionException source) {
throw theActualExceptionOf(source);
}
}
private Throwable theActualExceptionOf(Exception source) {
return source.getCause() != null ? source.getCause() : source;
}
};
}
}
just annotated with #RunWith(ConcurrentRunner.class) on the test, you needn't change your test code at all. for example:
#RunWith(ConcurrentRunner.class)
public class ConcurrentExpectedExceptionTest {
#Test(expected = IllegalArgumentException.class)
public void caughtTheActualException() throws Throwable {
myApiCall().join();
}
private CompletableFuture<Object> myApiCall() {
return CompletableFuture.supplyAsync(() -> {
throw new IllegalArgumentException();
});
}
}

Should assert be placed in the test case or verification method?

For regression testing (not unit testing), where we have elaborate scenarios written in TestNG, is there a proper place the Assert checks should be done? Does it matter or not if it's in the test case, or in a calling method? For example:
This test case calls a validation method that contains the asserts:
#Test
public void test1() {
validateResponse();
}
public void validateResponse() {
Assert.assertEquals(a, "123");
Assert.assertEquals(b, "455");
Assert.assertEquals(c, "5678");
Assert.assertEquals(d, "3333");
}
This test case asserts based on the return value of the verification method:
#Test
public void test1() {
Assert.assertTrue(validateResponse());
}
public boolean void validateResponse() throws Exception {
try {
if (!a.equals("123")) throw new Exception();
if (!b.equals("455")) throw new Exception();
if (!c.equals("5678")) throw new Exception();
if (!d.equals("3333")) throw new Exception();
} catch (Exception e) {
e.printStackTrace();
return false;
}
return true;
}

Your assert should be as specific and granular as possible to help the developer quickly identify the problem. e.g.
#Test
public void testResponseFields(){
// create response to be tested
// JUnit style
Assert.assertEquals("Response 'alpha' should be '123'", 123, response.getAlpha());
// TestNG style
Assert.assertEquals(response.getAlpha(), 123, "Response 'alpha' should be '123'");
}
Once you set a failure message in the Assert.assertXX call, it becomes more of a moot point as to where the Assert is called as you will have a message explaining the problem and a stack trace to see where and when it failed.

Using junit #Rule, expectMessage(), matcher for multipe exception in 1 tested method [duplicate]

This question already has answers here:
How to continue test after JUnit ExpectedException if thrown?
(4 answers)
Closed 6 years ago.
In my Android project I want to test, with the same #Test, a class which can throw several times the same exception with different messages.
I want my test to pass for a given list of messages and to fail for others.
Making some research on Junit I tried to implement this using #Rule, expectMessage() and Hamcrest matcher.
My implementation is currently based on the "Custom matcher" described here.
#RunWith(AndroidJUnit4.class)
public class TestException extends ApplicationTestCase {
#Rule public ExpectedException thrown= ExpectedException.none();
public TestException(){
super(AplicatyApplication.class);
}
#Test
public void testException() throws Exception {
thrown.expect(IndexOutOfBoundsException.class);
thrown.expectMessage(new MatchesPattern("*"));
Dummy.exec(0);
// do more stuff here ...
Dummy.exec(1);
// ...
Dummy.exec(2);
// ...
Dummy.exec(3); // I want my test to fail here
// ...
}
class MatchesPattern extends TypeSafeMatcher<String> {
private String pattern;
public MatchesPattern(String pattern) {
this.pattern = pattern;
}
#Override
protected boolean matchesSafely(String item) {
return item.matches(pattern)
&&
item.startsWith("My message")
&& (
item.endsWith("1")
||
item.endsWith("2")
);
}
#Override
public void describeTo(Description description) {
description.appendText("matches pattern ").appendValue(pattern);
}
#Override
protected void describeMismatchSafely(String item, Description mismatchDescription) {
mismatchDescription.appendText("does not match");
}
}
static class Dummy {
static void exec(int i){
if(i == 0)
return;
if(i == 1)
throw new IndexOutOfBoundsException("My message1");
if(i == 2)
throw new IndexOutOfBoundsException("My message2");
if(i == 3)
throw new IndexOutOfBoundsException("My message3");
}
}
}
Running this test I can see that the matcher is called just once, executing Dummy.exec(1);.
The matchesSafely(String item) returns true and the the test ends with the status Passed.
All this seems to be Okay, with my understanding of the #Rule. I was waiting an exception : I got it; I was waiting a given message : I got it.
I can not find a way to continue the execution of my test once the first exception has been thrown.
My questions are :
Is it possible to use #Rule to check more than one exception thrown into one tested method, or do I have to use the typical try/catch testing the exception message in every catch block?.
Is there another/more elegant way to test this type of concerns.

I suggest to split the test method into multiple tests, one for each requirement.
#Test
public void testException_1() throws Exception {
thrown.expect(IndexOutOfBoundsException.class);
thrown.expectMessage("My message1");
Dummy.exec(1);
}
In case it needs to be in one test method, I would build it with try-catch and ErrorCollector.
#Test
public void testException_1() throws Exception {
try {
Dummy.exec(1);
fail();
} catch (IndexOutOfBoundsException e) {
errorCollector.checkThat(e.getMessage(), is("My message1"));
}
try {
Dummy.exec(2);
...
} ...
}
I would try to avoid to build a custom Matcher.

Checking for 2 expected values in Junit

I have a java program which throws an exception with 2 different messages for 2 different scenarios and I want the Junit test case to check for equality for both of these messages. As an example -
public void amethod() {
// do some processing
if(scenario1 == true) {
throw new MySystemException("An error occured due to case 1 being incorrect.");
}
else if(scenario2 == true) {
throw new MySystemException("An error occured as case 2 could not be found");
}
}
Now the JUnit for this would be something like-
public void testAMethod() {
// do something
assertEquals("Expected", "Actual");
}
As I understand, in this above example, if I use the Scenario1 exception message the junit will fail when an exception is thrown for Scenario2 and vice versa.
I would like to know if there is any other way provided in Junit by which I can use this one test method and check for both the messages for the test to pass?
Something like an OR, if possible to provide the "Expected" value with both these expected message.
I hope my query is clear enough.
Thanks
UPDATE
Sorry for the delayed response, had got caught up with some other urgent matter.
Thank you all for the very nice suggestions, it certainly has helped me to understand a bit better now.
Eventually, to keep it rather simple I decided to implement a somewhat similar solution suggested by Don Roby. So created a new test class which looks like -
public void testAMethodScenario1() {
// do the necessary
assertEquals("Expected Exception Message 1", "Actual");
}
public void testAMethodScenario2() {
// do the necessary
assertEquals("Expected Exception Message 2", "Actual");
}
Thank you all again for your responses.

I think you need to manually catch the exception (for each scenario) and individually check the message:
try {
// trigger scenario 1
fail("An exception should have been thrown here !");
} catch (MySystemException e1) {
assertEquals("Wrong error message", m1, e1.getMessage());
}
try {
// trigger scenario 2
fail("An exception should have been thrown here !");
} catch (MySystemException e2) {
assertEquals("Wrong error message", m2, e2.getMessage());
}
Of course, you can have these scenarios defined as enum constants and simply iterate through them and check each of them within a loop, since the "copy/paste design pattern" is pretty obvious in the above code. :)

You seem to be asking two things here, how to test an exception and how to assert that a value matches either of two possible expected values.
To test for an exception, you can either use a JUnit4 annotation:
#Test(expected=MySystemException.class)
public void testException() {
amethod();
}
or use a try-catch in your test:
#Test
public void testException() {
try {
amethod();
fail("MySystemException expected");
}
catch (MySystemException e) {
// Success!
}
}
And if you have only one message, in the try-catch version you can assert that you got it with an AssertEquals in the catch block.
The best testing would have separate tests for your two scenarios, and expect the correct single message. Better code might in fact have distinct exceptions for the two situations.
But the need for a more complex assertion than simple equality does come up anyway, and there's an elegant solution for it in Hamcrest matchers.
Using that for this situation, you could write something like (untested - don't trust my syntax completely):
#Test
public void testException() {
try {
amethod();
fail("MySystemException expected");
}
catch (MySystemException e) {
String expectedMessage1 = "An error occured due to case 1 being incorrect.";
String expectedMessage2 = "An error occured as case 2 could not be found";
assertThat(e.getMessage(),
anyOf(equalTo(expectedMessage1), equalTo(expectedMessage2)));
}
}

Can you predict which scenario will occur? If so, Costi's answer is correct. If not, because there's some randomness or whatever, you can write:
#Test
public void testAmethodThrowsException() {
try {
amethod();
fail("amethod() should have thrown an exception");
}
catch (MySystemException e) {
String msg = e.getMessage();
assertTrue("bad message: " + msg, msg.equals("An error occured due to case 1 being incorrect.") || msg.equals("An error occured as case 2 could not be found"));
}
}

The declared types of exception thrown bya method are part of its API. If you really want to distinguish different failure modes, you should declare a different exception type for each failure mode.
So, something like this:
/**
* Do something.
* #throws MySystemException1 in case 1.
* #throws MySystemException2 if Foo not found.
*/
public void amethod() {
// do some processing
if(scenario1 == true) {
throw new MySystemException1("Case 1.");
}
else if(scenario2 == true) {
throw new MySystemException2("Foo not found");
}
}

#Rule solution in JUnit4:
public class ExceptionRule implements MethodRule {
#Override
public Statement apply(final Statement base, final FrameworkMethod method, Object target) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
try {
base.evaluate();
Assert.fail();
} catch (MySystemException e) {
if(scenario1)
assertEquals("Expected error message1", e1.getMessage();
if(scenario2)
assertEquals("Expected error message2",e1.getMessage();
}
}
};
}
}
In your testcase, use the Rule:
#Rule public ExceptionRule rule = new ExceptionRule();

JUnit 4 provides (Expected Exception.class)
#Test(expected= MySystemException.class) public void empty() {
// what ever you want
}
Google: Expected Exceptions JUnit for more info.

BDD Style Solution with Catch Exception
#Test
public void testAMethodScenario1() {
//given scenario 1
when(foo).amethod();
then(caughtException())
.isInstanceOf(MySystemException.class)
.hasMessage("An error occured due to case 1 being incorrect.");
}
#Test
public void testAMethodScenario2() {
//given scenario 2
when(foo).amethod();
then(caughtException())
.isInstanceOf(MySystemException.class)
.hasMessage("An error occured as case 2 could not be found");
}
Source code
https://gist.github.com/mariuszs/7490875
Dependencies
com.googlecode.catch-exception:catch-exception:1.2.0

A better solution with #Rule, you can assert both exception and expection message as well.
#Rule
public ExpectedException expectedException = ExpectedException.none();
#Test
public void aMethod_Scenario1True_ThrowsException() {
expectedException.expect(MySystemException.class);
expectedExcepion.expectMessage("An error occured due to case 1 being incorrect.");
//when().thenReturn();
//handle the repositories, static methods and other sub methods, if needed
amethod();
}
#Rule is the more elegant way to write the exception.

How do I assert my exception message with JUnit Test annotation?

I have written a few JUnit tests with #Test annotation. If my test method throws a checked exception and if I want to assert the message along with the exception, is there a way to do so with JUnit #Test annotation? AFAIK, JUnit 4.7 doesn't provide this feature but does any future versions provide it? I know in .NET you can assert the message and the exception class. Looking for similar feature in the Java world.
This is what I want:
#Test (expected = RuntimeException.class, message = "Employee ID is null")
public void shouldThrowRuntimeExceptionWhenEmployeeIDisNull() {}

You could use the #Rule annotation with ExpectedException, like this:
#Rule
public ExpectedException expectedEx = ExpectedException.none();
#Test
public void shouldThrowRuntimeExceptionWhenEmployeeIDisNull() throws Exception {
expectedEx.expect(RuntimeException.class);
expectedEx.expectMessage("Employee ID is null");
// do something that should throw the exception...
System.out.println("=======Starting Exception process=======");
throw new NullPointerException("Employee ID is null");
}
Note that the example in the ExpectedException docs is (currently) wrong - there's no public constructor, so you have to use ExpectedException.none().

In JUnit 4.13 you can do:
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThrows;
...
#Test
void exceptionTesting() {
IllegalArgumentException exception = assertThrows(
IllegalArgumentException.class,
() -> { throw new IllegalArgumentException("a message"); }
);
assertEquals("a message", exception.getMessage());
}
This also works in JUnit 5 but with different imports:
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
...

I like the #Rule answer. However, if for some reason you don't want to use rules. There is a third option.
#Test (expected = RuntimeException.class)
public void myTestMethod()
{
try
{
//Run exception throwing operation here
}
catch(RuntimeException re)
{
String message = "Employee ID is null";
assertEquals(message, re.getMessage());
throw re;
}
fail("Employee Id Null exception did not throw!");
}

Do you have to use #Test(expected=SomeException.class)? When we have to assert the actual message of the exception, this is what we do.
#Test
public void myTestMethod()
{
try
{
final Integer employeeId = null;
new Employee(employeeId);
fail("Should have thrown SomeException but did not!");
}
catch( final SomeException e )
{
final String msg = "Employee ID is null";
assertEquals(msg, e.getMessage());
}
}

Actually, the best usage is with try/catch. Why? Because you can control the place where you expect the exception.
Consider this example:
#Test (expected = RuntimeException.class)
public void someTest() {
// test preparation
// actual test
}
What if one day the code is modified and test preparation will throw a RuntimeException? In that case actual test is not even tested and even if it doesn't throw any exception the test will pass.
That is why it is much better to use try/catch than to rely on the annotation.

I never liked the way of asserting exceptions with Junit. If I use the "expected" in the annotation, seems from my point of view we're violating the "given, when, then" pattern because the "then" is placed at the top of the test definition.
Also, if we use "#Rule", we have to deal with so much boilerplate code. So, if you can install new libraries for your tests, I'd suggest to have a look to the AssertJ (that library now comes with SpringBoot)
Then a test which is not violating the "given/when/then" principles, and it is done using AssertJ to verify:
1 - The exception is what we're expecting.
2 - It has also an expected message
Will look like this:
#Test
void should_throwIllegalUse_when_idNotGiven() {
//when
final Throwable raisedException = catchThrowable(() -> getUserDAO.byId(null));
//then
assertThat(raisedException).isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("Id to fetch is mandatory");
}

Raystorm had a good answer. I'm not a big fan of Rules either. I do something similar, except that I create the following utility class to help readability and usability, which is one of the big plus'es of annotations in the first place.
Add this utility class:
import org.junit.Assert;
public abstract class ExpectedRuntimeExceptionAsserter {
private String expectedExceptionMessage;
public ExpectedRuntimeExceptionAsserter(String expectedExceptionMessage) {
this.expectedExceptionMessage = expectedExceptionMessage;
}
public final void run(){
try{
expectException();
Assert.fail(String.format("Expected a RuntimeException '%s'", expectedExceptionMessage));
} catch (RuntimeException e){
Assert.assertEquals("RuntimeException caught, but unexpected message", expectedExceptionMessage, e.getMessage());
}
}
protected abstract void expectException();
}
Then for my unit test, all I need is this code:
#Test
public void verifyAnonymousUserCantAccessPrivilegedResourceTest(){
new ExpectedRuntimeExceptionAsserter("anonymous user can't access privileged resource"){
#Override
protected void expectException() {
throw new RuntimeException("anonymous user can't access privileged resource");
}
}.run(); //passes test; expected exception is caught, and this #Test returns normally as "Passed"
}

If using #Rule, the exception set is applied to all the test methods in the Test class.

I would prefer AssertJ for this.
assertThatExceptionOfType(ExpectedException.class)
.isThrownBy(() -> {
// method call
}).withMessage("My message");

I like user64141's answer but found that it could be more generalized. Here's my take:
public abstract class ExpectedThrowableAsserter implements Runnable {
private final Class<? extends Throwable> throwableClass;
private final String expectedExceptionMessage;
protected ExpectedThrowableAsserter(Class<? extends Throwable> throwableClass, String expectedExceptionMessage) {
this.throwableClass = throwableClass;
this.expectedExceptionMessage = expectedExceptionMessage;
}
public final void run() {
try {
expectException();
} catch (Throwable e) {
assertTrue(String.format("Caught unexpected %s", e.getClass().getSimpleName()), throwableClass.isInstance(e));
assertEquals(String.format("%s caught, but unexpected message", throwableClass.getSimpleName()), expectedExceptionMessage, e.getMessage());
return;
}
fail(String.format("Expected %s, but no exception was thrown.", throwableClass.getSimpleName()));
}
protected abstract void expectException();
}
Note that leaving the "fail" statement within the try block causes the related assertion exception to be caught; using return within the catch statement prevents this.

Import the catch-exception library, and use that. It's much cleaner than the ExpectedException rule or a try-catch.
Example form their docs:
import static com.googlecode.catchexception.CatchException.*;
import static com.googlecode.catchexception.apis.CatchExceptionHamcrestMatchers.*;
// given: an empty list
List myList = new ArrayList();
// when: we try to get the first element of the list
catchException(myList).get(1);
// then: we expect an IndexOutOfBoundsException with message "Index: 1, Size: 0"
assertThat(caughtException(),
allOf(
instanceOf(IndexOutOfBoundsException.class),
hasMessage("Index: 1, Size: 0"),
hasNoCause()
)
);

#Test (expectedExceptions = ValidationException.class, expectedExceptionsMessageRegExp = "This is not allowed")
public void testInvalidValidation() throws Exception{
//test code
}