Develop Reference

Java test with expected = exception fails with assertion error [duplicate]

This question already has answers here:
Why won't this Expect Exception Junit Test Work?
(2 answers)
Closed 6 years ago.
I'm having an issue with a test case.
The method being tested has a try / catch that catches a MalformedURLException but during testing I get a failure due to a Junit AssertionError that expects a MalformedURLException. But I can't find out what it is actually throwing! Here is my code (created as a MWE in eclipse).
My method that I want to test
public void throwMalFormedURLException(){
String s = new String("www.google.com");
try{
url = new URL(s);
}catch (Exception e){
e.printStackTrace();
e.getClass();
e.getCause();
}
}
the test method
#Test (expected = MalformedURLException.class)
public void testThrowMalFormedURLException() {
MWE testClass = new MWE();
testClass.throwMalFormedURLException();
System.out.println("End of test");
}
This is the output in the console
End of test error details are: java.net.MalformedURLException: no protocol: www.google.com
at java.net.URL.(URL.java:593)
at java.net.URL.(URL.java:490)
at java.net.URL.(URL.java:439)
at MWE.throwMalFormedURLException(MWE.java:12)
at testMWE.testThrowMalFormedURLException(testMWE.java:12)
In the Junit console it says :
java.lang.AssertionError: Expected exception: Java.net.MalformedURLException
But the Junit is reporting failure, even though the console is telling me I've got a MalformedURLException.
What am I doing wrong with this test ?
Thanks for your thoughts.
David

You are catching the exception and therefore it is not being thrown.
If your intent is to test that you are capturing the exception and relaying it back to the 'user' properly, you should create tests for that specifically. You probably don't want UI elements in your unit tests* so this is a place where abstraction and DI have a lot of value. One simple approach is to create a mock object that will listen for the error message and mark a flag when the message is received. Your unit test trigger the error and then pass if the flag is set. You should also have a negative test or assert that the flag is not set prior to throwing the exception.
*Testing the UI is also a good idea but it can be a little slow. There are various tools for automating that. It generally falls in to a different phase of testing. You really want the unit tests to be really fast so that you can run them very frequently.

You have written production code that simply isn't testable. And more specifically: this code doesn't have any "programmatically" observable side effects in the first place.
Meaning: when you write "production code", the code within a method can do three things that could be observed:
Make method calls on objects that are fields of the class under test
Return some value
Throw an exception
For each of these options, you might be able to write testing code:
Using dependency injection, you can put a mocked object into your class under test. And then you can check that the expected methods are invoked on your mock object; with the parameter that you would expect.
You compare the result you get from calling that method with some expected value
You use expected to ensure that a specific exception was thrown
When you look at your code; you can see: it does none of that. It only operates on objects that are created within the method. It doesn't return any value. And, most importantly: it doesn't throw an exception!
Long story short: a caught exception isn't "leaving" the method. It is caught there, and the method ends normally. The fact that you print details about the caught exception doesn't change that.
So, the first thing you have to do: remove the whole try/catch from your production code!
And, if you want to have a more specific test, you can do something like:
#Test
public void testException() {
try {
new MWE().throwMalFormedURLException();
fail("should have thrown!");
} catch ( MalFormedURLException me ) {
assertThat(me.getMessage(), containsString("whatever"));
}
The above:
fails when no exception is thrown
fails when any other exception than MalFormedURLException is thrown
allows you to check further properties of the thrown exception

This is a valid test failure. The test asserts that calling throwMalFormedURLException() will throw MalformedURLException, but since you're catching the exception, it doesn't throw it, so the test fails.

How to capture all uncaucht exceptions on junit tests?

If we have created a singleton object to handle an Java Exceptions, why Thread.setDefaultUncaughtExceptionHandler runs ok in Java Application Server, Java Console Application but not works on JUnit tests?
For example, the following code works:
public class Main extends Object {
public static void main(String[] arguments) {
Thread.setDefaultUncaughtExceptionHandler(UncaughtExceptionHandler.getInstance());
double a = 1/0;
}
}
but this JUnit test not:
public class UncaughtExceptionHandlerTest {
#Test
public void throwException() {
Thread.setDefaultUncaughtExceptionHandler(UncaughtExceptionHandler.getInstance());
double a = 1/0;
}
}
but why? And, how can we solve this, to automatically handle all JUnit test exceptions without using a moody try catch to each test?

The JUnit will be catching all unexpected exceptions that are thrown by the unit tests on the unit test threads1. The normal behavior is to catch / display / record the exception as a FAILed test, and then continue with the next unit test.
This means that the there is no "uncaught exception" in the Java sense, and your uncaught exception handler is not going to be called.
It is not entirely clear what you are trying to achieve here, but I suspect that the answer would be to implement a custom runner:
https://github.com/junit-team/junit4/wiki/Test-runners
1 - If the code under test spawns its own threads, the JUnit framework has no way of knowing. It certainly cannot catch / detect uncaught exceptions on those threads. However, this doesn't seem to be what you are talking about in this question.
The main motivation, is, for example, send an e-mail or perform another administrative tasks if a junit test fail. If I have a global exception handler I could do this, instead put a catch block to each test. After the handling, maybe I will throw this exception and let junit go ahead as it does.
Well if that is what you are trying to do, then you are (IMO) doing it the wrong way. There are existing runners that provide a structured report file, or a report data structure that can give you a list of all tests that passed, failed from an assertion, failed from an exception, etc. What you should do is:
choose an appropriate runner
analyse its output
send a single email (or whatever) if there are errors that meet your criteria.
Advantages:
less effort
you deal with all errors not just uncaught exceptions (though actually assertion failures manifest as AssertionError exceptions ...)
you don't spam yourself on each and every failed test.
And there's another way. Look at JUnitCore (link). This allows you register a listener for the various test events, and then run a bunch of tests or test suites.
The other point is that you appear to be trying to duplicate (some of) the functionality of a Continuous Integration (CI) server such as Jenkins.
You then asked why this doesn't work:
#Test
public void throwException() {
Thread.setDefaultUncaughtExceptionHandler(/* some handler */));
double a = 1/0;
}
An uncaught exception handler is only invoked if nothing else catches the exception. But a typical JUnit test runner catches all exceptions that propagate from each unit test using a conventional exception handler. That means that the ArithmeticException thrown in your test never reaches your handler.

Exceptions thrown by your junit #Test method are not uncaught. JUnit catches them and uses them to fail your tests.
Now, if you had started a new Thread of your own that is not running inside JUnit's try/catch execution, a thrown exception will be essentially ignored and your test will pass.
Just think of the name... Thread.setDefaultUncaughtExceptionHandler. This only covers threads that do not explicitly have an uncaught exception handler, and then it doesn't cover exceptions that are caught by the code calling your code (JUnit, etc).
Here is relevant code from ParentRunner class:
protected final void runLeaf(Statement statement,
Description description, RunNotifier notifier) {
EachTestNotifier eachNotifier = new EachTestNotifier(notifier, description);
eachNotifier.fireTestStarted();
try {
statement.evaluate();
} catch (AssumptionViolatedException e) {
eachNotifier.addFailedAssumption(e);
} catch (Throwable e) {
eachNotifier.addFailure(e);
} finally {
eachNotifier.fireTestFinished();
}

Are you sure that jUnit isn't catching it somewhere? The method signature says that it throws Exception so I'd guess that there has to be a pretty broad catch statement up-stream.

Try catch in a JUnit test

I'm writing unit tests for an application that already exists for a long time. Some of the methods I need to test are build like this:
public void someMethod() throws Exception {
//do something
}
If I want to test these methods I have to write something like this in my unit test:
#Test
public void someTest() {
try {
someMethod();
}
catch (Exception e) {
e.printStackTrace();
}
}
Is it a good practice to do this? Or is there an other way to test these methods?
I did some research on the internet and I found a few solutions with the #Rule annotation and #Test(expected=Exception.class), but that's not working (Eclipse keeps showing the someMethod() line in the test as wrong).
I don't know if these are good solutions, because I'm pretty new to the whole unit testing story.
If someone who knows a lot about this could help me out, I would be really thankful.

Since Exception is a checked exception, you either:
Have to catch the exception in a try...catch statement, or
Declare the exception to be thrown in the method itself.
What you have up there works fine, but my personal preference is to declare the exception to be thrown. This way, if an exception I'm not expecting is thrown during the run of the test, the test will fail.
#Test
public void someTest() throws Exception {
// dodgy code here
}
If we need to see if a specific exception is thrown, then you have the option of using #Rule or adding the value to the #Test annotation directly.
#Test(expected = FileNotFoundException.class)
public void someTest() throws Exception {
// dodgy code here
}
In JUnit 5, you can leverage Assertions.assertThrows to accomplish the same thing. I'm less familiar with this overall since it's not yet GA at the time of editing, but it appears to accept an Executable coming from JUnit 5.
#Test
public void someTest() {
assertThrows(FileNotFoundException.class, () ->
{ dodgyService.breakableMethod() };
}

#Test
public void someTest() {
try {
someMethod();
}
catch (Exception e) {
Assert.fail("Exception " + e);
}
}
Is what you can do, if the exception should not occur. An alternative would be to throw the exception in the signature like this:
#Test
public void someTest() throws Exception {
someMethod();
}
The difference is, that in one case the test will fail with an assertion exception and in the other case it will fail because the test crashed. (like somewhere in your code you get a NPE and the test will because of that)
The reason you have to do this, is because Exception is a checked exception. See Checked versus unchecked exception
The #Test(expected=Exception.class) is for tests, that want to test that the exception will be thrown.
#Test(expected=ArrayIndexOutOfBounds.class)
public void testIndex() {
int[] array = new int[0];
int var = array[0]; //exception will be thrown here, but test will be green, because we expect this exception
}

Do not catch your application's exception in your test code. Instead, declare it to be thrown upwards.
Because, when JUnit's TestRunner finds an exception thrown, it will automatically log it as an error for the testcase.
Only if you testcase expects that the method should thrown an Exception you should use #Test(expected=Exception.class) or catch the exception.
In other cases, just throw it upwards with,
public void someTest() throws Exception {

You can add exception in test method signature. Then, if you are testing whether exception is thrown, you have to use #Test(expected=Exception.class). In the test cases where exception has not to be thrown, test will pass successfully.
#Test
public void testCaseWhereExceptionWontBeThrown() throws Exception {
someMethod(); //Test pass
}
#Test(expected = Exception.class)
public void testCaseWhereExceptionWillBeThrown() throws Exception {
someMethod(); //Test pass
}

There are two main rules on how to process exceptions at Junit testers:
If the exception was originated into the tested code:
If it was expected, declare it in the expected attribute of the Test annotation. Or, if further checks should be done on the exception object itself, catch it and ignore it. (In this case, there must be also a call to Assert.fail at the end of the try block, to indicate that the expected exception was not produced).
If it was not expected, catch it and execute Assert.fail. (A previous call to Exception.printStackTrace is also useful).
If the exception was not originated into the tested code or it is not interesting to the test (for example, most of the IOExceptions are produced at network level, before the test could even be completed), rethrow it at the throws clause.
Why you should expect an exception in the tester? Remind: You should code one test method for every possible result on the tested code (in order to achieve a high code coverage): In your case, one method that must return successfully, and at least another one that must produce an Exception.

Three points about JUnit:
Tests should be precise, they should pass or fail unambiguously based solely on how the test inputs are set up.
Tests should have failures reported back into the framework.
Tests should not rely on having their output read.
Your example fails on all three counts. If an exception gets thrown or not, the test still passes. If an exception is thrown JUnit never finds out about it and can't include it in the test results. The only way to know something went wrong is to read what the test writes to stdout, which makes errors too easy to ignore. This is not a useful way to write tests.
JUnit was designed to make doing the right thing easy and to give developers useful feedback. If an exception gets thrown from a test method, it gets caught by the framework. If the test was annotated with an exception indicating that exception is expected, then the framework marks the test as passing. Otherwise the framework fails the test and records the stacktrace for reporting. The framework reports what assertions fail and what unexpected exceptions occurred so that everybody knows if the tests worked or not.
If you expect a test to succeed without throwing an exception, then if anything in the test can throw a checked exception, add throws Exception to the test method signature. Adding the throws to the signature doesn't say the method has to throw anything, it just lets any exceptions that happen to occur get thrown so that the test framework can catch them.
The only instance where you would actually catch the exception in the test is where you want to test assertions about the exception; for instance, you could test that the message on the exception is what you expect, or if the exception has a cause set on it. In that case you would add Assert.fail() at the end of the try-block so that not having an exception thrown will cause the test to fail.
It isn’t having a try-catch block that is so bad, it’s the absence of anything that will cause the test to fail that is bad.
When you write a test at first, make it fail. That way you prove to yourself that you know what the test is doing, and you confirm that, when there is a failure, you will be made aware of it.

What kind of exception is it? Is it
an exception from doing something like using streams that won't happen in your unit test or
an exception that can happen because of some kind of bad input?
If it's 1. I would just put it at the method signature level because a try-catch is serving no real purpose other than ceremony.
#Test
public void testFoo() throws Exception {
// ...
}
If it's 2. it becomes a little more complicated. You need to ask yourself what should be happening if the Exception is thrown. Should the test fail? Is it expected? Is it irrelevant? Examples below of how to handle all of these. BEWARE: I only used Exception because you did. I hope it really isn't though because if it's possible for some other exception to be thrown other than the expected then these will be very wonky. If possible don't use Exception, use something more specific (in the junit and code).
// The below code assumes you've imported the org.junit.Assert class.
#Test
public void thisShouldFailIfExceptionCaught() {
//Given...
try {
// When...
} catch (Exception e) {
Assert.fail();
}
// Then...
}
#Test
public void thisShouldPassOnlyIfTheExceptionIsCaught() {
//Given...
try {
// When...
Assert.fail();
} catch (Exception expected) {}
// No "then" needed, the fact that it didn't fail is enough.
}
#Test
public void irrelevantExceptionThatCouldBeThrown() {
//Given...
try {
// When...
} catch (Exception e) {}
// Then...
}

How to combine logging and unit-testing in java?

I've just started my very first toy-project in java and faced with misunderstanding of how it should be done. I'm using java.util.logging and JUnit4 library.
For example we have something like this:
public class SomeClass {
private static Logger log = Logger.getLogger(SomeClass.class.getName());
public static void SomeMethod() {
try{
...some code...
} catch(Exception e){
log.warning("Something bad happened");
}
}
And the unit-test will be:
#Test
public void SomeClassTest(){
SomeClass.SomeMethod();
}
But there will never be an exception, cause I've already handled it in method.
Should I generate new exception in catch-block? Or may be using junit combined with logging is not a good idea?

A method that does not throw an exception (and returns the expected value if any) is meant to work correctly from the perspective of a user.
So you should use try - catch and logging inside a method, when you can catch an exception and the method will still work correctly (do something else when this error happens for example but still return the expected result or perform the supposed operation).
In this case the unit test should check if the operation was performed correctly (if the object is in the expected state and the result value (in your case void) is correct
You should rethrow the exception (and usually not log it, but that depends) if the method cannot do what it is supposed to do when the exception occurs.
In this case the unit test should check if the operation was performed correctly (if the object is in the expected state and the result value (in your case void) is correct if there is no exception, and if there is an exception it should check if this exception was expected

If you want to test that the exception is thrown then you would have to re-throw, or not catch, the Exception.
Otherwise you can unit test that the class is in the correct state after the exception, i.e. that the exception was correctly handled.
I would say one other thing. Don't catch(Exception e), catch the specific exception you are expecting. Otherwise you will handle other, unexpected, exceptions in the same way and that is really quite dangerous.

You can simply rethrow the caught exception:
public class SomeClass {
private static Logger log = Logger.getLogger(SomeClass.class.getName());
public static void SomeMethod() {
try {
// your stuff
} catch (Exception e) {
log.warning("Something happened");
throw e;
}
}
}

Should I generate new exception in catch-block?
No. don't do that! you can test your existing code! when you only want to log the message but you don't want to handle it in the method that call someMethod() don't throw it!
using junit combined with logging is not a good idea?
both are good ideas and can be used together without problems.
Think about how you can test your method. i would not modify the code just that you can easily test it. because you WANTED to catch the exception and log for a reason.
try verifing what variables or objects are modified in your test-method

JUnit4 fail() is here, but where is pass()?

There is a fail() method in JUnit4 library. I like it, but experiencing a lack of pass() method which is not present in the library. Why is it so?
I've found out that I can use assertTrue(true) instead but still looks unlogical.
#Test
public void testSetterForeignWord(){
try {
card.setForeignWord("");
fail();
} catch (IncorrectArgumentForSetter ex){
}
// assertTrue(true);
}

Call return statement anytime your test is finished and passed.

As long as the test doesn't throw an exception, it passes, unless your #Test annotation specifies an expected exception. I suppose a pass() could throw a special exception that JUnit always interprets as passing, so as to short circuit the test, but that would go against the usual design of tests (i.e. assume success and only fail if an assertion fails) and, if people got the idea that it was preferable to use pass(), it would significantly slow down a large suite of passing tests (due to the overhead of exception creation). Failing tests should not be the norm, so it's not a big deal if they have that overhead.
Note that your example could be rewritten like this:
#Test(expected=IncorrectArgumentForSetter.class)
public void testSetterForeignWord("") throws Exception {
card.setForeignWord("");
}
Also, you should favor the use of standard Java exceptions. Your IncorrectArgumentForSetter should probably be an IllegalArgumentException.

I think this question needs an updated answer, since most of the answers here are fairly outdated.
Firstly to the OP's question:
I think its pretty well accepted that introducing the "expected excepetion" concept into JUnit was a bad move, since that exception could be raised anywhere, and it will pass the test. It works if your throwing (and asserting on) very domain specific exceptions, but I only throw those kinds of exceptions when I'm working on code that needs to be absolutely immaculate, --most APIS will simply throw the built in exceptions like IllegalArgumentException or IllegalStateException. If two calls your making could potentitally throw these exceptions, then the #ExpectedException annotation will green-bar your test even if its the wrong line that throws the exception!
For this situation I've written a class that I'm sure many others here have written, that's an assertThrows method:
public class Exceptions {
private Exceptions(){}
public static void assertThrows(Class<? extends Exception> expectedException, Runnable actionThatShouldThrow){
try{
actionThatShouldThrow.run();
fail("expected action to throw " + expectedException.getSimpleName() + " but it did not.");
}
catch(Exception e){
if ( ! expectedException.isInstance(e)) {
throw e;
}
}
}
}
this method simply returns if the exception is thrown, allowing you to do further assertions/verification in your test.
with java 8 syntax your test looks really nice. Below is one of the simpler tests on our model that uses the method:
#Test
public void when_input_lower_bound_is_greater_than_upper_bound_axis_should_throw_illegal_arg() {
//setup
AxisRange range = new AxisRange(0,100);
//act
Runnable act = () -> range.setLowerBound(200);
//assert
assertThrows(IllegalArgumentException.class, act);
}
these tests are a little wonky because the "act" step doesn't actually perform any action, but I think the meaning is still fairly clear.
there's also a tiny little library on maven called catch-exception that uses the mockito-style syntax to verify that exceptions get thrown. It looks pretty, but I'm not a fan of dynamic proxies. That said, there syntax is so slick it remains tempting:
// given: an empty list
List myList = new ArrayList();
// when: we try to get the first element of the list
// then: catch the exception if any is thrown
catchException(myList).get(1);
// then: we expect an IndexOutOfBoundsException
assert caughtException() instanceof IndexOutOfBoundsException;
Lastly, for the situation that I ran into to get to this thread, there is a way to ignore tests if some conidition is met.
Right now I'm working on getting some DLLs called through a java native-library-loading-library called JNA, but our build server is in ubuntu. I like to try to drive this kind of development with JUnit tests --even though they're far from "units" at this point--. What I want to do is run the test if I'm on a local machine, but ignore the test if we're on ubuntu. JUnit 4 does have a provision for this, called Assume:
#Test
public void when_asking_JNA_to_load_a_dll() throws URISyntaxException {
//this line will cause the test to be branded as "ignored" when "isCircleCI"
//(the machine running ubuntu is running this test) is true.
Assume.assumeFalse(BootstrappingUtilities.isCircleCI());
//an ignored test will typically result in some qualifier being put on the results,
//but will also not typically prevent a green-ton most platforms.
//setup
URL url = DLLTestFixture.class.getResource("USERDLL.dll");
String path = url.toURI().getPath();
path = path.substring(0, path.lastIndexOf("/"));
//act
NativeLibrary.addSearchPath("USERDLL", path);
Object dll = Native.loadLibrary("USERDLL", NativeCallbacks.EmptyInterface.class);
//assert
assertThat(dll).isNotNull();
}

I was looking for pass method for JUnit as well, so that I could short-circuit some tests that were not applicable in some scenarios (there are integration tests, rather than pure unit tests). So too bad it is not there.
Fortunately, there is a way to have a test ignored conditionally, which actually fits even better in my case using assumeTrue method:
Assume.assumeTrue(isTestApplicable);
So here the test will be executed only if isTestApplicable is true, otherwise test will be ignored.

There is no need for the pass method because when no AssertionFailedException is thrown from the test code the unit test case will pass.
The fail() method actually throws an AssertionFailedException to fail the testCase if control comes to that point.

I think that this question is a result of a little misunderstanding of the test execution process. In JUnit (and other testing tools) results are counted per method, not per assert call. There is not a counter, which keeps track of how many passed/failured assertX was executed.
JUnit executes each test method separately. If the method returns successfully, then the test registered as "passed". If an exception occurs, then the test registered as "failed". In the latter case two subcase are possible: 1) a JUnit assertion exception, 2) any other kind of exceptions. Status will be "failed" in the first case, and "error" in the second case.
In the Assert class many shorthand methods are avaiable for throwing assertion exceptions. In other words, Assert is an abstraction layer over JUnit's exceptions.
For example, this is the source code of assertEquals on GitHub:
/**
* Asserts that two Strings are equal.
*/
static public void assertEquals(String message, String expected, String actual) {
if (expected == null && actual == null) {
return;
}
if (expected != null && expected.equals(actual)) {
return;
}
String cleanMessage = message == null ? "" : message;
throw new ComparisonFailure(cleanMessage, expected, actual);
}
As you can see, in case of equality nothing happens, otherwise an excepion will be thrown.
So:
assertEqual("Oh!", "Some string", "Another string!");
simply throws a ComparisonFailure exception, which will be catched by JUnit, and
assertEqual("Oh?", "Same string", "Same string");
does NOTHING.
In sum, something like pass() would not make any sense, because it did not do anything.