I have Spring boot application. I use Junit+Mockito to unit test it. All the test cases were written using Java. I recently made a decision to write test cases using Groovy though the application code will remain in Java.
I encountered a weird scenario while testing expected exceptions.
Scenario 1: Testing Expected exception using Junit + Groovy (without shouldFail):
#Test(expected = NoResultException.class)
void testFetchAllNoResultsReturned() throws Exception {
List<Name> namesLocal = null;
when(Service.fetchAllNames(id)).thenThrow(
new NoResultException(""))
namesLocal = (service.fetchAllNames(id)
assert(namesLocal==null)
verify(service, times(1)).fetchAllNames(id)
}
As per the above test case, service.fetchAllNames call should throw a NoResultException. This aspect of testing seems to work well. However, the assert and verify after that are not called. As soon as the exception is encountered, the method execution stops. However, my earlier test case written in Java worked perfectly well. This issue happened only after I switched to Groovy.
After doing some Google search I found there is a method called shouldFail provided by GroovyTestCase class which can be used for this scenario as per this link. And it did resolve my issue.
Scenario 2: Testing Expected exception using Junit + Groovy (with shouldFail ):
#Test
void testFetchAllNoResultsReturned() throws Exception {
List<Name> namesLocal = null;
when(Service.fetchAllNames(id)).thenThrow(
new NoResultException(""))
shouldFail(NoResultException.class) {
namesLocal = (Service.fetchAllNames(id)
}
assert(namesLocal==null)
verify(Service, times(1)).fetchAllNames(id)
}
My doubt is, is this how it is supposed to work or am I missing something. If this is how it is supposed to work, is there any reason behind Groovy doing it this way? I tried to look for reasons on the internet but I couldn't get many leads.
However, the assert and verify after that are not called. As soon as the exception is encountered, the method execution stops. However, my earlier test case written in Java worked perfectly well.
Given this code in java:
#Test(expected = NoResultException.class)
void testFetchAllNoResultsReturned() throws Exception {
List<Name> namesLocal = null;
when(Service.fetchAllNames(id)).thenThrow(
new NoResultException(""))
namesLocal = (service.fetchAllNames(id)
....
}
Irrespective of what you have after service.fetchAllNames(id), the call will throw an Exception and the test case ends there. Since you have an expected exception defined, the test case will pass. So the assert and verify after this line of code are never executed in java.
I am not familiar with groovy but from the documentation it looks like your second example using shouldFail is the correct way to test for exceptions in groovy. The shouldFail does not terminate the program - so its similar to putting your method call in a try catch in java
Related
I have a project where I have tests where I deliberately cause a problem and then verify the code responds the way I want it. For this I want to be sure the exceptions not only are the right class but they must also carry the right message.
So in one of my existing (junit 4) tests I have something similar to this:
public class MyTests {
#Rule
public final ExpectedException expectedEx = ExpectedException.none();
#Test
public void testLoadingResourcesTheBadWay(){
expectedEx.expect(MyCustomException.class);
expectedEx.expectMessage(allOf(startsWith("Unable to load "), endsWith(" resources.")));
doStuffThatShouldFail();
}
}
I'm currently looking into fully migrating to junit 5 which no longer supports the #Rule and now has the assertThrows that seems to replace this.
What I have not been able to figure out how to write a test that not only checks the exception(class) that is thrown but also the message attached to that exception.
What is the proper way to write such a test in Junit 5?
Since Assertions.assertThrows returns instance of your exception you can invoke getMessage on the returned instance and make assertions on this message :
Executable executable = () -> sut.method(); //prepare Executable with invocation of the method on your system under test
Exception exception = Assertions.assertThrows(MyCustomException.class, executable); // you can even assign it to MyCustomException type variable
assertEquals(exception.getMessage(), "exception message"); //make assertions here
Thanks to #michalk and one of my colleagues this works:
Exception expectedEx = assertThrows(MyCustomException.class, () ->
doStuffThatShouldFail()
);
assertTrue(expectedEx.getMessage().startsWith("Unable to load "));
assertTrue(expectedEx.getMessage().endsWith(" resources."));
This question already has answers here:
Why won't this Expect Exception Junit Test Work?
(2 answers)
Closed 6 years ago.
I'm having an issue with a test case.
The method being tested has a try / catch that catches a MalformedURLException but during testing I get a failure due to a Junit AssertionError that expects a MalformedURLException. But I can't find out what it is actually throwing! Here is my code (created as a MWE in eclipse).
My method that I want to test
public void throwMalFormedURLException(){
String s = new String("www.google.com");
try{
url = new URL(s);
}catch (Exception e){
e.printStackTrace();
e.getClass();
e.getCause();
}
}
the test method
#Test (expected = MalformedURLException.class)
public void testThrowMalFormedURLException() {
MWE testClass = new MWE();
testClass.throwMalFormedURLException();
System.out.println("End of test");
}
This is the output in the console
End of test error details are: java.net.MalformedURLException: no protocol: www.google.com
at java.net.URL.(URL.java:593)
at java.net.URL.(URL.java:490)
at java.net.URL.(URL.java:439)
at MWE.throwMalFormedURLException(MWE.java:12)
at testMWE.testThrowMalFormedURLException(testMWE.java:12)
In the Junit console it says :
java.lang.AssertionError: Expected exception: Java.net.MalformedURLException
But the Junit is reporting failure, even though the console is telling me I've got a MalformedURLException.
What am I doing wrong with this test ?
Thanks for your thoughts.
David
You are catching the exception and therefore it is not being thrown.
If your intent is to test that you are capturing the exception and relaying it back to the 'user' properly, you should create tests for that specifically. You probably don't want UI elements in your unit tests* so this is a place where abstraction and DI have a lot of value. One simple approach is to create a mock object that will listen for the error message and mark a flag when the message is received. Your unit test trigger the error and then pass if the flag is set. You should also have a negative test or assert that the flag is not set prior to throwing the exception.
*Testing the UI is also a good idea but it can be a little slow. There are various tools for automating that. It generally falls in to a different phase of testing. You really want the unit tests to be really fast so that you can run them very frequently.
You have written production code that simply isn't testable. And more specifically: this code doesn't have any "programmatically" observable side effects in the first place.
Meaning: when you write "production code", the code within a method can do three things that could be observed:
Make method calls on objects that are fields of the class under test
Return some value
Throw an exception
For each of these options, you might be able to write testing code:
Using dependency injection, you can put a mocked object into your class under test. And then you can check that the expected methods are invoked on your mock object; with the parameter that you would expect.
You compare the result you get from calling that method with some expected value
You use expected to ensure that a specific exception was thrown
When you look at your code; you can see: it does none of that. It only operates on objects that are created within the method. It doesn't return any value. And, most importantly: it doesn't throw an exception!
Long story short: a caught exception isn't "leaving" the method. It is caught there, and the method ends normally. The fact that you print details about the caught exception doesn't change that.
So, the first thing you have to do: remove the whole try/catch from your production code!
And, if you want to have a more specific test, you can do something like:
#Test
public void testException() {
try {
new MWE().throwMalFormedURLException();
fail("should have thrown!");
} catch ( MalFormedURLException me ) {
assertThat(me.getMessage(), containsString("whatever"));
}
The above:
fails when no exception is thrown
fails when any other exception than MalFormedURLException is thrown
allows you to check further properties of the thrown exception
This is a valid test failure. The test asserts that calling throwMalFormedURLException() will throw MalformedURLException, but since you're catching the exception, it doesn't throw it, so the test fails.
If we have created a singleton object to handle an Java Exceptions, why Thread.setDefaultUncaughtExceptionHandler runs ok in Java Application Server, Java Console Application but not works on JUnit tests?
For example, the following code works:
public class Main extends Object {
public static void main(String[] arguments) {
Thread.setDefaultUncaughtExceptionHandler(UncaughtExceptionHandler.getInstance());
double a = 1/0;
}
}
but this JUnit test not:
public class UncaughtExceptionHandlerTest {
#Test
public void throwException() {
Thread.setDefaultUncaughtExceptionHandler(UncaughtExceptionHandler.getInstance());
double a = 1/0;
}
}
but why? And, how can we solve this, to automatically handle all JUnit test exceptions without using a moody try catch to each test?
The JUnit will be catching all unexpected exceptions that are thrown by the unit tests on the unit test threads1. The normal behavior is to catch / display / record the exception as a FAILed test, and then continue with the next unit test.
This means that the there is no "uncaught exception" in the Java sense, and your uncaught exception handler is not going to be called.
It is not entirely clear what you are trying to achieve here, but I suspect that the answer would be to implement a custom runner:
https://github.com/junit-team/junit4/wiki/Test-runners
1 - If the code under test spawns its own threads, the JUnit framework has no way of knowing. It certainly cannot catch / detect uncaught exceptions on those threads. However, this doesn't seem to be what you are talking about in this question.
The main motivation, is, for example, send an e-mail or perform another administrative tasks if a junit test fail. If I have a global exception handler I could do this, instead put a catch block to each test. After the handling, maybe I will throw this exception and let junit go ahead as it does.
Well if that is what you are trying to do, then you are (IMO) doing it the wrong way. There are existing runners that provide a structured report file, or a report data structure that can give you a list of all tests that passed, failed from an assertion, failed from an exception, etc. What you should do is:
choose an appropriate runner
analyse its output
send a single email (or whatever) if there are errors that meet your criteria.
Advantages:
less effort
you deal with all errors not just uncaught exceptions (though actually assertion failures manifest as AssertionError exceptions ...)
you don't spam yourself on each and every failed test.
And there's another way. Look at JUnitCore (link). This allows you register a listener for the various test events, and then run a bunch of tests or test suites.
The other point is that you appear to be trying to duplicate (some of) the functionality of a Continuous Integration (CI) server such as Jenkins.
You then asked why this doesn't work:
#Test
public void throwException() {
Thread.setDefaultUncaughtExceptionHandler(/* some handler */));
double a = 1/0;
}
An uncaught exception handler is only invoked if nothing else catches the exception. But a typical JUnit test runner catches all exceptions that propagate from each unit test using a conventional exception handler. That means that the ArithmeticException thrown in your test never reaches your handler.
Exceptions thrown by your junit #Test method are not uncaught. JUnit catches them and uses them to fail your tests.
Now, if you had started a new Thread of your own that is not running inside JUnit's try/catch execution, a thrown exception will be essentially ignored and your test will pass.
Just think of the name... Thread.setDefaultUncaughtExceptionHandler. This only covers threads that do not explicitly have an uncaught exception handler, and then it doesn't cover exceptions that are caught by the code calling your code (JUnit, etc).
Here is relevant code from ParentRunner class:
protected final void runLeaf(Statement statement,
Description description, RunNotifier notifier) {
EachTestNotifier eachNotifier = new EachTestNotifier(notifier, description);
eachNotifier.fireTestStarted();
try {
statement.evaluate();
} catch (AssumptionViolatedException e) {
eachNotifier.addFailedAssumption(e);
} catch (Throwable e) {
eachNotifier.addFailure(e);
} finally {
eachNotifier.fireTestFinished();
}
Are you sure that jUnit isn't catching it somewhere? The method signature says that it throws Exception so I'd guess that there has to be a pretty broad catch statement up-stream.
I'm in the process of migrating a test framework from JUnit to TestNG. This framework is used to perform large end-to-end integration tests with Selenium that take several minutes to run and consist of several hundred steps across dozens of browser pages.
DISCLAIMER: I understand that this makes unit testing idealists very uneasy, but this sort of testing is required at most large service oriented companies and using unit testing tools to manage these integration tests is currently the most widespread solution. It wasn't my decision. It's what I've been asked to work on and I'm attempting to make the best of it.
At any rate, these tests fail very frequently (surprise) and making them easy to debug is of high importance. For this reason we like to detect test failures before they're reported, append some information about the failure, and then allow JUnit to fail with this extra information. For instance, without this information a failure may look like:
java.lang.<'SomeObscureException'>: <'Some obscure message'> at <'StackTrace'>
But with the added information it will look like:
java.lang.AssertionError:
Reproduction Seed: <'Random number used to generate test case'>
Country: <'Country for which test was set to run'>
Language: <'Localized language used by test'>
Step: <'Test step where the exception occurred'>
Exception Message: <'Message explaining probable cause of failure'>
Associated Exception Type: <'SomeObscureException'>
Associated Exception Message: <'Some obscure message'>
Associated Exception StackTrace: <'StackTrace'>
Exception StackTrace: <'StackTrace where we appended this information'>
It's important to note that we add this information before the test actually fails. Because our reporting tool is based entirely on the exceptions thrown by JUnit this ensures that the information we need is present in those exceptions. Ideally I'd like to add this information to an HTML or XML document using a reporter class after the test fails but before teardown is performed and then modify our reporting tool to pick up this extra information and append it to our e-mail reports. However, this has been a hard sell at our sprint planning meetings and I have not been allotted any time to work on it (running endless regressions for the developers is given higher priority than working on the test framework itself. Such is the life of the modern SDET). I also believe strongly in balance and refuse to cut into other parts of my life to get this done outside of tracked time.
What we're currently doing is this:
public class SomeTests extends TestBase {
#Test
public void someTest() {
// Test code
}
// More tests
}
public abstract class TestBase {
#Rule
public MyWatcher watcher = new MyWatcher();
// More rules and variables
#Before
public final void setup() {
// Read config, generate test data, create Selenium WebDriver, etc.
// Send references to all test objects to MyWatcher
}
}
public class MyWatcher extends TestWatcher {
// Test object references
#Override
public void failed(Throwable throwable, Description description) {
StringBuilder sb = new StringBuilder();
// Append custom test information to sb.
String exceptionSummary = sb.toString();
Assert.fail(exceptionSummary);
}
#Override
public void finished(Description description) {
// Shut down Selenium WebDriver, kill proxy server, etc.
}
// Miscellaneous teardown and logging methods
}
JUnit starts.
SomeTests inherits from TestBase class. TestBase instantiates our own instance of a TestWatcher via #Rule annotation (MyWatcher).
Test setup is run in TestBase class.
References to test objects are sent to MyWatcher.
JUnit begins someTest() method.
someTest fails at some point.
JUnit calls overridden failed() method in MyWatcher.
failed() method appends custom test information to new message using references passed by TestBase.
failed() method calls JUnit's Assert.fail() method with the customized message.
JUnit throws a java.lang.Assertion error for this new failure with the customized message. This is the exception that actually gets recorded in the test results.
JUnit calls overridden finished() method.
finished() method performs test teardown.
Our reporting tool picks up the summarized errors thrown by JUnit, and includes them in the e-mails we receive. This makes life easier than debugging the original exceptions would be without any of the extra information added by MyWatcher after the original failure.
I'd now like to implement a similar mechanism using TestNG. I first tried adding an IInvokedMethodListener in a #Listener annotation to our TestBase class as a way of replacing the TestWatcher that we were using in JUnit. Unfortunately the methods in this listener were getting called after every #BeforeMethod and #AfterMethod call as well as for the actual tests. This was causing quite a mess when I called Assert.fail from inside the IInvokedMethodListener so I opted to scrap this approach and insert the code directly into an #AfterMethod call in our TestBase class.
Unfortunately TestNG does not appear to handle the 'failing twice' approach that we were using in JUnit. When I call Assert.fail in the #AfterMethod of a test that has already failed it gets reported as an additional failure. It seems like we're going to have to come up with another way of doing this until I can get authorization to write a proper test reporter that includes the information we need for debugging.
In the meantime, we still need to dress up the exceptions that get thrown by TestNG so that the debugging information will appear in our e-mail reports. One idea I have for doing this is to wrap every single test in a try/catch block. If the test fails (an exception gets thrown), then we can catch that exception, dress it up in a summary exception with the debugging information added to that exception's message, and call Assert.fail with our new summarized exception. That way TestNG only ever sees that one exception and should only report one failure. This feels like a kludge on top of a kludge though, and I can't help but feel that there's a better way of doing this.
Does anybody know of a better method for modifying what gets reported by TestNG? Is there some kind of trick I can use for replacing the original exception with my own using ITestContext or ITestResult? Can I dive in somewhere and remove the original failure from some list, or is it already too late to stop TestNG's internal reporting by the time I get to the #AfterMethod functions?
Do you have any other advice regarding this sort of testing or exception handling in general? I don't have many knowledgeable co-workers to help with this stuff so I'm pretty much just winging it.
Implement IInvokedMethodListener:
public class InvokedMethodListener implements IInvokedMethodListener {
#Override
public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
}
#Override
public void afterInvocation(IInvokedMethod method, ITestResult result) {
if (method.isTestMethod() && ITestResult.FAILURE == result.getStatus()) {
Throwable throwable = result.getThrowable();
String originalMessage = throwable.getMessage();
String newMessage = originalMessage + "\nReproduction Seed: ...\nCountry: ...";
try {
FieldUtils.writeField(throwable, "detailMessage", newMessage, true);
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
Register it in your test:
#Listeners(InvokedMethodListener.class)
public class YourTest {
#Test
public void test() {
Assert.fail("some message");
}
}
or in testng.xml.
If you execute it, you should get:
java.lang.AssertionError: some message
Reproduction Seed: ...
Country: ...
You can user SoftAssert Class in testNG for implementing above scenario. SoftAssert Class has an hash map array which stores all the error message from Asserts in test cases and prints them in the end of the test case. you can also extend Assertion class to implement methods as per your requirement.
More information regarding SoftAssert class and its implementation can be found here
There is a fail() method in JUnit4 library. I like it, but experiencing a lack of pass() method which is not present in the library. Why is it so?
I've found out that I can use assertTrue(true) instead but still looks unlogical.
#Test
public void testSetterForeignWord(){
try {
card.setForeignWord("");
fail();
} catch (IncorrectArgumentForSetter ex){
}
// assertTrue(true);
}
Call return statement anytime your test is finished and passed.
As long as the test doesn't throw an exception, it passes, unless your #Test annotation specifies an expected exception. I suppose a pass() could throw a special exception that JUnit always interprets as passing, so as to short circuit the test, but that would go against the usual design of tests (i.e. assume success and only fail if an assertion fails) and, if people got the idea that it was preferable to use pass(), it would significantly slow down a large suite of passing tests (due to the overhead of exception creation). Failing tests should not be the norm, so it's not a big deal if they have that overhead.
Note that your example could be rewritten like this:
#Test(expected=IncorrectArgumentForSetter.class)
public void testSetterForeignWord("") throws Exception {
card.setForeignWord("");
}
Also, you should favor the use of standard Java exceptions. Your IncorrectArgumentForSetter should probably be an IllegalArgumentException.
I think this question needs an updated answer, since most of the answers here are fairly outdated.
Firstly to the OP's question:
I think its pretty well accepted that introducing the "expected excepetion" concept into JUnit was a bad move, since that exception could be raised anywhere, and it will pass the test. It works if your throwing (and asserting on) very domain specific exceptions, but I only throw those kinds of exceptions when I'm working on code that needs to be absolutely immaculate, --most APIS will simply throw the built in exceptions like IllegalArgumentException or IllegalStateException. If two calls your making could potentitally throw these exceptions, then the #ExpectedException annotation will green-bar your test even if its the wrong line that throws the exception!
For this situation I've written a class that I'm sure many others here have written, that's an assertThrows method:
public class Exceptions {
private Exceptions(){}
public static void assertThrows(Class<? extends Exception> expectedException, Runnable actionThatShouldThrow){
try{
actionThatShouldThrow.run();
fail("expected action to throw " + expectedException.getSimpleName() + " but it did not.");
}
catch(Exception e){
if ( ! expectedException.isInstance(e)) {
throw e;
}
}
}
}
this method simply returns if the exception is thrown, allowing you to do further assertions/verification in your test.
with java 8 syntax your test looks really nice. Below is one of the simpler tests on our model that uses the method:
#Test
public void when_input_lower_bound_is_greater_than_upper_bound_axis_should_throw_illegal_arg() {
//setup
AxisRange range = new AxisRange(0,100);
//act
Runnable act = () -> range.setLowerBound(200);
//assert
assertThrows(IllegalArgumentException.class, act);
}
these tests are a little wonky because the "act" step doesn't actually perform any action, but I think the meaning is still fairly clear.
there's also a tiny little library on maven called catch-exception that uses the mockito-style syntax to verify that exceptions get thrown. It looks pretty, but I'm not a fan of dynamic proxies. That said, there syntax is so slick it remains tempting:
// given: an empty list
List myList = new ArrayList();
// when: we try to get the first element of the list
// then: catch the exception if any is thrown
catchException(myList).get(1);
// then: we expect an IndexOutOfBoundsException
assert caughtException() instanceof IndexOutOfBoundsException;
Lastly, for the situation that I ran into to get to this thread, there is a way to ignore tests if some conidition is met.
Right now I'm working on getting some DLLs called through a java native-library-loading-library called JNA, but our build server is in ubuntu. I like to try to drive this kind of development with JUnit tests --even though they're far from "units" at this point--. What I want to do is run the test if I'm on a local machine, but ignore the test if we're on ubuntu. JUnit 4 does have a provision for this, called Assume:
#Test
public void when_asking_JNA_to_load_a_dll() throws URISyntaxException {
//this line will cause the test to be branded as "ignored" when "isCircleCI"
//(the machine running ubuntu is running this test) is true.
Assume.assumeFalse(BootstrappingUtilities.isCircleCI());
//an ignored test will typically result in some qualifier being put on the results,
//but will also not typically prevent a green-ton most platforms.
//setup
URL url = DLLTestFixture.class.getResource("USERDLL.dll");
String path = url.toURI().getPath();
path = path.substring(0, path.lastIndexOf("/"));
//act
NativeLibrary.addSearchPath("USERDLL", path);
Object dll = Native.loadLibrary("USERDLL", NativeCallbacks.EmptyInterface.class);
//assert
assertThat(dll).isNotNull();
}
I was looking for pass method for JUnit as well, so that I could short-circuit some tests that were not applicable in some scenarios (there are integration tests, rather than pure unit tests). So too bad it is not there.
Fortunately, there is a way to have a test ignored conditionally, which actually fits even better in my case using assumeTrue method:
Assume.assumeTrue(isTestApplicable);
So here the test will be executed only if isTestApplicable is true, otherwise test will be ignored.
There is no need for the pass method because when no AssertionFailedException is thrown from the test code the unit test case will pass.
The fail() method actually throws an AssertionFailedException to fail the testCase if control comes to that point.
I think that this question is a result of a little misunderstanding of the test execution process. In JUnit (and other testing tools) results are counted per method, not per assert call. There is not a counter, which keeps track of how many passed/failured assertX was executed.
JUnit executes each test method separately. If the method returns successfully, then the test registered as "passed". If an exception occurs, then the test registered as "failed". In the latter case two subcase are possible: 1) a JUnit assertion exception, 2) any other kind of exceptions. Status will be "failed" in the first case, and "error" in the second case.
In the Assert class many shorthand methods are avaiable for throwing assertion exceptions. In other words, Assert is an abstraction layer over JUnit's exceptions.
For example, this is the source code of assertEquals on GitHub:
/**
* Asserts that two Strings are equal.
*/
static public void assertEquals(String message, String expected, String actual) {
if (expected == null && actual == null) {
return;
}
if (expected != null && expected.equals(actual)) {
return;
}
String cleanMessage = message == null ? "" : message;
throw new ComparisonFailure(cleanMessage, expected, actual);
}
As you can see, in case of equality nothing happens, otherwise an excepion will be thrown.
So:
assertEqual("Oh!", "Some string", "Another string!");
simply throws a ComparisonFailure exception, which will be catched by JUnit, and
assertEqual("Oh?", "Same string", "Same string");
does NOTHING.
In sum, something like pass() would not make any sense, because it did not do anything.