Develop Reference

Mockito throws UnnecessaryStubbingException for stub defined in generic #BeforeEach setup method

I have a unit test class where my unit under test is dependent on another class. The dependency is mocked using Mockito and then setup with a generic stub that's run before every unit test, using JUnit's #BeforeEach annotation. See the below pseudo-code.
#ExtendWith(MockitoExtension.class)
public class FooFactoryTest {
#Mock
private BarDependency barDependency;
#InjectMocks
private FooFactory unitUnderTest;
#BeforeEach
public void setup() {
when(barDependency.leftBar(any())).thenReturn(new Bar());
when(barDependency.rightBar(any())).thenReturn(new Bar());
}
... many tests ...
This setup works perfectly for 9 of my 10 unit tests. Unfortunately one of my tests is failing with the following error:
org.mockito.exceptions.misusing.UnnecessaryStubbingException:
Unnecessary stubbings detected.
Clean & maintainable test code requires zero unnecessary code.
Following stubbings are unnecessary (click to navigate to relevant line of code):
1. -> at nl.devillers.mockito.FooFactoryTest.setup(FooFactoryTest.java:69)
Please remove unnecessary stubbings or use 'lenient' strictness. More info: javadoc for UnnecessaryStubbingException class.
Now, I understand the error and why it's thrown, because in that specific test my unit under test short-circuits early and doesn't hit all the stubs that have been setup. I'm aware there's a lenient option in Mockito, but that'll disable the check for the entire class/project.
My question is: how do I disable this strictness for one specific unit test?
Again: I do not want to disable the strictness at the class or project level, since I think it's valuable check. I also do not want to move my setup code to the tests that do need the stubs, because then I have to duplicate the setup code nine times. In this case I just want to disable or skip the check for this specific test.

You can avoid this check at the test-level by resetting the mock with the stub that's not being called. This is a bit of a work around, but it'll allow you to pass your test without having to make the mock lenient for your entire test class or removing the mock entirely from your setup method.
To reset your barDependency mock, add the following line of code to the end of your test that is failing with a UnnecessaryStubbingException.
Mockito.reset(barDependency);
If none of the stubs on barDependency are being used, you can also place this line at the beginning of your test with your other arrangements, which is a little cleaner. Or alternately, put it at the top of your test method and then setup the stubs you do need. This essentially overrides whatever is being setup in your generic setup method.

You can also use lenient in the code of your #BeforeEach, like this:
#BeforeEach
public void setup() {
lenient().when(barDependency.leftBar(any())).thenReturn(new Bar());
lenient().when(barDependency.rightBar(any())).thenReturn(new Bar());
}
This should not be necessary, based on this doc in the UnnecessaryStubbingException:
Mockito JUnit Runner triggers UnnecessaryStubbingException only when none of the test methods use the stubbings. This means that it is ok to put default stubbing in a 'setup' method or in test class constructor. That default stubbing needs to be used at least once by one of the test methods.

JUnit assertion to force a line to be executed

Is there any junit assertion, with which i can force a line to be executed?
For example:
doAnswer(new Answer<Void>() {
#SuppressWarnings("unchecked")
#Override
public Void answer(final InvocationOnMock invocation) throws Throwable {
Object[] arguments = invocation.getArguments();
Map<String, String> fieldMapActual = (Map<String, String>) arguments[0];
assertEquals(fieldMap, fieldMapActual);
**assertFailIfThisLineIsNotExecuted();**
return null;
}
}).when(x).myMethod(xxx);
As i simulate the behaviour of myMethod, the method answer from the anonymous inner type will be executed at runtime of the myMethod (not at runtime of junit test), if myMethod will be called with the intended value/parameter.
In order to assert that the method is called, i must additionally define a verify (otherwise my test would still run even if the method is not called).
verify(x).myMethod(xxx);
If i had a chance to write sth like assertFailIfThisLineIsNotExecuted in the answer method, i would not have to define an extra verify. So again, Is there any junit assertion, with which i can force a line to be executed? Opposite of fail() so to speak, without immediately defining the method as "successful".

If you want to make sure that a certain line of your test is being executed, use a boolean flag:
final boolean[] wasExecuted = { false };
...
wasExecuted[0] = true;
...
assertTrue("Some code wasn't executed", wasExecuted);
But my gut feeling is that you're trying to solve a different problem.
The verify says "This method must have been called". It doesn't matter if you mocked an answer or not. So this is the approach that you should use.
I'm using my flag approach only when I can't create a mock for some reason. If that happens, I extend the class under test in my test code and add the flag.
The advantage of the flag over verify is that the flag documents in which place I expect the code to be (you can have the IDE search all places where the flag is used). verify() is not that easy to locate when it fails.
verify(x).myMethod(xxx); should be what you want. It also expresses intent.
assertFailIfThisLineIsNotExecuted() would also be a single line of code (so how would it be "better" than verify?), it's not supported by JUnit, you would have to write code to get a good error message, etc.

Adding custom messages to TestNG failures

I'm in the process of migrating a test framework from JUnit to TestNG. This framework is used to perform large end-to-end integration tests with Selenium that take several minutes to run and consist of several hundred steps across dozens of browser pages.
DISCLAIMER: I understand that this makes unit testing idealists very uneasy, but this sort of testing is required at most large service oriented companies and using unit testing tools to manage these integration tests is currently the most widespread solution. It wasn't my decision. It's what I've been asked to work on and I'm attempting to make the best of it.
At any rate, these tests fail very frequently (surprise) and making them easy to debug is of high importance. For this reason we like to detect test failures before they're reported, append some information about the failure, and then allow JUnit to fail with this extra information. For instance, without this information a failure may look like:
java.lang.<'SomeObscureException'>: <'Some obscure message'> at <'StackTrace'>
But with the added information it will look like:
java.lang.AssertionError:
Reproduction Seed: <'Random number used to generate test case'>
Country: <'Country for which test was set to run'>
Language: <'Localized language used by test'>
Step: <'Test step where the exception occurred'>
Exception Message: <'Message explaining probable cause of failure'>
Associated Exception Type: <'SomeObscureException'>
Associated Exception Message: <'Some obscure message'>
Associated Exception StackTrace: <'StackTrace'>
Exception StackTrace: <'StackTrace where we appended this information'>
It's important to note that we add this information before the test actually fails. Because our reporting tool is based entirely on the exceptions thrown by JUnit this ensures that the information we need is present in those exceptions. Ideally I'd like to add this information to an HTML or XML document using a reporter class after the test fails but before teardown is performed and then modify our reporting tool to pick up this extra information and append it to our e-mail reports. However, this has been a hard sell at our sprint planning meetings and I have not been allotted any time to work on it (running endless regressions for the developers is given higher priority than working on the test framework itself. Such is the life of the modern SDET). I also believe strongly in balance and refuse to cut into other parts of my life to get this done outside of tracked time.
What we're currently doing is this:
public class SomeTests extends TestBase {
#Test
public void someTest() {
// Test code
}
// More tests
}
public abstract class TestBase {
#Rule
public MyWatcher watcher = new MyWatcher();
// More rules and variables
#Before
public final void setup() {
// Read config, generate test data, create Selenium WebDriver, etc.
// Send references to all test objects to MyWatcher
}
}
public class MyWatcher extends TestWatcher {
// Test object references
#Override
public void failed(Throwable throwable, Description description) {
StringBuilder sb = new StringBuilder();
// Append custom test information to sb.
String exceptionSummary = sb.toString();
Assert.fail(exceptionSummary);
}
#Override
public void finished(Description description) {
// Shut down Selenium WebDriver, kill proxy server, etc.
}
// Miscellaneous teardown and logging methods
}
JUnit starts.
SomeTests inherits from TestBase class. TestBase instantiates our own instance of a TestWatcher via #Rule annotation (MyWatcher).
Test setup is run in TestBase class.
References to test objects are sent to MyWatcher.
JUnit begins someTest() method.
someTest fails at some point.
JUnit calls overridden failed() method in MyWatcher.
failed() method appends custom test information to new message using references passed by TestBase.
failed() method calls JUnit's Assert.fail() method with the customized message.
JUnit throws a java.lang.Assertion error for this new failure with the customized message. This is the exception that actually gets recorded in the test results.
JUnit calls overridden finished() method.
finished() method performs test teardown.
Our reporting tool picks up the summarized errors thrown by JUnit, and includes them in the e-mails we receive. This makes life easier than debugging the original exceptions would be without any of the extra information added by MyWatcher after the original failure.
I'd now like to implement a similar mechanism using TestNG. I first tried adding an IInvokedMethodListener in a #Listener annotation to our TestBase class as a way of replacing the TestWatcher that we were using in JUnit. Unfortunately the methods in this listener were getting called after every #BeforeMethod and #AfterMethod call as well as for the actual tests. This was causing quite a mess when I called Assert.fail from inside the IInvokedMethodListener so I opted to scrap this approach and insert the code directly into an #AfterMethod call in our TestBase class.
Unfortunately TestNG does not appear to handle the 'failing twice' approach that we were using in JUnit. When I call Assert.fail in the #AfterMethod of a test that has already failed it gets reported as an additional failure. It seems like we're going to have to come up with another way of doing this until I can get authorization to write a proper test reporter that includes the information we need for debugging.
In the meantime, we still need to dress up the exceptions that get thrown by TestNG so that the debugging information will appear in our e-mail reports. One idea I have for doing this is to wrap every single test in a try/catch block. If the test fails (an exception gets thrown), then we can catch that exception, dress it up in a summary exception with the debugging information added to that exception's message, and call Assert.fail with our new summarized exception. That way TestNG only ever sees that one exception and should only report one failure. This feels like a kludge on top of a kludge though, and I can't help but feel that there's a better way of doing this.
Does anybody know of a better method for modifying what gets reported by TestNG? Is there some kind of trick I can use for replacing the original exception with my own using ITestContext or ITestResult? Can I dive in somewhere and remove the original failure from some list, or is it already too late to stop TestNG's internal reporting by the time I get to the #AfterMethod functions?
Do you have any other advice regarding this sort of testing or exception handling in general? I don't have many knowledgeable co-workers to help with this stuff so I'm pretty much just winging it.

Implement IInvokedMethodListener:
public class InvokedMethodListener implements IInvokedMethodListener {
#Override
public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
}
#Override
public void afterInvocation(IInvokedMethod method, ITestResult result) {
if (method.isTestMethod() && ITestResult.FAILURE == result.getStatus()) {
Throwable throwable = result.getThrowable();
String originalMessage = throwable.getMessage();
String newMessage = originalMessage + "\nReproduction Seed: ...\nCountry: ...";
try {
FieldUtils.writeField(throwable, "detailMessage", newMessage, true);
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
Register it in your test:
#Listeners(InvokedMethodListener.class)
public class YourTest {
#Test
public void test() {
Assert.fail("some message");
}
}
or in testng.xml.
If you execute it, you should get:
java.lang.AssertionError: some message
Reproduction Seed: ...
Country: ...

You can user SoftAssert Class in testNG for implementing above scenario. SoftAssert Class has an hash map array which stores all the error message from Asserts in test cases and prints them in the end of the test case. you can also extend Assertion class to implement methods as per your requirement.
More information regarding SoftAssert class and its implementation can be found here

Invoke a specific JUnit test programmatically

I would like to run a specific JUnit test method programmatically. I know I can run the whole test using the following code snippet:
Class<?> test = Class.forName("MyTestClass");
JUnitCore junit = new JUnitCore();
Result result = junit.run(test);
However, I would like to run a specific method within this test class which contains multiple methods.
It would also be fantastic if I could control the setUp/tearDown behaviour.
Thank you

There's an overloaded JUnitCore#run method version that accepts Request. While Request has Request#method factory method to:
Create a Request that, when processed, will run a single test. This is
done by filtering out all other tests. This method is used to support
rerunning single tests.
If you need to control #setUp / #tearDown (i.e., methods marked with #Before and/or #After annotations) you can extend the class and override methods you need to alter.

How can I make JUnit 4.8 run code after a failed test, but before any #After methods?

I'm driving a suite of Selenium tests (actually WebDriver-backed Selenium) using JUnit 4.8.2. I'd like the tests to automatically take a screenshot of the browser as soon as the test fails an assertion. All the tests inherit from SeleniumBaseTestCase, and the majority then further inherit from from SeleniumBastTestCaseWithCompany (which uses #Before and #After methods to create and then clean up common test data via Selenium).
I've tried adding a subclass of TestWatchman as a #Rule in SeleniumBaseTestCase, overriding TestWatchman's failed method to take the screenshot. The trouble is that the #After methods cleaning up the test data are being run before TestWatchman's failed method is called, so the screenshots are all of the final step of the clean-up, not the test that failed.
Looking into it a little, it seems that TestWatchman's apply method just calls the passed Statement's evaluate method (the only exposed method), which calls the #After methods, leaving TestWatchman (or any other Rule) no chance to insert any code between the execution of the test and of the #After methods, as far as I can tell.
I've also seen approaches that create a custom Runner to alter the Statements created so that methods annotated with the custom #AfterFailure are run before #After methods (so the screenshot can be taken in such an #AfterFailure method), but this relies on overriding BlockJUnit4ClassRunner's withAfters method, which is deprecated and due to become private, according to the documentation, which suggests using Rules instead.
I've found another answer on SO about the #Rule lifecycle that makes it sound like this simply might not be possible in JUnit 4.8, but may be possible in JUnit 4.10. If that's correct then fair enough, I'd just like confirmation of that first.
Any thoughts on an elegant and future-proof way in which I can achieve what I want would be much appreciated!

You are right in your analysis, #Befores and #Afters are added to the list of Statements before any Rules. The #Before gets executed after the #Rule and the #After gets executed before the #Rule. How you fix this depends on how flexible you can be with SeleniumBaseTestCaseWithCompany.
The easiest way would be to remove your #Before/#After methods and replace them with an ExternalResource. This could look something like:
public class BeforeAfterTest {
#Rule public TestRule rule = new ExternalResource() {
protected void before() throws Throwable { System.out.println("externalResource before"); }
protected void after() { System.out.println("externalResource after"); }
};
#Test public void testHere() { System.out.println("testHere"); }
}
this gives:
externalResource before
testHere
externalResource after
This field can be put into your base class, so it gets inherited/overridden. Your problem with ordering between #After and your rules then goes away, because you can order your rules how you like, using #RuleChain (in 4.10, not 4.8).
If you can't change SeleniumBaseTestCaseWithCompany, then you can extend BlockJUnit4ClassRunner, but don't override withAfters, but override BlockJUnit4ClassRunner#methodBlock(). You can then call super.methodBlock, and reorder the Statements as necessary[*].
[*]You could just copy the code, and reorder the lines, but withRules is private and therefore not callable from a subclass.