I'm in the process of migrating a test framework from JUnit to TestNG. This framework is used to perform large end-to-end integration tests with Selenium that take several minutes to run and consist of several hundred steps across dozens of browser pages.
DISCLAIMER: I understand that this makes unit testing idealists very uneasy, but this sort of testing is required at most large service oriented companies and using unit testing tools to manage these integration tests is currently the most widespread solution. It wasn't my decision. It's what I've been asked to work on and I'm attempting to make the best of it.
At any rate, these tests fail very frequently (surprise) and making them easy to debug is of high importance. For this reason we like to detect test failures before they're reported, append some information about the failure, and then allow JUnit to fail with this extra information. For instance, without this information a failure may look like:
java.lang.<'SomeObscureException'>: <'Some obscure message'> at <'StackTrace'>
But with the added information it will look like:
java.lang.AssertionError:
Reproduction Seed: <'Random number used to generate test case'>
Country: <'Country for which test was set to run'>
Language: <'Localized language used by test'>
Step: <'Test step where the exception occurred'>
Exception Message: <'Message explaining probable cause of failure'>
Associated Exception Type: <'SomeObscureException'>
Associated Exception Message: <'Some obscure message'>
Associated Exception StackTrace: <'StackTrace'>
Exception StackTrace: <'StackTrace where we appended this information'>
It's important to note that we add this information before the test actually fails. Because our reporting tool is based entirely on the exceptions thrown by JUnit this ensures that the information we need is present in those exceptions. Ideally I'd like to add this information to an HTML or XML document using a reporter class after the test fails but before teardown is performed and then modify our reporting tool to pick up this extra information and append it to our e-mail reports. However, this has been a hard sell at our sprint planning meetings and I have not been allotted any time to work on it (running endless regressions for the developers is given higher priority than working on the test framework itself. Such is the life of the modern SDET). I also believe strongly in balance and refuse to cut into other parts of my life to get this done outside of tracked time.
What we're currently doing is this:
public class SomeTests extends TestBase {
#Test
public void someTest() {
// Test code
}
// More tests
}
public abstract class TestBase {
#Rule
public MyWatcher watcher = new MyWatcher();
// More rules and variables
#Before
public final void setup() {
// Read config, generate test data, create Selenium WebDriver, etc.
// Send references to all test objects to MyWatcher
}
}
public class MyWatcher extends TestWatcher {
// Test object references
#Override
public void failed(Throwable throwable, Description description) {
StringBuilder sb = new StringBuilder();
// Append custom test information to sb.
String exceptionSummary = sb.toString();
Assert.fail(exceptionSummary);
}
#Override
public void finished(Description description) {
// Shut down Selenium WebDriver, kill proxy server, etc.
}
// Miscellaneous teardown and logging methods
}
JUnit starts.
SomeTests inherits from TestBase class. TestBase instantiates our own instance of a TestWatcher via #Rule annotation (MyWatcher).
Test setup is run in TestBase class.
References to test objects are sent to MyWatcher.
JUnit begins someTest() method.
someTest fails at some point.
JUnit calls overridden failed() method in MyWatcher.
failed() method appends custom test information to new message using references passed by TestBase.
failed() method calls JUnit's Assert.fail() method with the customized message.
JUnit throws a java.lang.Assertion error for this new failure with the customized message. This is the exception that actually gets recorded in the test results.
JUnit calls overridden finished() method.
finished() method performs test teardown.
Our reporting tool picks up the summarized errors thrown by JUnit, and includes them in the e-mails we receive. This makes life easier than debugging the original exceptions would be without any of the extra information added by MyWatcher after the original failure.
I'd now like to implement a similar mechanism using TestNG. I first tried adding an IInvokedMethodListener in a #Listener annotation to our TestBase class as a way of replacing the TestWatcher that we were using in JUnit. Unfortunately the methods in this listener were getting called after every #BeforeMethod and #AfterMethod call as well as for the actual tests. This was causing quite a mess when I called Assert.fail from inside the IInvokedMethodListener so I opted to scrap this approach and insert the code directly into an #AfterMethod call in our TestBase class.
Unfortunately TestNG does not appear to handle the 'failing twice' approach that we were using in JUnit. When I call Assert.fail in the #AfterMethod of a test that has already failed it gets reported as an additional failure. It seems like we're going to have to come up with another way of doing this until I can get authorization to write a proper test reporter that includes the information we need for debugging.
In the meantime, we still need to dress up the exceptions that get thrown by TestNG so that the debugging information will appear in our e-mail reports. One idea I have for doing this is to wrap every single test in a try/catch block. If the test fails (an exception gets thrown), then we can catch that exception, dress it up in a summary exception with the debugging information added to that exception's message, and call Assert.fail with our new summarized exception. That way TestNG only ever sees that one exception and should only report one failure. This feels like a kludge on top of a kludge though, and I can't help but feel that there's a better way of doing this.
Does anybody know of a better method for modifying what gets reported by TestNG? Is there some kind of trick I can use for replacing the original exception with my own using ITestContext or ITestResult? Can I dive in somewhere and remove the original failure from some list, or is it already too late to stop TestNG's internal reporting by the time I get to the #AfterMethod functions?
Do you have any other advice regarding this sort of testing or exception handling in general? I don't have many knowledgeable co-workers to help with this stuff so I'm pretty much just winging it.
Implement IInvokedMethodListener:
public class InvokedMethodListener implements IInvokedMethodListener {
#Override
public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
}
#Override
public void afterInvocation(IInvokedMethod method, ITestResult result) {
if (method.isTestMethod() && ITestResult.FAILURE == result.getStatus()) {
Throwable throwable = result.getThrowable();
String originalMessage = throwable.getMessage();
String newMessage = originalMessage + "\nReproduction Seed: ...\nCountry: ...";
try {
FieldUtils.writeField(throwable, "detailMessage", newMessage, true);
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
Register it in your test:
#Listeners(InvokedMethodListener.class)
public class YourTest {
#Test
public void test() {
Assert.fail("some message");
}
}
or in testng.xml.
If you execute it, you should get:
java.lang.AssertionError: some message
Reproduction Seed: ...
Country: ...
You can user SoftAssert Class in testNG for implementing above scenario. SoftAssert Class has an hash map array which stores all the error message from Asserts in test cases and prints them in the end of the test case. you can also extend Assertion class to implement methods as per your requirement.
More information regarding SoftAssert class and its implementation can be found here
Related
If we have created a singleton object to handle an Java Exceptions, why Thread.setDefaultUncaughtExceptionHandler runs ok in Java Application Server, Java Console Application but not works on JUnit tests?
For example, the following code works:
public class Main extends Object {
public static void main(String[] arguments) {
Thread.setDefaultUncaughtExceptionHandler(UncaughtExceptionHandler.getInstance());
double a = 1/0;
}
}
but this JUnit test not:
public class UncaughtExceptionHandlerTest {
#Test
public void throwException() {
Thread.setDefaultUncaughtExceptionHandler(UncaughtExceptionHandler.getInstance());
double a = 1/0;
}
}
but why? And, how can we solve this, to automatically handle all JUnit test exceptions without using a moody try catch to each test?
The JUnit will be catching all unexpected exceptions that are thrown by the unit tests on the unit test threads1. The normal behavior is to catch / display / record the exception as a FAILed test, and then continue with the next unit test.
This means that the there is no "uncaught exception" in the Java sense, and your uncaught exception handler is not going to be called.
It is not entirely clear what you are trying to achieve here, but I suspect that the answer would be to implement a custom runner:
https://github.com/junit-team/junit4/wiki/Test-runners
1 - If the code under test spawns its own threads, the JUnit framework has no way of knowing. It certainly cannot catch / detect uncaught exceptions on those threads. However, this doesn't seem to be what you are talking about in this question.
The main motivation, is, for example, send an e-mail or perform another administrative tasks if a junit test fail. If I have a global exception handler I could do this, instead put a catch block to each test. After the handling, maybe I will throw this exception and let junit go ahead as it does.
Well if that is what you are trying to do, then you are (IMO) doing it the wrong way. There are existing runners that provide a structured report file, or a report data structure that can give you a list of all tests that passed, failed from an assertion, failed from an exception, etc. What you should do is:
choose an appropriate runner
analyse its output
send a single email (or whatever) if there are errors that meet your criteria.
Advantages:
less effort
you deal with all errors not just uncaught exceptions (though actually assertion failures manifest as AssertionError exceptions ...)
you don't spam yourself on each and every failed test.
And there's another way. Look at JUnitCore (link). This allows you register a listener for the various test events, and then run a bunch of tests or test suites.
The other point is that you appear to be trying to duplicate (some of) the functionality of a Continuous Integration (CI) server such as Jenkins.
You then asked why this doesn't work:
#Test
public void throwException() {
Thread.setDefaultUncaughtExceptionHandler(/* some handler */));
double a = 1/0;
}
An uncaught exception handler is only invoked if nothing else catches the exception. But a typical JUnit test runner catches all exceptions that propagate from each unit test using a conventional exception handler. That means that the ArithmeticException thrown in your test never reaches your handler.
Exceptions thrown by your junit #Test method are not uncaught. JUnit catches them and uses them to fail your tests.
Now, if you had started a new Thread of your own that is not running inside JUnit's try/catch execution, a thrown exception will be essentially ignored and your test will pass.
Just think of the name... Thread.setDefaultUncaughtExceptionHandler. This only covers threads that do not explicitly have an uncaught exception handler, and then it doesn't cover exceptions that are caught by the code calling your code (JUnit, etc).
Here is relevant code from ParentRunner class:
protected final void runLeaf(Statement statement,
Description description, RunNotifier notifier) {
EachTestNotifier eachNotifier = new EachTestNotifier(notifier, description);
eachNotifier.fireTestStarted();
try {
statement.evaluate();
} catch (AssumptionViolatedException e) {
eachNotifier.addFailedAssumption(e);
} catch (Throwable e) {
eachNotifier.addFailure(e);
} finally {
eachNotifier.fireTestFinished();
}
Are you sure that jUnit isn't catching it somewhere? The method signature says that it throws Exception so I'd guess that there has to be a pretty broad catch statement up-stream.
Background
I am working with a Selenium/Junit test environment and I want to implement a class to perform "soft asserts": meaning that I want it to record whether or not the assert passed, but not actually fail the test case until I explicitly tell it to validate the Asserts. This way I can check multiple fields on a page an record all of the ones which do not match.
Current Code
My "verify" methods appear as such (similar ones exist for assertTrue/assertFalse):
public static void verifyEquals(Object expected, Object actual) {
try {
assertEquals(expected, actual);
} catch (Throwable e) {
verificationFailuresList.add(e);
}
}
Once all the fields have been verified, I call the following method:
public static void checkAllPassed() {
if (!verificationFailuresList.isEmpty()) {
for (Throwable failureThrowable : verificationFailuresList) {
log.error("Verification failure:" + failureThrowable.getMessage(), failureThrowable);
// assertTrue(false);
}
}
}
Question
At the moment, I am currently just using assertTrue(false) as a way to quickly fail the test case; however, this clutters the log with a nonsense failure and pushes the real problem further up. Is there a cleaner way to purposefully fail a JUnit testcase? If not, is there a better solution to implement soft asserts? I know of an article which has a very well done implementation, but to my knowledge JUnit has no equivalent to the IInvokedMethodListener class
In case you want to fail a JUnit test on purpose you should use org.junit.Assert.fail()
Other option is to switch to TestNG framework which already has a SoftAssert class in it's latest version.
You can use JUnit's ErrorCollector rule.
Our test environment has a variety of integration tests that rely on middleware (CMS platform, underlying DB, Elasticsearch index).
They're automated and we manage our middleware with Docker, so we don't have issues with unreliable networks. However, sometimes our DB crashes and our test fails.
The problem is that the detection of this failure is through a litany of org.hibernate.exception.JDBCConnectionException messages. These come about via a timeout. When that happens, we end up with hundreds of tests failing with this exception, each one taking many seconds to fail. As a result, it takes an age for our tests to complete. Indeed, we generally just kill these builds manually when we realise they are done.
My question: In a Maven-driven Java testing environment, is there a way to direct the build system to watch out for specific kinds of Exceptions and kill the whole process, should they arrive (or reach some kind of threshold)?
We could watchdog our containers and kill the build process that way, but I'm hoping there's a cleaner way to do it with maven.
If you use TestNG instead of JUnit, there are other possibilities to define tests as dependent on other tests.
For example, like others mentioned above, you can have a method to check your database connection and declare all other tests as dependent on this method.
#Test
public void serverIsReachable() {}
#Test(dependsOnMethods = { "serverIsReachable" })
public void queryTestOne() {}
With this, if the serverIsReachable test fails, all other tests which depends on this one will be skipped and not marked as failed. Skipped methods will be reported as such in the final report, which is important since skipped methods are not necessarily failures. But since your initial test serverIsReachable failed, the build should fail completely.
The positive effect is, that non of your other tests will be executed, which should fail very fast.
You could also extend this logic with groups. Let's say you're database queries are used by some domain logic tests afterwards, you can declare each database test with a group, like
#Test(groups = { "jdbc" })
public void queryTestOne() {}
and declare you domain logic tests as dependent on these tests, with
#Test(dependsOnGroups = { "jdbc.* })
public void domainTestOne() {}
TestNG will therefore guarantee the order of execution for your tests.
Hope this helps to make your tests a bit more structured. For more infos, have a look at the TestNG dependency documentation.
I realize this is not exactly what you are asking for, but could help none the less to speed up the build:
JUnit assumptions allow to let a test pass when an assumption fails. You could have an assumption like assumeThat(db.isReachable()) that would skip those tests when a timeout is reached.
In order to actually speed things up and to not repeat this over and over, you could put this in a #ClassRule:
A failing assumption in a #Before or #BeforeClass method will have the same effect as a failing assumption in each #Test method of the class.
Of cause you would then have to mark your build as unstable via another way, but that should be easily doable.
I don't know if you can fail-fast the build itself, or even want to - since the administrative aspects of the build may not then complete, but you could do this:
In all your test classes that depend on the database - or the parent classes, because something like this is inheritable - add this:
#BeforeClass
public void testJdbc() throws Exception {
Executors.newSingleThreadExecutor()
.submit(new Callable() {
public Object call() throws Exception {
// execute the simplest SQL you can, eg. "SELECT 1"
return null;
}
})
.get(100, TimeUnit.MILLISECONDS);
}
If the JDBC simple query fails to return within 100ms, the entire test class won't run and will show as a "fail" to the build.
Make the wait time as small as you can and still be reliable.
One thing you could do is to write a new Test Runner which will stop if such an error occurs. Here is an example of what that might look like:
import org.junit.internal.AssumptionViolatedException;
import org.junit.runner.Description;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.Statement;
public class StopAfterSpecialExceptionRunner extends BlockJUnit4ClassRunner {
private boolean failedWithSpecialException = false;
public StopAfterSpecialExceptionRunner(Class<?> klass) throws InitializationError {
super(klass);
}
#Override
protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
Description description = describeChild(method);
if (failedWithSpecialException || isIgnored(method)) {
notifier.fireTestIgnored(description);
} else {
runLeaf(methodBlock(method), description, notifier);
}
}
#Override
protected Statement methodBlock(FrameworkMethod method) {
return new FeedbackIfSpecialExceptionOccurs(super.methodBlock(method));
}
private class FeedbackIfSpecialExceptionOccurs extends Statement {
private final Statement next;
public FeedbackIfSpecialExceptionOccurs(Statement next) {
super();
this.next = next;
}
#Override
public void evaluate() throws Throwable {
boolean complete = false;
try {
next.evaluate();
complete = true;
} catch (AssumptionViolatedException e) {
throw e;
} catch (SpecialException e) {
StopAfterSpecialExceptionRunner.this.failedWithSpecialException = true;
throw e;
}
}
}
}
Then annotate your test classes with #RunWith(StopAfterSpecialExceptionRunner.class).
Basically what this does is that it checks for a certain Exception (here it's SpecialException, an Exception I wrote myself) and if this occurs it will fail the test that threw that and skip all following Tests. You could of course limit that to tests annotated with a specific annotation if you liked.
It is also possible, that a similar behavior could be achieved with a Rule and if so that may be a lot cleaner.
Can I "stop" running test from beforeInvocation method of IInvokedMethodListener2?
I do some kind of soft skip with "throw new SkipException" in beforeInvocation, but I need to do "hard skip" - stop test without any trace of execution.
I first need to check some test method's annotations and then run or not run that specific test.
Thanks ...
Throwing new SkipException in IInvokedMethodListener2.beforeInvocation() yields the same result (and report) as for a test depending on a failed test: test method makes into the report and is duly marked as skipped. So the test method does not get executed.
If it's not hard enough, I'd recommend test exclusion, not skipping. There is
BeanShell expression feature to run custom inclusions and exclusions.
Also, IAnnotationTransformer would allow you to disable (equivalent of "enabled=false" attribute) a test method at runtime, the simplest example being:
public class SkippingTransformer implements IAnnotationTransformer {
#Override
public void transform(ITestAnnotation ita, Class type, Constructor c, Method method) {
if(method.getName().startsWith("skipMe")) {
ita.setEnabled(false);
}
}
}
This will opt out test methods completely (that is, test count reduces).
But you'll most likely need to take care of disabling chains of dependent methods. There may be other aspects to consider.
I'm driving a suite of Selenium tests (actually WebDriver-backed Selenium) using JUnit 4.8.2. I'd like the tests to automatically take a screenshot of the browser as soon as the test fails an assertion. All the tests inherit from SeleniumBaseTestCase, and the majority then further inherit from from SeleniumBastTestCaseWithCompany (which uses #Before and #After methods to create and then clean up common test data via Selenium).
I've tried adding a subclass of TestWatchman as a #Rule in SeleniumBaseTestCase, overriding TestWatchman's failed method to take the screenshot. The trouble is that the #After methods cleaning up the test data are being run before TestWatchman's failed method is called, so the screenshots are all of the final step of the clean-up, not the test that failed.
Looking into it a little, it seems that TestWatchman's apply method just calls the passed Statement's evaluate method (the only exposed method), which calls the #After methods, leaving TestWatchman (or any other Rule) no chance to insert any code between the execution of the test and of the #After methods, as far as I can tell.
I've also seen approaches that create a custom Runner to alter the Statements created so that methods annotated with the custom #AfterFailure are run before #After methods (so the screenshot can be taken in such an #AfterFailure method), but this relies on overriding BlockJUnit4ClassRunner's withAfters method, which is deprecated and due to become private, according to the documentation, which suggests using Rules instead.
I've found another answer on SO about the #Rule lifecycle that makes it sound like this simply might not be possible in JUnit 4.8, but may be possible in JUnit 4.10. If that's correct then fair enough, I'd just like confirmation of that first.
Any thoughts on an elegant and future-proof way in which I can achieve what I want would be much appreciated!
You are right in your analysis, #Befores and #Afters are added to the list of Statements before any Rules. The #Before gets executed after the #Rule and the #After gets executed before the #Rule. How you fix this depends on how flexible you can be with SeleniumBaseTestCaseWithCompany.
The easiest way would be to remove your #Before/#After methods and replace them with an ExternalResource. This could look something like:
public class BeforeAfterTest {
#Rule public TestRule rule = new ExternalResource() {
protected void before() throws Throwable { System.out.println("externalResource before"); }
protected void after() { System.out.println("externalResource after"); }
};
#Test public void testHere() { System.out.println("testHere"); }
}
this gives:
externalResource before
testHere
externalResource after
This field can be put into your base class, so it gets inherited/overridden. Your problem with ordering between #After and your rules then goes away, because you can order your rules how you like, using #RuleChain (in 4.10, not 4.8).
If you can't change SeleniumBaseTestCaseWithCompany, then you can extend BlockJUnit4ClassRunner, but don't override withAfters, but override BlockJUnit4ClassRunner#methodBlock(). You can then call super.methodBlock, and reorder the Statements as necessary[*].
[*]You could just copy the code, and reorder the lines, but withRules is private and therefore not callable from a subclass.