Develop Reference

Gradle ignores priority in testng

Can someone explain how does gradle works with priority parameter in #Test annotation in testng?
For example I have the next code:
public class TestGradle {
#Test (priority = 2)
public void testA() throws Exception {
System.out.println("Test A");
}
#Test (priority = 1)
public void testB() throws Exception {
System.out.println("Test B");
}
#Test (priority = 3)
public void testC() throws Exception {
System.out.println("Test C");
}
}
So if I will run it via gradle test --tests TestGradle I will get the next output:
Test A
Test B
Test C
but I thought, that it should be like this:
Test B
Test A
Test C

I think this is a bug in TestNG. I created an issue.
You can create a workaround to solve this problem.
With an IMethodInterceptor you can change the execution order of your methods.
public class ExecutionOrderInterceptor implements IMethodInterceptor {
#Override
public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
Comparator<IMethodInstance> comparator = new Comparator<IMethodInstance>() {
private int getPriority(IMethodInstance mi) {
int result = 0;
Method method = mi.getMethod().getConstructorOrMethod().getMethod();
Test a1 = method.getAnnotation(Test.class);
if (a1 != null) {
result = a1.priority();
}
return result;
}
public int compare(IMethodInstance m1, IMethodInstance m2) {
return getPriority(m1) - getPriority(m2);
}
};
IMethodInstance[] array = methods.toArray(new IMethodInstance[methods.size()]);
Arrays.sort(array, comparator);
return Arrays.asList(array);
}
}
To use this interceptor you need to add an annotation to your test classes:
#Listeners({ TestReportListener.class })
public class MyTestClass {
...
You can also create an abstract class which your test classes extend:
#Listeners({ TestReportListener.class })
public abstract class BaseTest {
//Add here common code
}
And use it in your test class:
public class MyTestClass extends BaseTest {
...
Note: You can use such an interceptor also to implement a custom execution order.

Custom Junit runner, Before and After running

In unit custom runner I want to perform action before and after running the test action,
so I came around with this solution.
how solid is doing it that way, is there a more clean way to achieve that?
public class SomeCustomRunner extends BlockJUnit4ClassRunner {
private int m_testMethodIndex = 0;
private int m_testMethodsCount = 0;
private boolean m_sessionSetup = false;
#Override
protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
if(m_sessionSetup == false) {
m_sessionSetup = true;
beforeTestClass(); //->> DO MY STUFF HERE
}
super.runChild(method, notifier);
m_testMethodIndex++;
if(m_testMethodIndex == m_testMethodsCount) {
afterTestClass(); //->> DO MY STUFF HERE
}
}
#Override
protected List<FrameworkMethod> getChildren() {
List<FrameworkMethod> methods = super.getChildren();
m_testMethodsCount = methods.size();
return methods;
}
}

Instead of creating a separate test runner, you can define the actions to perform before and after in the test class itself in methods annotated with #BeforeClass resp. #AfterClass.
To reuse them in more than one test you can easily inherit them from a base class.

the easiest way is override the run method as below:
public class LifecycleRunner extends BlockJUnit4ClassRunner {
public LifecycleRunner(Class<?> klass) throws InitializationError {
super(klass);
}
#Override
public void run(RunNotifier notifier) {
beforeRunTests();
try {
super.run(notifier);
} finally {
afterRunTests();
}
}
private void afterRunTests() {
trace();
}
private void beforeRunTests() {
trace();
}
private void trace() {
System.out.println(Thread.currentThread().getStackTrace()[2]);
}
}

To gain full control over test execution, install a proper RunListener in your runner.
#Override
public void run(final RunNotifier notifier)
{
final RunListener listener = new RunListener()
{
#Override
public void testStarted(final Description description) throws Exception
{
// do something before each (non-ignored) Test method is executed
}
#Override
public void testFinished(final Description description) throws Exception
{
// do something after each Test method was performed (failed or successful)
}
// check further listener methods yourself!
};
// install your listener
notifier.addListener(listener);
super.run(notifier);
// and remove it again
notifier.removeListener(listener);
}

java testng retry logic

I have written a code for retrying a failed test case in selenium web driver and java.
How should we enhance the script to hold only one record in the test case output even the same test case as been executed multiple times.
I don't want the test output report to contain any redundant result entries.
Code:
Retry Logic :
package tests;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
public class Retry implements IRetryAnalyzer {
private int retryCount = 0;
private int maxRetryCount = 2; // retry a failed test 2 additional times
#Override
public boolean retry(ITestResult result) {
if (retryCount <maxRetryCount) {
retryCount++;
return true;
}
return false;
}
}
Implementation in the class
#Test(retryAnalyzer=Retry.class)
public void example() throws CustomException
{
throw new CustomException("Example");
}
Please let me know what changes are needed.
Thanks and Regards
Sushanth.G

I had the same problem before.
We used jenkins to run CI, and need to make sure all test case results were SUCCESS (even after some times of retry) then we can deploy build.
The solution is adding TestListenerAdapter to re-write test result status to SKIP if this test will be run again.
public class MyTestListenerAdapter extends TestListenerAdapter {
#Override
public void onTestFailure(ITestResult result) {
if (result.getMethod().getRetryAnalyzer() != null) {
MyRetryAnalyzer retryAnalyzer = (MyRetryAnalyzer)result.getMethod().getRetryAnalyzer();
if(retryAnalyzer.isRetryAvailable()) {
result.setStatus(ITestResult.SKIP);
} else {
result.setStatus(ITestResult.FAILURE);
}
Reporter.setCurrentTestResult(result);
}
}
}
RetryAnalyzer needs to provide another method (isRetryAvailable() in this example) for TestListenerAdapter.
public class MyRetryAnalyzer implements IRetryAnalyzer {
private static int MAX_RETRY_COUNT = 3;
AtomicInteger count = new AtomicInteger(MAX_RETRY_COUNT);
public boolean isRetryAvailable() {
return (count.intValue() > 0);
}
#Override
public boolean retry(ITestResult result) {
boolean retry = false;
if (isRetryAvailable()) {
System.out.println("Going to retry test case: " + result.getMethod() + ", " + (MAX_RETRY_COUNT - count.intValue() + 1) + " out of " + MAX_RETRY_COUNT);
retry = true;
count.decrementAndGet();
}
return retry;
}
}
Now we can add this TestListenerAdapter to test class.
#Listeners({MyTestListenerAdapter.class})
public class AppTest {
#Test(retryAnalyzer=MyRetryAnalyzer.class)
public void testSum(){
MyClass c = new MyClass();
Assert.assertEquals(c.sum(2, 3), 5);
}
}
Please see full example here: https://github.com/haojiwu/testng-retry-example

Setting up a test system with "real data"

So right now I'm using JUnit 4 and in the #BeforeClass methods I setup everything needed to reset the user schema or to prepare sample data.
Now, it's not that I don't like this approach but I found it quite frustrating for the following reason:
I'm using the Parameterized annotation to run the very same tests with different input data. Parameterized doesn't work on #BeforeClass because #BeforeClass works with a static method.
This means I have to replicate tests if I want to keep the #BeforeClass logic. I can't use #After and #Before because those will happen after every test and it would be an overhead.
I was thinking I could refactor this Unit Tests in the sense that I'll write an abstract class that handles the test and a subclass for every group parameters I want to try so that I can have the test code written only once.
I'm hoping you can suggest a cleaner option with the following starting point: the use of #Parameterized, the need to run the "database" method only once per parameter group.
EDIT:
this is an example of my class without the BeforeClass
RunWith(LabelledParameterized.class)
public class TestCreateCampaign extends AbstractTestSubscriberCampaign {
public TestCreateCampaign(String label, String apiKey, String userKey,
int customerId) {
super(label, apiKey, userKey, customerId);
}
#Before
public void setUp() throws Exception {
super.setUp();
}
#After
public void tearDown() throws Exception {
super.tearDown();
}
#Parameters
public static Collection<Object[]> generatedData() {
return DataProvider.generatedCorrectSubscriberData();
}
#Test
public void testCreateEmailCampaignBothTriggered() {
// TEST
}
#Test
public void testCreateTextCampaignTriggered() {
// TEST
}
#Test
public void testCreateTextCampaignTest() {
// TEST
}
// Other Tests
}

This depends on how you want to set up your classes, but you can use a ClassRule for this. This does the same job as a TestRule, but it runs once for each class, rather than each test. This can be combined with Parameterized and TestRule, such as:
#RunWith(Parameterized.class)
public class TestCreateCampaign {
#ClassRule
public static ExternalResource beforeAfterClass = new ExternalResource() {
#Override
protected void before() throws Throwable {
System.out.println("before each class");
}
#Override
protected void after() {
System.out.println("after each class");
}
};
#Rule
public ExternalResource beforeAfter = new ExternalResource() {
#Override
protected void before() throws Throwable {
System.out.println("before each test");
}
#Override
protected void after() {
System.out.println("after each test");
}
};
#Parameters(name = "{index}: fib({0})={1}")
public static Iterable<Object[]> data() {
return Arrays.asList(new Object[][] { { 3, 0 }, { 4, 1 } });
}
private int fInput;
private int fExpected;
public TestCreateCampaign(int input, int expected) {
fInput = input;
fExpected = expected;
}
#Test
public void test1() {
System.out.println("test1 fInput=" + fInput);
}
}
This produces the following output:
before each class
before each test
test1 3
after each test
before each test
test1 4
after each test
after each class
This seems to be what you're looking for. To cut down on the amount of duplication, you can of course define beforeAfterClass and beforeAfter in a separate java class.
These are available in JUnit 4.9+.

What about calling your setup method from the constructor of your parameterized test class?
Edit:
OK, do I don't know of anything that does this automatically, but I think you could code up a Rule to do it. You could either implement a Rule from scratch of extend ExternalResource. Here is what I think it would do.
The constructor would take an instance of the test class and an ExternalResource instance.
In the constructor it would find the list of methods that contain the #Test annotation a get a count. It would set an iteration count to 0.
In the before method it would increment the iteration count and if it is 1 after increment (or 0 before) it would invoke the before method on the passed ExternalResource.
In the after method it would check to see if the iteration count was equal to the number of tests and if so call the after method on the passed ExternalResource.
You might need to use a different callback class / interface and ExternalResource since the before and after methods are protected. If you really wanted to be cool, you would define your own BeforeParameters and AfterParameter annotations in your rule and it would look for those methods in the passed instance.
If you develop this please post it or submit it to JUnit for inclusion.
Here is what I came up with, not as nice as I would like:
#RunWith(Parameterized.class)
public class TestExample {
private interface BeforeAfter {
void before();
void after();
}
public static class Resource extends ExternalResource {
private final int count;
private final BeforeAfter ba;
private int iteration = 0;
Resource(Object instance, BeforeAfter ba) {
int localCount = 0;
for (Method method : instance.getClass().getMethods()) {
if (method.getAnnotation(Test.class) != null) {
localCount++;
}
}
this.count = localCount;
this.ba = ba;
}
#Override
protected void before() throws Throwable {
if (iteration == 0) {
ba.before();
}
iteration++;
}
#Override
protected void after() {
if (iteration == count) {
ba.after();
iteration = 0;
}
}
}
#Parameters
public static Iterable<Object[]> data() {
return Arrays.asList(new Object[][] { { 3, 0 }, { 4, 1 } });
}
#Rule
public static Resource resource = new Resource(new TestExample(0, 0), new BeforeAfter() {
#Override
public void before() {
System.out.println("setup");
}
#Override
public void after() {
System.out.println("cleanup");
}
});
private int fInput;
private int fExpected;
public TestExample(int input, int expected) {
// System.out.println("Constructor invoked" + fInput);
fInput = input;
fExpected = expected;
}
#Test
public void test1() {
System.out.println("test1 fInput=" + fInput);
}
#Test
public void test2() {
System.out.println("test2 fInput=" + fInput);
}
}
Resulted in:
setup
test1 fInput=3
test2 fInput=3
cleanup
setup
test1 fInput=4
test2 fInput=4
cleanup

See How to load DBUnit test data once per case with Spring Test for a way of initialising your test data only once per test run.

Log information inside a JUnit Suite

I'm currently trying to write inside a log file the total number of failed tests from a JUnite Suite.
My testsuite is defined as follows:
#RunWith(Suite.class)
#SuiteClasses({Class1.class, Class2.class etc.})
public class SimpleTestSuite {}
I tried to define a rule which would increase the total number of errors when a test fails, but apparently my rule is never called.
#Rule
public MethodRule logWatchRule = new TestWatchman() {
public void failed(Throwable e, FrameworkMethod method) {
errors += 1;
}
public void succeeded(FrameworkMethod method) {
}
};
Any ideas on what I should to do to achieve this behaviour?

The following code works for me. I suspect you are not declaring errors as static.
Consider that JUnit creates a new instance of the fixture before exercising each test method.
public class WatchmanExample {
private static int failedCount = 0;
#Rule
public MethodRule watchman = new TestWatchman() {
#Override
public void failed(Throwable e, FrameworkMethod method) {
failedCount++;
}
};
#Test
public void test1() {
fail();
}
#Test
public void test2() {
fail();
}
}