Develop Reference

Get Test method name in TestNG/Maven reporting in case of DataProvider running multiple tests with different data for single Test method

I have a Test method in my Automation script that has to be run for 6 different people.
There is a custom DataProvider that provides those 6 people and method name is getSalaryDetails(). Execution is working perfectly fine and tests are running fine. Reporting is where the issue comes.
For these 6 Test runs for a single method, results are being displayed as 6 times for getSalaryDetails() which is making difficult to understand the DataProvider for which the test would be failing.
Is there a way to implement in TestNG/Maven so i can get the Test method name by appending the parameter/Data provider somewhere so it would read somewhat like
Person1getSalaryDetails()
Person2getSalaryDetails() and so on..!
Here is the code that is being used for reporting
public class ReportListener implements ITestListener {
//Extent Report Declarations
private static ExtentReports extent = ExtentManager.createInstance();
private static ThreadLocal<ExtentTest> test = new ThreadLocal<>();
#Override
public void onTestStart(ITestResult result) {
System.out.println((result.getMethod().getMethodName() + " started!"));
//Start operation for extentreports.
ExtentTest extentTest = extent.createTest(result.getMethod().getMethodName(), result.getMethod().getDescription());
test.set(extentTest);
}
#Override
public void onTestSuccess(ITestResult result) {
System.out.println((result.getMethod().getMethodName() + " passed!"));
test.get().pass("Test passed" + result);
}
#Override
public void onTestFailure(ITestResult result) {
System.out.println((result.getMethod().getMethodName() + " failed!"));
test.get().fail(result.getThrowable());
}
#Override
public void onTestSkipped(ITestResult result) {
System.out.println((result.getMethod().getMethodName() + " skipped!"));
test.get().skip(result.getThrowable());
}
#Override
public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
System.out.println(("onTestFailedButWithinSuccessPercentage for " + result.getMethod().getMethodName()));
}
#Override
public void onStart(ITestContext context) {
System.out.println("Extent Reports Version 3 Test Suite started!");
}
#Override
public void onFinish(ITestContext context) {
System.out.println(("Extent Reports Version 3 Test Suite is ending!"));
extent.flush();
}

Parameters passed by the data provider to test methods can be fetched in Listener methods as follows
#Override
public void onTestStart(ITestResult result) {
System.out.println((result.getMethod().getMethodName() + " started!"));
//Start operation for extentreports.
String methodName = result.getMethod().getMethodName();
Object[] parameters = result.getParameters();
if (parameters != null && parameters.length > 0) {
methodName = parameters[0].toString + methodName;
}
ExtentTest extentTest = extent.createTest(methodName, result.getMethod().getDescription());
test.set(extentTest);
}
Another option is we can add the parameter from data provider method to test description from the test method.
#Test
public void testMethod(String personName) {
ITestResult result = Reporter.getCurrentTestResult();
result.getMethod().setDescription(personName + " along with other string");
}

#Test(dataProvider = "inputData")
public void testMethod(final Map<String, Object> dataMap){
try{
//Note: As you said you are using dataProvider every test run you will get the personName- add name like shown below line.
//dataMap.get("personName") - input data from dataprovider
extentTest = extent.createTest(Thread.currentThread().getStackTrace()[1].getMethodName().toString()+dataMap.get("personName"));
}catch(Exception e){
extentTest.log(Status.ERROR, e.getMessage());
}

Running Android Tests setUp() method gets called multiple times

I am creating a test suite for my android application and have this setUp method
private static final String TAG_NAME = "TESTING_SUITE";
public TestingMusicDAO musicDAO;
public List<Song> songs;
public Instrumentation instr;
MusicService musicService;
#Override
public void setUp() throws Exception {
instr = this.getInstrumentation();
Log.d(TAG_NAME, "Setting up testing songs");
musicDAO = new TestingMusicDAO(instr.getContext());
musicService = new MusicServiceImpl(musicDAO);
musicDAO.getAllSongsFromFile();
songs = musicDAO.getAllSongs();
for(Song song : songs)
Log.d( TAG_NAME, song.toString() );
}
And then have these tests which are created by a python tool from a text file
public void test1() {
List<Song> testPlaylist;
String testArtist = ("The Beatles");
String actualArtist = ("TheBeatles");
testPlaylist = testingPlaySongsByKeyword(testArtist);
if(testPlaylist.isEmpty()){
fail("No Songs Were Found");
} else {
for( Song loopsongs : testPlaylist){
if (!(loopsongs.getArtist().equals(actualArtist))){
fail("Song Doesnt Contain the artist" + actualArtist + "... Contains ->" + loopsongs.getArtist());
}
}
}
}
and every time one of these gets called the musicDAO is regenerated. How can I stop the setup method from being called

You don't. The design of JUnit is that setUp() and tearDown() are done once per test. If you want it done per class, do it in the constructor. Just make sure that you don't alter anything inside the classes. The reason for doing it once per test is to make sure all tests start with the same data.

You could use #BeforeClass and #AfterClass annotations from JUnit.
#BeforeClass
public static void test_setUp_Once(){
// Code which you want to be executed only once
createDb();
}
#AfterClass
public static void test_tearDown_Once(){
// Code which you want to be executed only once
deleteDb();
}
Note: You need to declare these methods static to work properly.

I had the same basic problem. I want to be able to test the structure of my database, so I create it in the setUp method and delete it in the tearDown. Using the constructor wouldn't solve my need to delete the database once all my tests are executed, so I used some reentrant logic:
static int testsExecutedSoFar = 0;
static boolean isFirstRun = true;
#Override
protected void setUp() throws Exception {
if(isFirstRun){
createDb();
isFirstRun = false;
}
}
#Override
protected void tearDown() throws Exception{
testsExecutedSoFar++;
if (testsExecutedSoFar == totalNumberOfTestCases())
deleteDb();
}
private int totalNumberOfTestCases() {
return countTestCases()+1; //have to add one for testandroidtestcasesetupproperly added by AndroidTestCase
}
The fields have to be static since JUnit creates a new instance of the class for each run. The magic 1 had to be added since AndroidTestCase adds it's own test (testandroidtestcasesetupproperly) to the test suite but it doesn't count towards the number returned by countTestCases().
A bit on the ugly side, but it did the trick.

How to create own annotation for junit that will skip test if concrete exception was thrown during execution?

My application have several execution modes, and in 1 mode it is normal that some of my tests will throw a concrete exception. I need to annotate this methods with something like #SkipOnFail that will set method as skipped if exception was thrown.
thanks in advance!
#Edit(for my question to be more clear)
#Test(expected=ConcreteException.class)
does not work for me because i need my tests to pass even if ConcreteException.class was not thrown(expected tag in junit will mark my test as failed if this exception won't be thrown), and to be skipped otherwise. In all other cases it should work as always.
#Solution that worked for me(junit v4.7) thx to #axtavt
#Rule
public MethodRule skipRule = new MethodRule() {
public Statement apply(final Statement base, FrameworkMethod method, Object target) {
if(method.getAnnotation(SkipOnFail.class) == null) return base;
return new Statement() {
#Override
public void evaluate() throws Throwable {
try{
base.evaluate();
} catch (ConcreteException e) {
Assume.assumeTrue(false);
}
}
};
}
};
#Thx

I don't think that such a feature is available out of the box, but it should be pretty easy to implement with custom TestRule and Assume, something like this:
#Rule
public TestRule skipRule = new TestRule() {
public Statement apply(final Statement base, Description desc) {
if (desc.getAnnotation(SkipOnFail.class) == null) return base;
return new Statement() {
public void evaluate() throws Throwable {
try {
base.evaluate();
} catch (MyExceptoion ex) {
Assume.assumeTrue(false);
}
}
};
}
};

What about using JUnit Extensions?
The following example is taken from their Tutorial.
It provides aditional annotations for Prerequisites (#Prerequisite): Ignore tests based on conditions.
The required approach would be to check this during running tests. So you can simply add a #Prerequisite(requires="") annotation.
public class TestFillDatabase {
#Prerequisite(requires = "databaseIsAvailable")
#Test public void fillData() {
// ...
}
public boolean databaseIsAvailable() {
boolean isAvailable = ...;
return isAvailable;
}
}
public class TestFillDatabase {
#Prerequisite(requires = "databaseIsAvailable")
#Test public void fillData() {
// ...
}
public boolean databaseIsAvailable() {
boolean isAvailable = ...;
return isAvailable ;
}
}
This specified methods with #Prerequisite(requires = "databaseIsAvailable") must be a public method, returning a boolean or Boolean value.
If these methods will be consolidated in helper classes, you can also specify static methods within a class to be called using #Prerequisite(requires = "databaseIsAvailable", callee="DBHelper").
public class TestFillDatabase {
#Prerequisite(requires = "databaseIsAvailable", callee="DBHelper")
#Test public void fillData() {
// ...
}
}
public class DBHelper {
public static boolean databaseIsAvailable() {
boolean isAvailable = ...;
return isAvailable ;
}
}

Also using the Assume class (since jUnit 4.4), you can use assumeNoException():
try{
base.evaluate();
} catch (ConcreteException e) {
Assume.assumeNoException("Concrete exception: skipping test", e);
}

I searched for the docs about JUnit and it appears that from version 4.9 they have introduced what they call test rules (see TestRule). You may start from this.
The ExpectedException class marked as #Rule could be of some help in order to check for exceptions thrown but not mandatory for the test to pass.
For more advanced usage I cannot say for the moment as I've just discovered it.

Get name of currently executing test in JUnit 4

In JUnit 3, I could get the name of the currently running test like this:
public class MyTest extends TestCase
{
public void testSomething()
{
System.out.println("Current test is " + getName());
...
}
}
which would print "Current test is testSomething".
Is there any out-of-the-box or simple way to do this in JUnit 4?
Background: Obviously, I don't want to just print the name of the test. I want to load test-specific data that is stored in a resource with the same name as the test. You know, convention over configuration and all that.

JUnit 4.7 added this feature it seems using TestName-Rule. Looks like this will get you the method name:
import org.junit.Rule;
public class NameRuleTest {
#Rule public TestName name = new TestName();
#Test public void testA() {
assertEquals("testA", name.getMethodName());
}
#Test public void testB() {
assertEquals("testB", name.getMethodName());
}
}

JUnit 4.9.x and higher
Since JUnit 4.9, the TestWatchman class has been deprecated in favour of the TestWatcher class, which has invocation:
#Rule
public TestRule watcher = new TestWatcher() {
protected void starting(Description description) {
System.out.println("Starting test: " + description.getMethodName());
}
};
Note: The containing class must be declared public.
JUnit 4.7.x - 4.8.x
The following approach will print method names for all tests in a class:
#Rule
public MethodRule watchman = new TestWatchman() {
public void starting(FrameworkMethod method) {
System.out.println("Starting test: " + method.getName());
}
};

JUnit 5 and higher
In JUnit 5 you can inject TestInfo which simplifies test metadata injection to test methods. For example:
#Test
#DisplayName("This is my test")
#Tag("It is my tag")
void test1(TestInfo testInfo) {
assertEquals("This is my test", testInfo.getDisplayName());
assertTrue(testInfo.getTags().contains("It is my tag"));
}
See more: JUnit 5 User guide, TestInfo javadoc.

Try this instead:
public class MyTest {
#Rule
public TestName testName = new TestName();
#Rule
public TestWatcher testWatcher = new TestWatcher() {
#Override
protected void starting(final Description description) {
String methodName = description.getMethodName();
String className = description.getClassName();
className = className.substring(className.lastIndexOf('.') + 1);
System.err.println("Starting JUnit-test: " + className + " " + methodName);
}
};
#Test
public void testA() {
assertEquals("testA", testName.getMethodName());
}
#Test
public void testB() {
assertEquals("testB", testName.getMethodName());
}
}
The output looks like this:
Starting JUnit-test: MyTest testA
Starting JUnit-test: MyTest testB
NOTE: This DOES NOT work if your test is a subclass of TestCase! The test runs but the #Rule code just never runs.

Consider using SLF4J (Simple Logging Facade for Java) provides some neat improvements using parameterized messages. Combining SLF4J with JUnit 4 rule implementations can provide more efficient test class logging techniques.
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.MethodRule;
import org.junit.rules.TestWatchman;
import org.junit.runners.model.FrameworkMethod;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggingTest {
#Rule public MethodRule watchman = new TestWatchman() {
public void starting(FrameworkMethod method) {
logger.info("{} being run...", method.getName());
}
};
final Logger logger =
LoggerFactory.getLogger(LoggingTest.class);
#Test
public void testA() {
}
#Test
public void testB() {
}
}

A convoluted way is to create your own Runner by subclassing org.junit.runners.BlockJUnit4ClassRunner.
You can then do something like this:
public class NameAwareRunner extends BlockJUnit4ClassRunner {
public NameAwareRunner(Class<?> aClass) throws InitializationError {
super(aClass);
}
#Override
protected Statement methodBlock(FrameworkMethod frameworkMethod) {
System.err.println(frameworkMethod.getName());
return super.methodBlock(frameworkMethod);
}
}
Then for each test class, you'll need to add a #RunWith(NameAwareRunner.class) annotation. Alternatively, you could put that annotation on a Test superclass if you don't want to remember it every time. This, of course, limits your selection of runners but that may be acceptable.
Also, it may take a little bit of kung fu to get the current test name out of the Runner and into your framework, but this at least gets you the name.

JUnit 4 does not have any out-of-the-box mechanism for a test case to get it’s own name (including during setup and teardown).

String testName = null;
StackTraceElement[] trace = Thread.currentThread().getStackTrace();
for (int i = trace.length - 1; i > 0; --i) {
StackTraceElement ste = trace[i];
try {
Class<?> cls = Class.forName(ste.getClassName());
Method method = cls.getDeclaredMethod(ste.getMethodName());
Test annotation = method.getAnnotation(Test.class);
if (annotation != null) {
testName = ste.getClassName() + "." + ste.getMethodName();
break;
}
} catch (ClassNotFoundException e) {
} catch (NoSuchMethodException e) {
} catch (SecurityException e) {
}
}

Based on the previous comment and further considering I created an extension of TestWather which you can use in your JUnit test methods with this:
public class ImportUtilsTest {
private static final Logger LOGGER = Logger.getLogger(ImportUtilsTest.class);
#Rule
public TestWatcher testWatcher = new JUnitHelper(LOGGER);
#Test
public test1(){
...
}
}
The test helper class is the next:
public class JUnitHelper extends TestWatcher {
private Logger LOGGER;
public JUnitHelper(Logger LOGGER) {
this.LOGGER = LOGGER;
}
#Override
protected void starting(final Description description) {
LOGGER.info("STARTED " + description.getMethodName());
}
#Override
protected void succeeded(Description description) {
LOGGER.info("SUCCESSFUL " + description.getMethodName());
}
#Override
protected void failed(Throwable e, Description description) {
LOGGER.error("FAILURE " + description.getMethodName());
}
}
Enjoy!

In JUnit 5 TestInfo acts as a drop-in replacement for the TestName rule from JUnit 4.
From the documentation :
TestInfo is used to inject information about the current test or
container into to #Test, #RepeatedTest, #ParameterizedTest,
#TestFactory, #BeforeEach, #AfterEach, #BeforeAll, and #AfterAll
methods.
To retrieve the method name of the current executed test, you have two options : String TestInfo.getDisplayName() and
Method TestInfo.getTestMethod().
To retrieve only the name of the current test method TestInfo.getDisplayName() may not be enough as the test method default display name is methodName(TypeArg1, TypeArg2, ... TypeArg3).
Duplicating method names in #DisplayName("..") is not necessary a good idea.
As alternative you could use
TestInfo.getTestMethod() that returns a Optional<Method> object.
If the retrieval method is used inside a test method, you don't even need to test the Optional wrapped value.
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.TestInfo;
import org.junit.jupiter.api.Test;
#Test
void doThat(TestInfo testInfo) throws Exception {
Assertions.assertEquals("doThat(TestInfo)",testInfo.getDisplayName());
Assertions.assertEquals("doThat",testInfo.getTestMethod().get().getName());
}

JUnit 5 via ExtensionContext
Advantage:
You get to have the added functionalities of ExtensionContext by overriding afterEach(ExtensionContext context).
public abstract class BaseTest {
protected WebDriver driver;
#RegisterExtension
AfterEachExtension afterEachExtension = new AfterEachExtension();
#BeforeEach
public void beforeEach() {
// Initialise driver
}
#AfterEach
public void afterEach() {
afterEachExtension.setDriver(driver);
}
}
public class AfterEachExtension implements AfterEachCallback {
private WebDriver driver;
public void setDriver(WebDriver driver) {
this.driver = driver;
}
#Override
public void afterEach(ExtensionContext context) {
String testMethodName = context.getTestMethod().orElseThrow().getName();
// Attach test steps, attach scsreenshots on failure only, etc.
driver.quit();
}
}

#ClassRule
public static TestRule watchman = new TestWatcher() {
#Override
protected void starting( final Description description ) {
String mN = description.getMethodName();
if ( mN == null ) {
mN = "setUpBeforeClass..";
}
final String s = StringTools.toString( "starting..JUnit-Test: %s.%s", description.getClassName(), mN );
System.err.println( s );
}
};

I usually use something like this:
/** Returns text with test method name
#param offset index of method on call stack to print, 1 for a caller of this method.
*/
static String getName(int offset)
{
Throwable t = new Throwable();
t.fillInStackTrace();
return
t.getStackTrace()[offset].getMethodName()+":"+t.getStackTrace()[offset].getLineNumber();
};
This is exactly what Exception do use when printing stack trace.
Depending on the exact context You may have to figure out correct offset value. It is crude and primitive tough and is not using any fancy modern futures.

I'd suggest you decouple the test method name from your test data set. I would model a DataLoaderFactory class which loads/caches the sets of test data from your resources, and then in your test case cam call some interface method which returns a set of test data for the test case. Having the test data tied to the test method name assumes the test data can only be used once, where in most case i'd suggest that the same test data in uses in multiple tests to verify various aspects of your business logic.

You can achieve this using Slf4j and TestWatcher
private static Logger _log = LoggerFactory.getLogger(SampleTest.class.getName());
#Rule
public TestWatcher watchman = new TestWatcher() {
#Override
public void starting(final Description method) {
_log.info("being run..." + method.getMethodName());
}
};

I have a Junit4 test class that extends TestCase so the example with #Rule didn't work (as mentioned in other answers).
However, if your class extends TestCase you can use getName() to get the current test name so this works:
#Before
public void setUp() {
System.out.println("Start test: " + getName());
}
#After
public void tearDown() {
System.out.println("Finish test: " + getName());
}

A more simpler way is to put this logic in setUp() and tearDown() methods.
Refer below code for better clarity,
import java.lang.reflect.Method;
#BeforeMethod
void setUp(Method method) {
log.info("###############################################");
log.info("Running Test: {}", method.getName());
}
#AfterMethod
void tearDown(Method method) {
log.info("Finished Test: {}", method.getName());
log.info("###############################################");
}
#Test
public void testMethodName() {
// Method logic implementation...
}
Here is the output of above test execution,
#############################################################
Running Test: testMethodName
// Logs related to method execution...
Finished Test: testMethodName
#############################################################

To find the number of test methods in a JUnit TestCase

Is there a way to know the number of test methods in a test case?
What I want to do is have a test case which tests several scenarios and for all these i would be doing the data setUp() only once. Similarly I would like to do the cleanup (tearDown()) once at the end of all the test methods.
The current approach i am using is to maintain a counter for the number of test methods that are present in the file and decrement them in the tearDown method and do the cleanup when the count reaches 0. But this counter needs to be taken care of whenever new test methods are added.

Instead of using setup/teardown you should probably use methods annotated with #BeforeClass and #AfterClass instead.

You can do this through #BeforeClass and #AfterClass in JUnit4:
http://junit.org/apidocs/org/junit/BeforeClass.html
Volker

Here is the piece of code I wrote to find all the test cases in my JUnit project. What it does is reads the files(under package mentioned in code) and using reflection APIs, finds the test cases with annotations "#Test" and also the ones which start with "test" but don't have the #Test annotation
public class TestCaseCount {
private static List<Class> getClasses(String packageName)
throws ClassNotFoundException, IOException {
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
assert classLoader != null;
String path = packageName.replace('.', '/');
Enumeration<URL> resources = classLoader.getResources(path);
List<File> dirs = new ArrayList<File>();
while (resources.hasMoreElements()) {
URL resource = resources.nextElement();
dirs.add(new File(resource.getFile()));
}
ArrayList<Class> classes = new ArrayList<Class>();
for (File directory : dirs) {
classes.addAll(findClasses(directory, packageName));
}
return classes /* .toArray(new Class[classes.size()]) */;
}
private static List<Class> findClasses(File directory, String packageName)
throws ClassNotFoundException {
List<Class> classes = new ArrayList<Class>();
if (!directory.exists()) {
return classes;
}
File[] files = directory.listFiles();
for (File file : files) {
if (file.isDirectory()) {
assert !file.getName().contains(".");
classes.addAll(findClasses(file, packageName + "." + file.getName()));
} else if (file.getName().endsWith(".class")) {
classes.add(Class.forName(packageName + '.'
+ file.getName().substring(0, file.getName().length() - 6)));
}
}
return classes;
}
public static void main(String args[]) {
ArrayList<Class> classes = new ArrayList<Class>();
try {
// Feature1 Test Cases
classes.addAll(TestCaseCount.getClasses("mypackage.feature1.tests"));
// Feature2 Test Cases
classes.addAll(TestCaseCount.getClasses("mypackage.feature2.tests1"));
classes.addAll(TestCaseCount.getClasses("mypackage.feature2.tests2"));
// Feature3 Test Cases
classes.addAll(TestCaseCount.getClasses("mypackage.feature3.tests"));
} catch (Exception e) {
e.printStackTrace();
}
int testcaseCount = 0;
for (Class cl : classes) {
System.out.println("Test Class Name : " + cl.getName());
Method[] methods = cl.getDeclaredMethods();
for (Method method : methods) {
Annotation[] annotations = method.getDeclaredAnnotations();
if (annotations.length == 0 && method.getName().startsWith("test")) {
testcaseCount++;
} else {
for (Annotation annotation : annotations) {
if (annotation.annotationType().toString()
.equals("interface org.junit.Test")) {
testcaseCount++;
}
}
}
}
}
System.out.println("Total Test Cases " + testcaseCount);
}
}

Short example for counting tests with #BeforeClass, #AfterClass and #Before.
public class CountTest {
static int count;
#BeforeClass
public static void beforeClass() {
count = 0;
}
#Before
public void countUp() {
count++;
}
#AfterClass
public static void printCount() {
System.out.println(count + " tests.");
}
#Test
public void test1() {
assertTrue(true);
}
// some more tests
Output will be, e.g.:
5 tests.

If you are using Junit4 and the suggestion given by others is the correct one. But if you using earlier version then use this technique to achieve what you want -
You can define a suite for all those tests for which you want to setup and teardown only once. Take a look at junit.extensions.TestSetup class. Instead of executing your test classes you need to then execute these suites.

A solution for junit 3 is to call a special setup method in every test which checks a static flag. if the flag is not set, run the global setup. If it is, skip the setup.
Make sure the global setup is properly synchronized if you want to run tests in parallel.

Using #Rules on TestWatcher you can take note of count and many other things like method name etc. You can override these methods and use .
#Override
public Statement apply(Statement base, Description description){
return super.apply(base, description);
}
#Override
protected void failed(Throwable e, Description description) {
failed.add(description);
LogUtil.error("[FAILED] "+description.getMethodName()+" [Test Failed]"+e.getMessage());
super.failed(e, description);
}
#Override
protected void finished(Description description) {
LogUtil.info("[FINISHED] "+description.getMethodName());
super.finished(description);
}
#Override
protected void skipped(AssumptionViolatedException e,Description description) {
LogUtil.error("[FAILED] Test Failed due to Assumption Voilation "+e.getMessage());
super.skipped(e,description);
}
#Override
protected void starting(Description description) {
LogUtil.info("-----------------------------------------------------------");
LogUtil.info("[STARTED] "+description.getMethodName());
super.starting(description);
}
#Override
protected void succeeded(Description description) {
passed.add(description);
LogUtil.info("[PASSED] "+description.getMethodName());
super.succeeded(description);
}
In your Junit Testcase Use
#Rule
public TestRule watcher = new TestWatcherChild();