Develop Reference

testng - creating KnownFault and IntermittentFailure annotations

I would like to annotate some of my test cases with KnownFault - which would do pretty much what expectedException does plus some magic using YouTrack's REST API. I would also like to have an IntermittentFailure attribute which would mean that I'm aware that the test might occasionally fail with [exception] [message] but I wouldn't want this to block the rest of my build chain.
After some research I found that my test class should implement IHookable, then I could have something like this:
#Override
public void run(IHookCallBack callBack, ITestResult result) {
callBack.runTestMethod(result);
if (result.getThrowable().getCause() instanceof IllegalArgumentException){
System.out.println("This is expected.");
result.setThrowable(null);
}
else{
System.out.println("Unexpected exception");
}
}
The problem with this is the actual implementation of invokeHookable:
final Throwable[] error = new Throwable[1];
IHookCallBack callback = new IHookCallBack() {
#Override
public void runTestMethod(ITestResult tr) {
try {
invokeMethod(thisMethod, testInstance, parameters);
} catch (Throwable t) {
error[0] = t;
tr.setThrowable(t); // make Throwable available to IHookable
}
}
#Override
public Object[] getParameters() {
return parameters;
}
};
hookable.run(callback, testResult);
if (error[0] != null) {
throw error[0];
}
Unfortunately that last line means that my test case is going to throw an exception no matter what as the error array is completely out of my hands in the run method.
So, what would be the proper way of intercepting an exception and handling it the way I want to?

What you are trying to do is really interesting. You should try to propose changes on https://github.com/cbeust/testng/pull/
But maybe IHookable is not the best listener you can use. Did you try IInvokedMethodListener?
void afterInvocation(IInvokedMethod method, ITestResult result) {
if (result.getThrowable().getCause() instanceof IllegalArgumentException) {
System.out.println("This is expected.");
result.setThrowable(null);
result.setStatus(SUCCESS); // If you want to change the status
} else {
System.out.println("Unexpected exception");
}
}

How to check multiple exceptions with one JUnit Method?

i have this code in my program which is needed to be tested with jUnit
void deleteCustomer(String name) throws UnknownCustomerException,
AccountNotEmptyException {
if (name == null) {
throw new NullPointerException();
} else if (!exists(name)) {
throw new UnknownCustomerException();
} else if (getCustomer(name).deletable()) {
customerList.remove(getCustomer(name));
}
}
I thought i can test it in one JUnit method like
#Test
public void createCustomer(){
System.out.println("createCustomerTest");
try {
element.createCustomer(null);
//fail("Expected an IndexOutOfBoundsException to be thrown");
} catch (NullPointerException anIndexOutOfBoundsException) {
assertTrue(anIndexOutOfBoundsException.getMessage().equals("NullPointerException"));
}
}
As you can see I already tried unsuccessfully to implement the NPE.
How can I check for several Exceptions in one JUnit Method? I checked some How-To's in the web but failed with that too.

I think in your case you should have separate tests, however you can achieve this like so if using Java 8:
Using an AssertJ 3 assertion, which can be used alongside JUnit:
import static org.assertj.core.api.Assertions.*;
#Test
public void test() {
Element element = new Element();
assertThatThrownBy(() -> element.createCustomer(null))
.isInstanceOf(NullPointerException.class)
.hasMessageContaining("NullPointerException");
assertThatThrownBy(() -> element.get(1))
.isInstanceOf(IndexOutOfBoundsException.class);
}
It's better than #Test(expected=IndexOutOfBoundsException.class) or .expect syntax because it guarantees the expected line in the test threw the exception and lets you check more details about the exception, such as message.
Maven/Gradle instructions here.

Write for each exception its own test. It will be only one thrown at a time anyway.
For example a simplified method:
void deleteCustomer( String name ) throws UnknownCustomerException
{
if ( name == null )
{
throw new NullPointerException();
}
else if ( !exists( name ) )
{
throw new UnknownCustomerException();
}
}
You have then two tests that each check if its exception is thrown:
#Test( expected = NullPointerException.class )
public void deleteCustomer_shouldThrowNullpointerIfNameIsNull() throws UnknownCustomerException
{
String name = null;
cut.deleteCustomer( name );
}
#Test( expected = UnknownCustomerException.class )
public void deleteCustomer_shouldThrowUnknownCustomerExceptionIfNameIsUnknown() throws UnknownCustomerException
{
String name = "someUnknownName";
cut.deleteCustomer( name );
}
The problem with the NullpointerException is, that the test is true/successful/green if the NPE is thrown anywhere in the method - so you should make sure, that that is not happening for the test to be meaningful.

You could add several "catch" statement into the test method for different exceptions, like:
try {
element.createCustomer(null);
Assert.fail("Exception was expected!");
} catch (NullPointerException _ignore) {
} catch (UnknownCustomerException _ignore) {
}
or with Java 87
try {
element.createCustomer(null);
Assert.fail("Exception was expected!");
} catch (NullPointerException | UnknownCustomerException _ignore) {
}
But if you switch from JUnit to TestNG, then your test will be much cleaner:
#org.testng.annotations.Test(expectedExceptions = { NullPointerException.class, UnknownCustomerException.class })
public void createCustomer() throws NullPointerException, UnknownCustomerException {
element.createCustomer(null);
}
More information about "expectedException" is here: http://testng.org/doc/documentation-main.html and example of the usage can be found here: http://www.mkyong.com/unittest/testng-tutorial-2-expected-exception-test/

I suggest that you take a closer look at the JavaDoc of ExpectedException and implement different tests for different validations, e.g.
public class CustomerTest {
#Rule
public ExpectedException exception = ExpectedException.none();
#Test
public void throwsNullPointerExceptionForNullArg() {
exception.expect(NullPointerException.class);
element.createCustomer(null);
}
#Test
public void throwsUnknwonCustomerExceptionForUnkownCustomer() {
exception.expect(UnknownCustomerException.class);
// exception.expectMessage("Some exception message"); uncomment to verify exception message
element.createCustomer("unknownCustomerName");
}
#Test
public void doesNotThrowExceptionForKnownCustomer() {
element.createCustomer("a known customer");
// this test pass since ExpectedException.none() defaults to no exception
}
}

Test code with Exceptions and with no exceptions at same time (JAVA)

I am writing a test for already built java class function. I am writing tests using Testng and Mockito and have a Data Provider.
This is my Test
#Test(dataProvider = "myProvider", dataProviderClass = StaticDataProvider.class,
expectedExceptions = SomeException.class)
public void myControllerTest(String argument) throws Exception {
// Mocked object bussiness\
Boolean resultantObject = business.getList(argument);
Assert.assertTrue(resultantObject);
}
This is my Controller which I want to test
public Boolean controller(String argument) {
if(argument != null) {
throw new someException();
} else {
System.out.println("Sucess");
return true;
}
}
This is my Data Providor
#DataProvider(name = "myProvider")
public static Object[][] getDirectoryList() throws Exception {
Object[][] result = null;
// case1 throws SomeException
String testData1 = null;
// case2 don't throw exception
String testData2 = "String";
result = new Object[][] { { testData1 }, { testData2 } };
return result;
}
The problem here I am facing is, I don't want to create another test just to test both buggy and non buggy code and complete my test coverage using a single test case. But when I put Expected Exception on top, it fails on correct code, and when I dont, it fails on buggy code.
NOTE: This is example code and may not work, this is just to take an idea of scenario I am working on and what I am expecting.

Even if you ignore the "one test, one assertion" purist perspective, I think most people agree you should split tests that involve error conditions from tests that prove normal behaviour.
If you want to test multiple error conditions within one test (or if you're really keen on continuing with your plan), you can use this pattern:
try {
// something that should cause an exception
fail("Exception expected");
} catch (ExactlyTheRightException e) {
// ignored
}

How to execute code which comes after a section where element is not found in a test method?

Suppose, I am testing method which checks whether all links are present on page. Ex :
#Test
public void testLinks(){
driver.findElement(By.linkText("link1"));
driver.findElement(By.linkText("link2"));
driver.findElement(By.linkText("link3"));
driver.findElement(By.linkText("link4"));
driver.findElement(By.linkText("link5"));
}
In above code, suppose all links are present except link3 then test method's execution will stop after throwing error for link3 but still if I want code to be executed for checking link4 and link5, how can I achieve that using Java?

I think you mean throwing exceptions.
You can test links one by one in another method which handle exceptions.
public void testOne(String link) {
try {
driver.findElement(By.linkText(link));
}
catch (Exception e) {
System.out.println(link+" failed to find");
}
}
public void test() {
testOne("link1");
testOne("link2");
testOne("link3");
testOne("link4");
testOne("link5");
}
The testOne() method will catch exceptions and print a failed note, and your test method won't be disrupted.

Try this, you can use TestNg reporter to make your test pass or fail or log4j logs, try this.
sResult = "Pass";
public void testOne(String link) {
try {
driver.findElement(By.linkText(link));
}
catch (Exception e) {
System.out.println(link+" failed to find");
sResult = "Fail";
}
}
public void test() {
testOne("link1");
testOne("link2");
testOne("link3");
testOne("link4");
testOne("link5");
}
if(sResult.equals("Pass")){
System.out.println("Pass");
}else{
System.out.println("Fail");
}

Checking for 2 expected values in Junit

I have a java program which throws an exception with 2 different messages for 2 different scenarios and I want the Junit test case to check for equality for both of these messages. As an example -
public void amethod() {
// do some processing
if(scenario1 == true) {
throw new MySystemException("An error occured due to case 1 being incorrect.");
}
else if(scenario2 == true) {
throw new MySystemException("An error occured as case 2 could not be found");
}
}
Now the JUnit for this would be something like-
public void testAMethod() {
// do something
assertEquals("Expected", "Actual");
}
As I understand, in this above example, if I use the Scenario1 exception message the junit will fail when an exception is thrown for Scenario2 and vice versa.
I would like to know if there is any other way provided in Junit by which I can use this one test method and check for both the messages for the test to pass?
Something like an OR, if possible to provide the "Expected" value with both these expected message.
I hope my query is clear enough.
Thanks
UPDATE
Sorry for the delayed response, had got caught up with some other urgent matter.
Thank you all for the very nice suggestions, it certainly has helped me to understand a bit better now.
Eventually, to keep it rather simple I decided to implement a somewhat similar solution suggested by Don Roby. So created a new test class which looks like -
public void testAMethodScenario1() {
// do the necessary
assertEquals("Expected Exception Message 1", "Actual");
}
public void testAMethodScenario2() {
// do the necessary
assertEquals("Expected Exception Message 2", "Actual");
}
Thank you all again for your responses.

I think you need to manually catch the exception (for each scenario) and individually check the message:
try {
// trigger scenario 1
fail("An exception should have been thrown here !");
} catch (MySystemException e1) {
assertEquals("Wrong error message", m1, e1.getMessage());
}
try {
// trigger scenario 2
fail("An exception should have been thrown here !");
} catch (MySystemException e2) {
assertEquals("Wrong error message", m2, e2.getMessage());
}
Of course, you can have these scenarios defined as enum constants and simply iterate through them and check each of them within a loop, since the "copy/paste design pattern" is pretty obvious in the above code. :)

You seem to be asking two things here, how to test an exception and how to assert that a value matches either of two possible expected values.
To test for an exception, you can either use a JUnit4 annotation:
#Test(expected=MySystemException.class)
public void testException() {
amethod();
}
or use a try-catch in your test:
#Test
public void testException() {
try {
amethod();
fail("MySystemException expected");
}
catch (MySystemException e) {
// Success!
}
}
And if you have only one message, in the try-catch version you can assert that you got it with an AssertEquals in the catch block.
The best testing would have separate tests for your two scenarios, and expect the correct single message. Better code might in fact have distinct exceptions for the two situations.
But the need for a more complex assertion than simple equality does come up anyway, and there's an elegant solution for it in Hamcrest matchers.
Using that for this situation, you could write something like (untested - don't trust my syntax completely):
#Test
public void testException() {
try {
amethod();
fail("MySystemException expected");
}
catch (MySystemException e) {
String expectedMessage1 = "An error occured due to case 1 being incorrect.";
String expectedMessage2 = "An error occured as case 2 could not be found";
assertThat(e.getMessage(),
anyOf(equalTo(expectedMessage1), equalTo(expectedMessage2)));
}
}

Can you predict which scenario will occur? If so, Costi's answer is correct. If not, because there's some randomness or whatever, you can write:
#Test
public void testAmethodThrowsException() {
try {
amethod();
fail("amethod() should have thrown an exception");
}
catch (MySystemException e) {
String msg = e.getMessage();
assertTrue("bad message: " + msg, msg.equals("An error occured due to case 1 being incorrect.") || msg.equals("An error occured as case 2 could not be found"));
}
}

The declared types of exception thrown bya method are part of its API. If you really want to distinguish different failure modes, you should declare a different exception type for each failure mode.
So, something like this:
/**
* Do something.
* #throws MySystemException1 in case 1.
* #throws MySystemException2 if Foo not found.
*/
public void amethod() {
// do some processing
if(scenario1 == true) {
throw new MySystemException1("Case 1.");
}
else if(scenario2 == true) {
throw new MySystemException2("Foo not found");
}
}

#Rule solution in JUnit4:
public class ExceptionRule implements MethodRule {
#Override
public Statement apply(final Statement base, final FrameworkMethod method, Object target) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
try {
base.evaluate();
Assert.fail();
} catch (MySystemException e) {
if(scenario1)
assertEquals("Expected error message1", e1.getMessage();
if(scenario2)
assertEquals("Expected error message2",e1.getMessage();
}
}
};
}
}
In your testcase, use the Rule:
#Rule public ExceptionRule rule = new ExceptionRule();

JUnit 4 provides (Expected Exception.class)
#Test(expected= MySystemException.class) public void empty() {
// what ever you want
}
Google: Expected Exceptions JUnit for more info.

BDD Style Solution with Catch Exception
#Test
public void testAMethodScenario1() {
//given scenario 1
when(foo).amethod();
then(caughtException())
.isInstanceOf(MySystemException.class)
.hasMessage("An error occured due to case 1 being incorrect.");
}
#Test
public void testAMethodScenario2() {
//given scenario 2
when(foo).amethod();
then(caughtException())
.isInstanceOf(MySystemException.class)
.hasMessage("An error occured as case 2 could not be found");
}
Source code
https://gist.github.com/mariuszs/7490875
Dependencies
com.googlecode.catch-exception:catch-exception:1.2.0

A better solution with #Rule, you can assert both exception and expection message as well.
#Rule
public ExpectedException expectedException = ExpectedException.none();
#Test
public void aMethod_Scenario1True_ThrowsException() {
expectedException.expect(MySystemException.class);
expectedExcepion.expectMessage("An error occured due to case 1 being incorrect.");
//when().thenReturn();
//handle the repositories, static methods and other sub methods, if needed
amethod();
}
#Rule is the more elegant way to write the exception.