Develop Reference

In Selenium, if a step fails for a test case, is it possible to just report the failure and continue with remaining steps?

In Selenium, if a step fails for a test case, is it possible to just report the failure and continue with remaining steps? Currently the execution halts if there is an exception. This is what my Test case looks like-
public class TC002_abc extends OpentapWrappers
{
#Test (description="Test")
public void main()
{
try
{
WebDriverWait wait=new WebDriverWait(driver, 60);
VerifyTitle(Constant.HomePage_Title);
Click(HomePage.link_Login(driver), "Login Link");
wait.until(ExpectedConditions.urlContains(Constant.LoginURL));
VerifyTextPopulated(CommunicationPref.lbl_EmailAddress_Input(driver), Constant.EmailAddress);
/* Validate Email Communications */
Click(CommunicationPref.link_EditEmailCommunications(driver),"Edit Email Communications");
VerifyText(CommunicationPref.lbl_UnCheckedEmailCommunications(driver), Constant.UnCheckedEmailCommunications_Text);
Click(CommunicationPref.btn_EmailCommunicationsSave(driver), "Save");
VerifyText(CommunicationPref.lbl_CheckedEmailCommunications(driver), Constant.CheckedEmailCommunications_Text);
}
catch (NoSuchElementException e)
{
e.printStackTrace();
Reporter.reportStep("NoSuchElementException" , "FAIL");
}
}
#BeforeClass
public void beforeClass()
{
browserName="firefox";
testCaseName = "TC002_abc";
testDescription = "Test";
}
}
Sample Method-
public static void VerifyTitle(String title){
try
{
if (driver.getTitle().equalsIgnoreCase(title))
{
Reporter.reportStep("Page is successfully loaded :"+title, "PASS");
}
else
Reporter.reportStep("Page Title :"+driver.getTitle()+" did not match with :"+title, "FAIL");
}
catch (Exception e)
{
e.printStackTrace();
Reporter.reportStep("The title did not match", "FAIL");
}
}

Since you're using TestNG, implement a Soft Assertion
public void VerifyTitle(String title)
{
SoftAssert assertion = new SoftAssert();
String returnedTitle = driver.getTitle();
if (assertion.assertTrue(returnedTitle.contains(title)))
{
Reporter.reportStep("Page is successfully loaded :" + title, "PASS");
} else
{
Reporter.reportStep("Page Title :" + driver.getTitle() + " did not match with :" + title, "FAIL");
}
}
Let me know if this helps.

if a step fails for a test case, is it possible to just report the
failure and continue with remaining steps?
Short answer: YES
Long answer: YES
Selenium is framework built on top of a test engine such as JUnit or TestNG. On those engines, if you do nothing, the tool will interpret as a pass. In other words, in the absence of an assertion, the engines will assume the test passed. Since Selenium is built on top of this, the same can be said about Selenium. The code snippet below is a representation of what a Cucumber step looks like.
#When("my test step here")
public void myTestStep(...) {
boolean result = false;
try {
result = myTest(...);
}
} catch (Exception e) {
// log your exception (don't rethrow)
}
if (result) {
// log your passing test
} else {
// log your failing test
Assert.fail(); // This is what prevents subsequent steps to be executed. Remove it, and you should be able to continue to test.
}
For a JUnit or TestNG style method is basically the same. You may have an #AfterClass or #AfterTest hook that might handle telling the test framework to pass of fail test. TYPICALLY, passing assertions are implied (by not doing anything - i.e. executing an empty method). However, failing assertions are EXPLICIT and must be included somewhere. Just look for those Assert.fail() methods and remove them. A better alternative is to add a configurable property to your test suite that will turn this on or off.
} else {
// log your failing test
if (skip_off) {
Assert.fail(); // This is what prevents subsequent steps to be executed. Remove it, and you should be able to continue to test.
}
}
In this context, skip_off is the value of Boolean property you might have stored in a configuration file that when set to true, it will skip enforcing fail assertions.

JUnit tests: How to check for errors with a try-catch block

So, I need to write a test for some (legacy) code I'm improving. In a method, I try to parse some string (which should be legal JSON). Then a possible JSONException is caught if the string doesn't represents valid JSON. Something like:
public void transformToJSON(String source) {
try {
JSONObject js = new JSONObject(new JSONTokener(item.getHtml()));
}
catch (JSONException e) {
log(e)
}
//than js is added to an Hashset and the method is done
}
So I want to write a test for good input (to see if I have generated a correct JSON-object). This is 'easy' by checking the object in the Set.
For wrong input however, I need to find out if the correct error has been thrown.
I know if an error was thrown in the code, I can check for it in the test.
By setting the rule public ExpectedException thrown=
ExpectedException.none(); and checking for it in test method.
By adding #Test(expected = JSONException.class) above the test
But both wont work for try..catch blocks.
How can I test if the proper exception is caught by catch block? I want to change as little of the source code as possible.

In the JUnit test class you can do is use fail("this should not have happened") in the try or catch block depending on what should and should not work (as in: try and catch in the JUnit class, not in your actual method!).
However, with a try/catch block within your method you cannot see whether an Exception occured or not, because it is handled within the method. So you would have to throw the exception in the method instead of catching it, i.e.,
public void transformToJSON(String source) throws JSONException { ... }
Then it will work to check whether an exception occured or not.
Alternatively you could return a boolean that states whether the transformation was successful or not. Then you can test whether the return value was true/false and if that was what you expected.
public boolean transformToJSON(String source) {
boolean success = true;
try {
JSONObject js = new JSONObject(new JSONTokener(item.getHtml()));
}
catch (JSONException e) {
log(e)
success = false;
}
//than js is added to an Hashset and the method is done
return success;
}
In your test class:
#Test
public void testTransformToJSON() {
assertTrue(transformToJSON("whatever"));
}

Based on the logging being used in the code, you can use Mockito to verify the message logged inside catch block.
Please go through the following link for more details on setting up the unit tests
http://bloodredsun.com/2010/12/09/checking-logging-in-unit-tests/

Your legacy code is swallowing the Exception. If it throws an exception, then your junit #Test ( expected = JSONException.class) would work.

I'd change the code slightly so it is
public void transformToJSON(String source) {
try {
JSONObject js = getJSON(item.getHtml()));
}
catch (JSONException e) {
log(e)
}
//than js is added to an Hashset and the method is done
}
public JSONObject getJSON(String source) throws JSONException {
return new JSONObject(new JSONTokener(source));
}
and then test against getJSON. This throws an exception and as other have said (and you) you can use the expectedException in the test class

use a bad formatted json string, and then do assertions or whatever in the catch block of ur test.
#Test
public void shouldCatchException(){
String source = "{ \"name\":\"John\", \"age\":30, \"car\":null ";
try {
jsonHelper.transformToJSON(source);
}catch (JSONException e){
Assert.assertThat(e, notNullValue());
assertTrue(StringUtils.isNotBlank(e.getMessage());
//whatever else u need to assert
}
}

Test code with Exceptions and with no exceptions at same time (JAVA)

I am writing a test for already built java class function. I am writing tests using Testng and Mockito and have a Data Provider.
This is my Test
#Test(dataProvider = "myProvider", dataProviderClass = StaticDataProvider.class,
expectedExceptions = SomeException.class)
public void myControllerTest(String argument) throws Exception {
// Mocked object bussiness\
Boolean resultantObject = business.getList(argument);
Assert.assertTrue(resultantObject);
}
This is my Controller which I want to test
public Boolean controller(String argument) {
if(argument != null) {
throw new someException();
} else {
System.out.println("Sucess");
return true;
}
}
This is my Data Providor
#DataProvider(name = "myProvider")
public static Object[][] getDirectoryList() throws Exception {
Object[][] result = null;
// case1 throws SomeException
String testData1 = null;
// case2 don't throw exception
String testData2 = "String";
result = new Object[][] { { testData1 }, { testData2 } };
return result;
}
The problem here I am facing is, I don't want to create another test just to test both buggy and non buggy code and complete my test coverage using a single test case. But when I put Expected Exception on top, it fails on correct code, and when I dont, it fails on buggy code.
NOTE: This is example code and may not work, this is just to take an idea of scenario I am working on and what I am expecting.

Even if you ignore the "one test, one assertion" purist perspective, I think most people agree you should split tests that involve error conditions from tests that prove normal behaviour.
If you want to test multiple error conditions within one test (or if you're really keen on continuing with your plan), you can use this pattern:
try {
// something that should cause an exception
fail("Exception expected");
} catch (ExactlyTheRightException e) {
// ignored
}

Try / Catch in Constructor - Recommended Practice?

Something I've always been curious of
public class FileDataValidator {
private String[] lineData;
public FileDataValidator(String[] lineData){
this.lineData = lineData;
removeLeadingAndTrailingQuotes();
try
{
validateName();
validateAge();
validateTown();
}
catch(InvalidFormatException e)
{
e.printStackTrace();
}
}
//validation methods below all throwing InvalidFormatException
Is is not advisable to include the try/catch block within my Constructor?
I know I could have the Constructor throw the Exception back to the caller. What do you guys prefer in calling methods like I have done in Constructor? In the calling class would you prefer creating an instance of FileDataValidator and calling the methods there on that instance? Just interested to hear some feedback!

In the code you show, the validation problems don't communicate back to the code that is creating this object instance. That's probably not a GOOD THING.
Variation 1:
If you catch the exception inside the method/constructor, be sure to pass something back to the caller. You could put a field isValid that gets set to true if all works. That would look like this:
private boolean isValid = false;
public FileDataValidator(String[] lineData){
this.lineData = lineData;
removeLeadingAndTrailingQuotes();
try
{
validateName();
validateAge();
validateTown();
isValid = true;
}
catch(InvalidFormatException e)
{
isValid = false;
}
}
public boolean isValid() {
return isValid;
}
Variation 2:
Or you could let the exception or some other exception propagate to the caller. I have shown it as a non-checked exception but do whatever works according to your exception handling religion:
public FileDataValidator(String[] lineData){
this.lineData = lineData;
removeLeadingAndTrailingQuotes();
try
{
validateName();
validateAge();
validateTown();
}
catch(InvalidFormatException e)
{
throw new com.myco.myapp.errors.InvalidDataException(e.getMessage());
}
}
Variation 3:
The third method I want to mention has code like this. In the calling code you have to call the constructor and then call the build() function which will either work or not.
String[] lineData = readLineData();
FileDataValidator onePerson = new FileDataValidator();
try {
onePerson.build(lineData);
} catch (InvalidDataException e) {
// What to do it its bad?
}
Here is the class code:
public FileDataValidator() {
// maybe you need some code in here, maybe not
}
public void build(String[] lineData){
this.lineData = lineData;
removeLeadingAndTrailingQuotes();
try
{
validateName();
validateAge();
validateTown();
}
catch(InvalidFormatException e)
{
throw new com.myco.myapp.errors.InvalidDataException(e.getMessage());
}
}
Of course, the build() function could use a isValid() method that you call to see if its right but an exception seems the right way to me for the build function.
Variation 4:
The fourth method I want to mention is what I like best. It has code like this. In the calling code you have to call the constructor and then call the build() function which will either work or not.
This sort of follows the way JaxB and JaxRS work, which is a similar situation to what you have.
An external source of data - you have a file, they have an incoming message in XML or JSON format.
Code to build the objects - you have your code, they have their libraries of code working according the specifications in the various JSRs.
Validation is not tied to the building of the objects.
The calling code:
String[] lineData = readLineData();
Person onePerson = new Person();
FileDataUtilities util = new FileDataUtilities();
try {
util.build(onePerson, lineData);
util.validate(onePerson);
} catch (InvalidDataException e) {
// What to do it its bad?
}
Here is the class code where the data lives:
public class Person {
private Name name;
private Age age;
private Town town;
... lots more stuff here ...
}
And the utility code to build and validate:
public FileDataValidator() {
// maybe you need some code in here, maybe not
}
public void build(Person person, String[] lineData){
this.lineData = lineData;
removeLeadingAndTrailingQuotes();
setNameFromData(person);
setAgeFromData(person);
setTownFromData(person);
}
public boolean validate(Person person) {
try
{
validateName(person);
validateAge(person);
validateTown(person);
return true;
}
catch(InvalidFormatException e)
{
throw new com.myco.myapp.errors.InvalidDataException(e.getMessage());
}
}

You should consider the static factory pattern. Make your all-arguments constructor private. Provide a static FileDataValidator(args...) method. This accepts and validates all the arguments. If everything is fine, it can call the private constructor and return the newly created object. If anything fails, throw an Exception to inform the caller that it provided bad values.
I must also mention that this:
catch (Exception e) {
printSomeThing(e);
}
Is the deadliest antipattern you could do with Exceptions. Yes, you can read some error values on the command line, and then? The caller (who provided the bad values) doesn't get informed of the bad values, the program execution will continue.

My preference is for exceptions to be dealt with by the bit of code that knows how to deal with them. In this case I would assume that the bit of code creating a FileDataValidator knows what should happen if the file data is not valid, and the exceptions should be dealt with there (I am advocating propagating to the caller).
Whilst discussing best practice - the class name FileDataValidator smells to me. If the object you're creating stores file data then I would call it FileData - perhaps with a validate method? If you only want to validate your file data then a static method would suffice.

How do I write a unit test to cover the case where an IOException is thrown?

I have the following class:
public class FileLoader {
private Map<Brand, String> termsOfUseText = new HashMap<Brand, String>();
public void load() {
for (Brand brand : Brand.values()) {
readAndStoreTermsOfUseForBrand(brand);
}
}
private void readAndStoreTermsOfUseForBrand(Brand brand) {
String resourceName = "termsOfUse/" + brand.name().toLowerCase() + ".txt";
InputStream in = this.getClass().getClassLoader().getResourceAsStream(resourceName);
try {
String content = IOUtils.toString(in);
termsOfUseText.put(brand, content);
} catch (IOException e) {
throw new IllegalStateException(String.format("Failed to find terms of use source file %s", resourceName),e);
}
}
public String getTextForBrand(Brand brand) {
return termsOfUseText.get(brand);
}
}
Brand is an enum, and I need all the valid .txt files to be on the classpath. How do I make the IOException occur, given that the Brand enum contains all the valid brands and therfore all the .txt files for them exist?
Suggestions around refactoring the current code are welcome if it makes it more testable!

Three options I see right off:
Use PowerMock to mock IOUtils.toString(). I consider PowerMock to be quite a last resort. I'd rather refactor the source to something a little more test-friendly.
Extract the IOUtils call to a protected method. Create a test-specific subclass of your class that overrides this method and throws the IOException.
Extract the InputStream creation to a protected method. Create a test-specific subclass to override the method and return a mock InputStream.

I would suggest a bit of refactoring. All your methods are void, this usually means they are not functional.
For example, you can extract this functionality:
private String readTermsOfUseForBrand(InputStream termsOfUserIs) {
try {
String content = IOUtils.toString(in);
return content;
} catch (IOException e) {
throw new IllegalStateException(String.format("Failed to find terms of use source file %s", resourceName), e);
}
return null;
}
So that we can assert on the String result in our tests.
Of course this is not functional code, as it reads from an Input Stream. And it does so with IOUtils.toString() method that cannot be mocked easily (well, there's PowerMock but as Ryan Stewart said it's the last resort).
To test IO exceptions you can create a failing input stream (tested with JDK7):
public class FailingInputStream extends InputStream {
#Override
public int read() throws IOException {
throw new IOException("Test generated exception");
}
}
And test like that:
#Test
public void testReadTermsOfUseForBrand() {
FileLoader instance = new FileLoader();
String result = instance.readTermsOfUseForBrand(new FailingInputStream());
assertNull(result);
}

Missing file will cause NullPointerException because getResourceAsStream will return null and you will have in==null. IOException in this case may actually be pretty rare. If it's critical for you to see it, I can only think of instrumenting this code to throw it if code is executed in test scope. But is it really that important?

I would use a mock to accomplish this.
Example (untested, just to give you some thought):
#Test(expected=IllegalStateException.class)
public void testThrowIOException() {
PowerMockito.mockStatic(IOUtils.class);
PowerMockito.when(IOUtils.toString()).thenThrow(
new IOException("fake IOException"));
FileLoader fileLoader = new FileLoader();
Whitebox.invokeMethod(fileLoader,
"readAndStoreTermsOfUseForBrand", new Brand(...));
// If IllegalStateException is not thrown then this test case fails (see "expected" above)
}

Code below is completely untested
To cause the IOException use:
FileInputStream in = this.getClass().getClassLoader().getResourceAsStream(resourceName);
in.mark(0);
//read some data
in.reset(); //IOException
To test the IOException case use:
void test
{
boolean success = false;
try
{
//code to force ioException
}
catch(IOException ioex)
{
success = true;
}
assertTrue(success);
}
In JUnit4
#Test(expected=IOException.class)
void test
{
//code to force ioException
}
Other JUnit
void test
{
try
{
//code to force IOException
fail("If this gets hit IO did not occur, fail test");
}
catch(IOException ioex)
{
//success!
}
}