I want to check some external log files after each test, if there were ary errors during execution. Throwing an exception in an AfterMethod doesn't work, because it is handled differently by TestNG: it will just fail the configuration method and not the preceding test.
My approach would be like this:
#AfterMethod(alwaysRun = true)
protected void tearDown(ITestResult result) {
if (thereWasAProblemDuringTestExecution()) {
result.setStatus(ITestResult.FAILURE);
result.setThrowable(getSomeThrowableSomebodyStoredAnywhere());
}
// doing other cleanUp-tasks
}
But still, my Eclipse TestNG plugin says the test passed.
Is it possible (and how) to fail a test (and not only a configuration method) in a configuration method?
Try setCurrentTestResult() method from class org.testng.Reporter:
result.setStatus(ITestResult.FAILURE);
Reporter.setCurrentTestResult(result);
This might be too late but I can help anyone else who is looking for this.
You can do this using the ITestContext
For some reason you cant really update a result which is already generated. The workaround is to add the same result, remove it, update it then add it again. The only way I have found to remove a result is to do the add and remove sequence.
def testContext = Reporter.currentTestResult.testContext
testContext.passedTests.addResult(result,Reporter.currentTestResult.method)
testContext.passedTests.getAllMethods().remove(Reporter.currentTestResult.method)
result.setStatus(ITestResult.FAILURE)
testContext.failedTests.addResult(result,Reporter.currentTestResult.method)
I am using Groovy.
Related
Usually when we run testcases using TestNG, assertion error stops further execution after that point. But, sometimes it would be better if we could run the whole script. Manually blocking/disabling those assertions become tedious process. Hence, if there were some ways to programmatically enable/disable assertions other than manual it would be of great help indeed.
Does TestNG support this? If not, can anyone help me please?
As Julien mentioned above you are better off making a custom softAssert of your own. I don't know, I could be horribly wrong but the standard softAssert that comes with testNG didn't give me the behaviour that I was after.
I suppose the most common reason that your tests are failing is an ElementNotFound or TimeOutException. So in your waitForElement method you can capture these exceptions (or any exception for that matter) and print a warning msg on the console (or don't print anything or even take a screenshot if you might, like a warning error but not a show-stopper error). Something like the below:
public boolean waitForElement(String elementName, int timeOut) {
try{
elementPresent=wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath(findXpath(elementName)))).isDisplayed());
}
catch(org.openqa.selenium.TimeOutException e1){e1.printStackTrace();elementPresent=false;takeScreenshot();}
}
return elementPresent;
}
Hope that helps!
Using QAF Validation you can fulfill your requirements, for selenium it provides inbuilt verification methods, if it fails then testcase also continues to execution.
As suggested , SoftAssert can be used which does not halt execution even if assertion fails.
Also enable/disable assertions is possible by flagging tests as enabled=false or enabled=true. This is in turn runs all tests [and thereby assertions] except those are marked as enabled=false
Example. Assertion in example won't be executed as test is disabled.
#Test (enabled=false)
public void verifySearchReport {
soft.assertEquals("About*", "About123Soft","FAILED ASSERT");
soft.assertAll();
}
Assertion in example will be executed as test is enabled. Further execution of tests won't be halted [Even if assertion is failing] as SoftAssert is used.
#Test (enabled=true)
public void verifySearchReport {
soft.assertEquals("About*", "About123Soft","FAILED ASSERT");
soft.assertAll();
}
//Further #Test here
I am doing some queries in a data provider. If the data queries do not return data I can use, rather than fail the test I would like to skip it. So I
throw new SkipException( "Could not find adequate data" ) but this is failing the test rather than skipping it.
Some research shows that SkipException has a method isSkip() which will skip if true and fail if false. I dumped it before throwing the exception and it showed true, but the test is still failing.
Am I doing something wrong or is there a better way to skip? (yes I know you can put it in the #test() but I don't know how to do that after the test is running.
If you throw a SkipException yes the test will fail. It's just like any other exception. If you want to skip a test you can do the following
Do #Test(enabled=false)
Implement iAnnotationTransformer and then do a annotation.setEnabled(false);
This will be a listener and you have to include that in your test or testng.xml or maven
You are not supposed to use SkipException in a data provider. Just send the result of the next query.
I have Spring boot application. I use Junit+Mockito to unit test it. All the test cases were written using Java. I recently made a decision to write test cases using Groovy though the application code will remain in Java.
I encountered a weird scenario while testing expected exceptions.
Scenario 1: Testing Expected exception using Junit + Groovy (without shouldFail):
#Test(expected = NoResultException.class)
void testFetchAllNoResultsReturned() throws Exception {
List<Name> namesLocal = null;
when(Service.fetchAllNames(id)).thenThrow(
new NoResultException(""))
namesLocal = (service.fetchAllNames(id)
assert(namesLocal==null)
verify(service, times(1)).fetchAllNames(id)
}
As per the above test case, service.fetchAllNames call should throw a NoResultException. This aspect of testing seems to work well. However, the assert and verify after that are not called. As soon as the exception is encountered, the method execution stops. However, my earlier test case written in Java worked perfectly well. This issue happened only after I switched to Groovy.
After doing some Google search I found there is a method called shouldFail provided by GroovyTestCase class which can be used for this scenario as per this link. And it did resolve my issue.
Scenario 2: Testing Expected exception using Junit + Groovy (with shouldFail ):
#Test
void testFetchAllNoResultsReturned() throws Exception {
List<Name> namesLocal = null;
when(Service.fetchAllNames(id)).thenThrow(
new NoResultException(""))
shouldFail(NoResultException.class) {
namesLocal = (Service.fetchAllNames(id)
}
assert(namesLocal==null)
verify(Service, times(1)).fetchAllNames(id)
}
My doubt is, is this how it is supposed to work or am I missing something. If this is how it is supposed to work, is there any reason behind Groovy doing it this way? I tried to look for reasons on the internet but I couldn't get many leads.
However, the assert and verify after that are not called. As soon as the exception is encountered, the method execution stops. However, my earlier test case written in Java worked perfectly well.
Given this code in java:
#Test(expected = NoResultException.class)
void testFetchAllNoResultsReturned() throws Exception {
List<Name> namesLocal = null;
when(Service.fetchAllNames(id)).thenThrow(
new NoResultException(""))
namesLocal = (service.fetchAllNames(id)
....
}
Irrespective of what you have after service.fetchAllNames(id), the call will throw an Exception and the test case ends there. Since you have an expected exception defined, the test case will pass. So the assert and verify after this line of code are never executed in java.
I am not familiar with groovy but from the documentation it looks like your second example using shouldFail is the correct way to test for exceptions in groovy. The shouldFail does not terminate the program - so its similar to putting your method call in a try catch in java
Background
I am working with a Selenium/Junit test environment and I want to implement a class to perform "soft asserts": meaning that I want it to record whether or not the assert passed, but not actually fail the test case until I explicitly tell it to validate the Asserts. This way I can check multiple fields on a page an record all of the ones which do not match.
Current Code
My "verify" methods appear as such (similar ones exist for assertTrue/assertFalse):
public static void verifyEquals(Object expected, Object actual) {
try {
assertEquals(expected, actual);
} catch (Throwable e) {
verificationFailuresList.add(e);
}
}
Once all the fields have been verified, I call the following method:
public static void checkAllPassed() {
if (!verificationFailuresList.isEmpty()) {
for (Throwable failureThrowable : verificationFailuresList) {
log.error("Verification failure:" + failureThrowable.getMessage(), failureThrowable);
// assertTrue(false);
}
}
}
Question
At the moment, I am currently just using assertTrue(false) as a way to quickly fail the test case; however, this clutters the log with a nonsense failure and pushes the real problem further up. Is there a cleaner way to purposefully fail a JUnit testcase? If not, is there a better solution to implement soft asserts? I know of an article which has a very well done implementation, but to my knowledge JUnit has no equivalent to the IInvokedMethodListener class
In case you want to fail a JUnit test on purpose you should use org.junit.Assert.fail()
Other option is to switch to TestNG framework which already has a SoftAssert class in it's latest version.
You can use JUnit's ErrorCollector rule.
I want to implement exception checking (like in JUnit 4) using JUnit 3. For example, I would like to be able to write tests like this:
public void testMyExceptionThrown() throws Exception {
shouldThrow(MyException.class);
doSomethingThatMightThrowMyException();
}
This should succeed if and only if a MyException is thrown.
There is the ExceptionTestCase class in JUnit, but but I want something that each test* method can decide to use or not use. What is the best way to achieve this?
Would the solution:
public void testMyExceptionThrown() throws Exception {
try {
doSomethingThatMightThrowMyException();
fail("Expected Exception MyException");
} catch(MyException e) {
// do nothing, it's OK
}
}
be suitable for what you're thinking of?
Also have a look at this thread, where someone created a Proxy-solution for JUnit3 which seems to be another possibility to solve your problem.
There is no need to implement your own solution because there is already one that can be used with JUnit3 (and any other testing framework): catch-exception.
The simplest approach is to use the Execute Around idiom to abstract away the try-catch that you would usually write.
More sophisticated is to note that TestCase is just a Test. I forget the details, but we can override the execution of the test (which the framework initially calls through run(TestResult) specified in Test). In that override we can place the try-catch, as per Execute Around. The testXxx method should call a set method to install the expected exception type.