I would like to know, is there any way to set up TestNG to handle unexpected exceptions like errors and not like failures?
I tried to throw RuntimeException in my test and it was considered as failure, not like an error.
TestNG documentation talks only about success and failure states -
http://testng.org/doc/documentation-main.html#success-failure.
I would like to have TestNG behaviour similar to JUnit in first question on address
http://www.querycat.com/question/d1c9a200f18e6829cb06dda8eda8ad61
Thanks for your help.
Edit: Ignore what I had before. Unfortunately, I'm fairly certain the answer to your question is no, TestNG does not have this feature. The entirety of the TestNG documentation is here, so if a feature isn't listed on that page it probably doesn't exist. Remember that although TestNG is inspired by JUnit, it should not be considered a super-set of JUnit's features. The only thing I can think of if you want your test suite to catastrophically fail on an exception is to make it call System.exit(1).
You are using the wrong exception.
JUnit is distinguishing between success, failure (AssertionException thrown by assertXXX) and error (all other not expected exception).
If you know that a method will throw an exception you can catch it with #Test(expected=MyExpectedException).
Related
I recently migrated a Unit Testing suite to Junit 5.8.2 and Mockito 4.5.1 + Mockito Inline to allow static mocking. Powermock was removed.
2000+ tests were migrated and they all run successfully when ran inside the IDE (IntelliJ). Both with the IDEA and Gradle runner.
However, when Jenkins attempts to run them there are over 900 failed tests. Some of the exceptions thrown.
org.mockito.exceptions.misusing.MissingMethodInvocationException:
when() requires an argument which has to be 'a method call on a mock'.
For example:
when(mock.getArticles()).thenReturn(articles);
org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
Boolean cannot be returned by someMethod()
someMethod() should return Date`
I understand what causes these errors as I've seen them multiple times during the migration so this is not a duplicate asking for the solution.(Unless there's something different with the Jenkins environment) The code that throws such exceptions should not be throwing them. And it does not in the IDE. It's thrown exclusively in Jenkins.
An additional exception which I have never seen before is thrown as well.
org.mockito.exceptions.misusing.UnfinishedMockingSessionException:
Unfinished mocking session detected.
Previous MockitoSession was not concluded with 'finishMocking()'.
For examples of correct usage see javadoc for MockitoSession class.
Most of the exceptions are of this type.
However, the MockitoSession inteface is not used anywhere in the test suite. All mocks are initialized with #ExtendWith(MockitoExtension.class)
I have no idea what could be causing this.
Jenkins is running the same versions of Java/Junit/Mockito/Spring as the code in the IDE.
It seems clear to me that the different environments are causing the issue. However, what could be the difference and how would I go about finding it?
I attempted to reproduce the results locally but was unable to. Any ideas towards that are also welcome.
Issue: I want to change TestNG assertions to AssertJ and SoftAssertions from AssertJ, but I do not know what exactly I need to override from SoftAssertions to execute custom code after an AssertionError is collected. Specifically, I would like to get screenshots for each collected error. I use Selenium for Automation and need to add screenshots for reporting.
I tried to override each fail method from AbstractSoftAssertions but it did not help.
Actual: I got screenshots only in the #After method.
Expected: I need to get screenshots for each soft assertion error.
The current setup is with Java, TestNG, Selenium, and Allure report.
There are two options:
Call setAfterAssertionErrorCollected on your SoftAssertions instance, providing your own AfterAssertionErrorCollected implementation (can be a lambda).
Extend AbstractSoftAssertions and overrride onAssertionErrorCollected​.
Both ways allow you to provide your own logic to be executed with each assertion error.
You can find examples about both solutions in the AssertJ documentation.
I'm assuming that the reason the assert statement throws an AssertionError to stop your program is because an assumption about your code is wrong. The problem is, how am I supposed to debug my code by using debugging tools when the assert statement kills my java process?
EDIT: My question is different because the answer to the supposedly duplicate question doesn't answer mine.
EDIT:
If you look for a way to throw an exception and not throw an exception at the same time, there is no solution for this.
In modern IDEs you can, during debug:
tell the IDE to block on any AssertionError without setting any breakpoint
change the code of a method during a test when blocked, such as commenting out an assert
tell the IDE to resume running a method again How to step back in Eclipse debugger?
Also assert can be disabled for the purpose of debugging. That's one reason to use an assert instead of an if-block.
See:
How do I disable Java assertions for a junit test in the code
Java assert statements are selectively enabled/disabled, based on config flags (eg, -ea). I read online that with JUnit assertions (eg, Assert.assertTrue), you don't need to worry about managing the assertion-config yourself. What exactly does this mean? Either:
1) JUnit assertions (mixed together with Prod code) are never disabled. They behave just like Validate statements, and always get run.
or
2) The JUnit test framework automatically enables all JUnit assertions when running JUnit tests. Whenever the application/code is run outside of JUnit, the JUnit assert methods do nothing at all.
or
3) JUnit assertions are always enabled/disabled by default, but there is a custom config/flag that can be used to disable/enable them
Which of the above is true?
--
Edit: Changed the wording a little bit to clear up any ambiguity.
This question seems to have opened up a can of worms with regards to coding-guidelines on when to use/not to use asserts. I'll post here the answer to my immediate question regarding whether JUnit Assert statements get disabled. Further down, you will find a discussion on the philosophical aspects of using assertions.
Summary: JUnit assert statements are never disabled, even when running in production with the disable-assertions flag.
Experiment: Created a small main function which throws an error in the very first line. The error was implemented in various different ways. I then ran the application from the command-line like so: mvn clean package; java $JAVA_OPTS -cp ....
Induced error by throwing IllegalArgument exception. Application failed with IllegalArgument exception.
Induced error by adding assert false and ran with assertions enabled (-ea). Application failed.
Induced error by adding assert false and ran with assertions disabled (-da). Application succeeded.
Induced error by adding Assert.assertTrue(false); and ran with assertions enabled (-ea). Application failed with AssertionError.
Induced error by adding Assert.assertTrue(false); and ran with assertions disabled (-da). Application failed with AssertionError.
Did a quick google search to see if there's any way to disable JUnit.Asserts. Didn't find anything. If anyone knows of a way to do so, please let me know, because I believe it's an important distinction.
Conclusion: assert keywords can be enabled/disabled using the -ea commandline flag. JUnit.Asserts cannot be enabled/disabled. If you want to add assertions to your production code, then this becomes a very important distinction. If you want the ability to disable them, use the assert keyword. If you want something that you know will always be enabled, then consider JUnit.Assert, or some other similar framework that throws AssertionErrors.
Regarding the philosophy behind assert usage:
It's good to add assertions of some form to your production code. See: https://docs.oracle.com/javase/8/docs/technotes/guides/language/assert.html#usage
Putting Assertions Into Your Code. There are many situations where it is good to use assertions, including:
Internal Invariants
Control-Flow Invariants
Preconditions, Postconditions, and Class Invariants
JUnit Asserts cannot be treated as a universal replacement for java assert keywords. The latter gives you the power to disable them, and the former doesn't. Hence, there is a design choice to be made in choosing which of the 2 to use.
Exceptions and Assertions serve very different purposes and should not be used interchangeably. The following discussion clarifies this further: When to use an assertion and when to use an exception
Related StackOverflow discussions:
assert vs. JUnit Assertions
I think you're confused between JUnit assertions and Java's built in assertions.
JUnit assertions are methods that throw an AssertionError exception and should only be used in your test code. JUnit code should not be used in production, only when running your JUnit tests during/between builds. The method names are named like assertEquals( expected, actual) and assertNull(Object) etc. You can't disable these assertions since they are method calls. Also your test code would be pretty useless if you could disable all the JUnit assert methodcalls.
Java's built in assertion is a boolean expression preceded by the assert keyword. If the expression returns false, then it too throws an AssertionError. This kind of assertion can be disabled in production to help speed up your code or turned on to help track down bugs. You can read more about the built in Java assert in the Java Programming Guide
When an Exception occurs while running a JUnit test (an unexpected Exception outside the scope of the test), no stacktrace is printed. Only the Exception message is. For example, I'm adding unit tests using JUnit to an existing Spring MVC project. I've got an empty test like this:
#RunWith(SpringJUnit4ClassRunner.class)
#WebAppConfiguration
#ContextConfiguration("file:src/main/webapp/WEB-INF/my-servlet.xml")
public class UserAccessTest {
#Autowired
private WebApplicationContext webApplicationContext;
#Test
public void test01() {}
}
But it fails with the message:
Tests in error:
test01(com.myproject.UserAccessTest): Failed to load ApplicationContext
However, it doesn't print the entire stack trace, including root causes, etc. Another example of this happening is a NullPointerException occurring when I've forgotten to mock certain objects when I'm using Mockito. In such a case, it doesn't even give a message, and it's not until after I add a try-catch block in the test that I find out it's a NullPointerException, and then it isn't until I add code to manually print all StackTraceElements in the catch that I find out where it's at!
Is it possible to have the entire stacktrace printed when exceptions occur during the test?
I'm using Maven and IntelliJ IDEA. I'm not sure though whether this is even something I have to configure in either one of these. I've found this somewhat related answer but 1) that involves Ant on a build server and 2) I have no clue where I would have to put that config if it would apply to me since that's not mentioned in the answer.
Please note: I'm not asking for help solving the issue in the example. I just want to know how I can get JUnit to provide more information about such problems so that I can debug it further myself.
There are some surefile plugin settings (eg maven-surefire-plugin useFile parameter set to false) that control this, refer to this