I'm trying to cleanup my tests by always resetting to a known state before each test. In JUnit it seems that the best way to do this is to have a setup() method that sets the values for some fields. When running tests in parallel the field is always correct since each test is executed in a new instance of the test.
However in TestNG this doesn't seem to be the case. According to a post on their mailing list, setting fields in #BeforeMethod in a multithreaded testing doesn't guarantee their value.
As I need the classes I'm testing to be in a known state, is there a cleaner solution to this than using DataProvider or saying "Don't ever run tests in mulithreaded mode"?
There is only one difference between TestNG and JUnit in this specific area: JUnit will create a brand new instance of your test before each test method, TestNG will not.
What this means is that with TestNG, values stored in fields by test methods will be preserved between invocations, which is very useful if this object is complex and takes time to create. It also helps speed up test runs since you don't have to recreate this state from scratch every time.
If you want this state to be reset every time, simply put the initialization code in #BeforeMethod, like you do with JUnit (except it's called #Before).
As for multithreading, I don't understand why you are saying that there is no guarantee about that value, can you be more specific?
Related
The following test is one of several tests that fail when I run my tests in random order using this Maven command: mvn -Dsurefire.runOrder=random clean test
#Test
public void ShouldReturnCorrectAccountLoanSumForDebtRatioWhenRedemptionAmountIsNull(){
AccountVO account = mock(AccountVO.class);
CustomerGroupInformationVO group = mock(CustomerGroupInformationVO.class);
when(group.getCustomerIds()).thenReturn(Set.of("199406208123"));
when(account.getAccountOwners()).thenReturn(List.of((new AccountOwnerVO(null, "199406208123", null))));
when(account.getAmount()).thenReturn(BigDecimal.valueOf(500000));
when(account.getRedemptionAmount()).thenReturn(null);
assertEquals(BigDecimal.valueOf(500000), getAdjustedAccountLoanSumForDebtRatio(account, group, caseClientVO));
}
More specifically this is the line mentioned:
when(account.getAccountOwners()).thenReturn(List.of((new AccountOwnerVO(null, "199406208123", null))));
Any idea what is causing this and how I can fix it? When I run my tests normally using mvn clean install there are no issues at all. The reason I want it to work with a random order is that our build tool seems to use it and it can't build. Like I said it works fine locally.
Because Mockito works through side-effects stored in ThreadLocal variables, it is particularly subject to test pollution: If your tests fail when run in random order, it may be because some previous test left a mock in a state it wasn't expecting to be in. Also, Mockito stubbing relies on observing method calls in a specific order, which can cause odd exceptions if it can't observe the method calls (such as when they're final) or if you interact with a different mock while preparing a thenVerb argument.
One of your first lines of defense is to use validateMockitoUsage, which is meant to be run at the end of every test and confirms that no interaction with Mockito is left unfinished. You could put this in an #After method, but Mockito does so automatically if using MockitoJUnitRunner or MockitoRule. I'd recommend either of those latter options if possible, particularly MockitoRule.
This should help you confirm which test(s) are problematic. Once you're there, try these:
Double-check that you're not trying to mock final methods without Mockito's opt-in final support. If Mockito can't override your mock method, it won't be able to detect your stubbing calls in its expected order. If any of your methods are written in Kotlin, remember that unlike Java the methods are final unless declared open.
Your test doesn't seem to use Matchers, but if you do, make sure you use them for all arguments in a method if you use them for any argument in a method.
Be careful about calling real methods in the middle of stubbing. new AccountOwnerVO(...) shouldn't interact with mocks, but if it does, then Mockito might interpreted it as if your when call never got a thenReturn (when in reality it just hasn't gotten to it yet). Extracting your return value as a local variable is a reasonable step to try.
I have a question about parameterized tests in JUnit. I am running a test suite with all of my test classes, it is a requirement for my course to have a test suite to run all of my test classes, so I cannot modify that. The issue is that I have a bunch of Entry objects (let's just take this as an object with a unique id starting from 1 and incremented every time a new instance of it is created), and they are being pre-processed by JUnit. On compiling and running my program, I have 9 entries which are declared in the ParamTest class. Within another class (EntryTest) I have one Entry that I have created and it should have an ID of one. However, it has an ID of 10, meaning the 9 entries from the parameterized test class have been created before hand.
My question is, is there anyway to force the ParamTest class not to do any of the pre-processing before the EntryTest class is run or is this impossible. In the suite I have made sure to declare EntryTest before ParamTest. If it's impossible is there anyway I can get around this other than creating separate suites or running the tests separately? I was thinking a public static int to keep track of the ID from the pre-processed amounts but it sounds like an ugly solution.
I think your testing is going to get ugly, fast, unless you have a way of resetting your static class to a known state.
I would recommend you expose a package-private method that allows you to reset the ID value to something specific (e.g. 0).
Tests should be entirely independent of one another, even within the same test class.
I'm using Spring Test with TestNG to test our DAOs, and I wanted to run a specific text fixture script before certain methods, allowing the modifications to be rolled back after every method so that the tests are free to do anything with the fixture data.
Initially I thought that 'groups' would be fit for it, but I already realized they're not intended for that (see this question: TestNG BeforeMethod with groups ).
Is there any way to configure a #BeforeMethod method to run only before specific #Tests? The only ways I see are workarounds:
Define an ordinary setup method and call at the beginning of every #Test method;
Move the #BeforeMethod method to a new class (top level or inner class), along with all methods that depend on it.
Neither is ideal, I'd like to keep my tests naturally grouped and clean, not split due to lack of alternatives.
You could add a parameter your #BeforeMethod with the type 'java.lang.reflect.Method'. TestNG will then inject the reflection information for the current test method including the method name, which you could use for switching.
If you add another 'Object' parameter, you will also get the invocation parameters of the test method.
You'all find all on possible parameters for TestNG-annotated methods in chapter 5.18.1 of the TestNG documentation.
Tests are simply not designed to do this. Technically speaking, a single tests is supposed to handle being idempotent for itself meaning it sets up, tests, and takes down. That is a single test. However, a lot of tests sometimes have the same set-up and take down method, whereas other tests need one set-up before they all run. This is the purpose of the #Before type tags.
If you don't like set-up and tear-down inside your test, your more then welcome to architect your own system, but technically speaking, if certain methods require specific set-ups or tear-downs, then that really should be embodied IN the test, since it is a requirement for test to pass. It is ok to call a set-up method, but ultimately, it should be OBVIOUS that a test needs a specific set-up in order to pass. After all, if your using specific set-ups, aren’t you actually testing states rather than code?
Recently a new concept of Theories was added to JUnit (since v4.4).
In a nutshell, you can mark your test method with #Theory annotation (instead of #Test), make your test method parametrized and declare an array of parameters, marked with #DataPoints annotation somewhere in the same class.
JUnit will sequentially run your parametrized test method passing parameters retrieved from #DataPoints one after another. But only until the first such invocation fails (due to any reason).
The concept seems to be very similar to #DataProviders from TestNG, but when we use data providers, all the scenarios are run inspite of their execution results. And it's useful because you can see how many scenarious work/don't work and you can fix your program more effectively.
So, I wonder what's the reason not to execute #Theory-marked method for every #DataPoint? (It appears not so difficult to inherit from Theories runner and make a custom runner which will ignore failures but why don't we have such behaviour out of the box?)
UPD: I have created a fault-tolerant version of Theories runner and made it available for a public access: https://github.com/rgorodischer/fault-tolerant-theories
In order to compare it with the standard Theories runner run StandardTheoriesBehaviorDemo then FaultTolerantTheoriesBehaviorDemo which are placed under src/test/... folder.
Reporting multiple failures in a single test is generally a sign that
the test does too much, compared to what a unit test ought to do.
Usually this means either that the test is really a
functional/acceptance/customer test or, if it is a unit test, then it
is too big a unit test.
JUnit is designed to work best with a number of small tests. It
executes each test within a separate instance of the test class. It
reports failure on each test. Shared setup code is most natural when
sharing between tests. This is a design decision that permeates JUnit,
and when you decide to report multiple failures per test, you begin to
fight against JUnit. This is not recommended.
Long tests are a design smell and indicate the likelihood of a design
problem. Kent Beck is fond of saying in this case that "there is an
opportunity to learn something about your design." We would like to
see a pattern language develop around these problems, but it has not
yet been written down.
Source: http://junit.sourceforge.net/doc/faq/faq.htm#tests_12
To ignore assertion failures you can also use a JUnit error collector rule:
The ErrorCollector rule allows execution of a test to continue after
the first problem is found (for example, to collect all the incorrect
rows in a table, and report them all at once)
For example you can write a test like this.
public static class UsesErrorCollectorTwice {
#Rule
public ErrorCollector collector= new ErrorCollector();
#Test
public void example() {
String x = [..]
collector.checkThat(x, not(containsString("a")));
collector.checkThat(y, containsString("b"));
}
}
The error collector uses hamcrest Matchers. Depending on your preferences this is positive or not.
AFAIK, the idea is the same as with asserts, the first failure stops the test. This is the difference between Parameterized & Theories.
Parameterized takes a set of data points and runs a set of test methods with each of them. Theories does the same, but fails when the first assert fails.
Try looking at Parameterized. Maybe it provides what you want.
A Theory is wrong if a single test in it is wrong, according to the definition of a Theory. If your test cases don't follow this rule, it would be wrong to call them a "Theory".
I have a problematic situation with some quite advanced unit tests (using PowerMock for mocking and JUnit 4.5). Without going into too much detail, the first test case of a test class will always succeed, but any following test cases in the same test class fails. However, if I select to only run test case 5 out of 10, for example, it will pass. So all tests pass when being run individually. Is there any way to force JUnit to run one test case at a time? I call JUnit from an ant-script.
I am aware of the problem of dependant test cases, but I can't pinpoint why this is. There are no saved variables across the test cases, so nothing to do at #Before annotation. That's why I'm looking for an emergency solution like forcing JUnit to run tests individually.
I am aware of all the recommendations, but to finally answer your question here is a simple way to achieve what you want. Just put this code inside your test case:
Lock sequential = new ReentrantLock();
#Override
protected void setUp() throws Exception {
super.setUp();
sequential.lock();
}
#Override
protected void tearDown() throws Exception {
sequential.unlock();
super.tearDown();
}
With this, no test can start until the lock is acquired, and only one lock can be acquired at a time.
It seems that your test cases are dependent, that is: the execution of case-X affects the execution of case-Y. Such a testing system should be avoided (for instance: there's no guarantee on the order at which JUnit will run your cases).
You should refactor your cases to make them independent of each other. Many times the use of #Before and #After methods can help you untangle such dependencies.
Your problem is not that JUnit runs all the tests at once, you problem is that you don't see why a test fails. Solutions:
Add more asserts to the tests to make sure that every variable actually contains what you think
Download an IDE from the Internet and use the built-in debugger to look at the various variables
Dump the state of your objects just before the point where the test fails.
Use the "message" part of the asserts to output more information why it fails (see below)
Disable all but a handful of tests (in JUnit 3: replace all strings "void test" with "void dtest" in your source; in JUnit 4: Replace "#Test" with "//D#TEST").
Example:
assertEquals(list.toString(), 5, list.size());
Congratulations. You have found a bug. ;-)
If the tests "shouldn't" effect each other, then you may have uncovered a situation where your code can enter a broken state. Try adding asserts and logging to figure out where the code goes wrong. You may even need to run the tests in a debugger and check your code's internal values after the first test.
Excuse me if I dont answer your question directly, but isn't your problem exactly what TestCase.setUp() and TestCase.tearDown() are supposed to solve? These are methods that the JUnit framework will always call before and after each test case, and are typically used to ensure you begin each test case in the same state.
See also the JavaDoc for TestCase.
You should check your whole codebase that there are no static variables which refer to mutable state. Ideally the program should have no static mutable state (or at least they should be documented like I did here). Also you should be very careful about cleaning up what you write, if the tests write to the file system or database. Otherwise running the tests may leak some side-effects, which makes it hard to make the tests independent and repeatable.
Maven and Ant contain a "forkmode" parameter for running JUnit tests, which specifies whether each test class gets its own JVM or all tests are run in the same JVM. But they do not have an option for running each test method in its own JVM.
I am aware of the problem of dependant
test cases, but I can't pinpoint why
this is. There are no saved variables
across the test cases, so nothing to
do at #Before annotation. That's why
I'm looking for an emergency solution
like forcing JUnit to run tests
individually.
The #Before statement is harmless, because it is called for every test case. The #BeforeClass is dangerous, because it has to be static.
It sounds to me that perhaps it isn't that you are not setting up or tearing down your tests properly (although additional setup/teardown may be part of the solution), but that perhaps you have shared state in your code that you are not aware of. If an early test is setting a static / singleton / shared variable that you are unaware of, the later tests will fail if they are not expecting this. Even with Mocks this is very possible. You need to find this cause. I agree with the other answers in that your tests have exposed a problem that should not be solved by trying to run the tests differently.
Your description shows me, that your unit tests depend each other. That is strongly not recommended in unit tests.
Unit test must be independent and isolated. You have to be able to execute them alone, all of them (in which order, it does not matter).
I know, that does not help you. The problem will be in your #BeforeClass or #Before statements. There will be dependencies. So refactor them and try to isolate the problem.
Probably your mocks are created in your #BeforeClass. Consider to put it into the #Before statement. So there's no instance that last longer than a test case.