I am working on a project with several spring-boot-based integrationtests that take ~5 Minutes to run in total. Upon trying to reduce this runtime i noticed that one very small and simple test takes ~20 seconds. When run "alone" it takes less than a second, which is what I would have expected.
The other tests are all integrationtests with annotations like:
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.MOCK)
#MockBean(value = SomeBean.class)
#MockMvcTest
#ContextConfiguration(initializers = someInitializer.class)
But the test in question has no annotations like this, requires no application context and should go through smoothly and quickly according to my understanding. It looks something like this:
public class SomeUnitTest{
#Test
public void testObjectOfClass(){
SomeClass someClass= new SomeClass ("1", "2", "3", "4");
SomeClass.assertThat(someClass).hasId("1-2-3-4");
}
}
Can anyone explain why the tests behave like this. Even if I can't change it I would really apreciate to understand what's going on.
I've seen this before where the JUnit runner is timing it incorrectly.
I had a test that would run sub-second on its own but slow in a full test run. It would run immediately before a test with a bunch of setup with plugins & #BeforeAll steps. After digging into it, I found that the timer was not switching to this next test class until after the setup was completed.
It's possible you're seeing the same thing if the other tests have lots of pre-test setup.
EDIT: #BeforeAll definitely gets timed poorly.
public class TestA {
#Test
public void test() {
}
}
public class TestB {
#BeforeAll
public static void before() throws InterruptedException {
Thread.sleep(10000);
}
#Test
public void test() {
}
}
public class TestC {
#BeforeAll
public static void before() throws InterruptedException {
Thread.sleep(10000);
}
#Test
public void test() {
}
}
I have found so far 2 ways to setup JUnit test timeout. Either using:
#Test(timeout=XXX)
Or using something like:
#ClassRule
public static Timeout timeoutRule = new Timeout(XXX, TimeUnit.MILLISECONDS);
In my case, I have a Test Runner as a main class to run all my test suites, so I can execute the tests as an executable jar.
I'd like this runner to setup the timeouts dinamically using reflection.
Is it possible to do?
You could add the timeout feature to a custom test runner like so:
public class TimeoutTestRunner extends BlockJUnit4ClassRunner {
public TimeoutTestRunner(Class<?> clazz) throws InitializationError {
super(clazz);
}
#Override
protected Statement withPotentialTimeout(FrameworkMethod method, Object test, Statement next) {
return FailOnTimeout.builder()
// you'll probably want to configure/inject this value rather than hardcode it ...
.withTimeout(1, TimeUnit.MILLISECONDS)
.build(next);
}
}
Using this test runner the tests in the following test case ...
#RunWith(TimeoutTestRunner.class)
public class YourTest {
#Test
public void willTimeout() throws InterruptedException {
Thread.sleep(50);
assertTrue(true);
}
#Test
public void willNotTimeout() throws InterruptedException {
assertTrue(true);
}
}
... will behave as follows:
willTimeout: will fail with a TestTimedOutException
willNotTimeout: will pass
Although you will need your tests to be run via this runner you will be able to control their timeout setting from one place and provide custom timeout derivation strategies such as if test name matches <some regex> then timeout is x else ....
At the moment I have the following code:
public class AlterServiceSelT
{
#Before
public void setupAndActivate()
{
System.out.println("setupAndActivate");
}
#Test
public void suspendService()
{
System.out.println("suspendService");
}
#Test
public void reActivateService()
{
System.out.println("reActivateService");
}
#After
public void terminateService()
{
System.out.println("terminateService");
}
}
and when ran I get the following in my console:
setupAndActivate
reActivateService
terminateService
setupAndActivate
suspendService
terminateService
The problem is that the full code for setupAndActivate() takes 15 minutes and its output is needed to run for the tests. Ideally I would like the console to output:
setupAndActivate
reActivateService
suspendService
terminateService
How could this be done?
Try looking at #BeforeClass in stead of using the #Before.
One of the downsides of BeforeClass is that it has to be defined on a static method, so all your fields that you set up have to be static.
Upside is that your setup is only done once for all the tests in your class.
Are you sure that a setup of 15 minutes is optimum for your application?
I have a java package which contains all my test case classes. Each class contains a lot of test cases. Each class is loaded and run one by one by JUnit. However each of the classes contains common configuration code which is run again and again and initialised everytime each of the classes are run.
These initializations take a lot of time.
Is there some way to load these configuration changes first and then run the test case so that I do not need to load them everytime.
JUnit4 has #BeforeClass annotation.
Just do something like this:
public class TestClass {
private static SomeConnection connection;
#BeforeClass
public static void setUp() {
//do common setup
connection = new SomeConnection();
}
#Test
public void testSomething() { }
#Test
public void testSomethingElse() { }
#AfterClass
public static void tearDown() {
//do teardown operations
connection.close();
}
}
Method marked with #BeforeClass will run only once. Just make sure you use JUnit4.
Update:
Also note, that it should be static, and as #ChristopheRoussy mentioned, you can use #AfterClass to destroy your common setup.
You can create a static method that is run before the tests within the class are ran.
#BeforeClass
public static void ranOnlyOnce() {
/*...*/
}
Moreover, if you want to run this once before all of your tests, then you should group your tests in a suite, and put this method in this class, and use JUnit to run the suite instead of the tests.
#RunWith(Suite.class)
#SuiteClasses(value = { Test1.class, ... , Testn.class })
public class AllTests {
#BeforeClass
public static void beforeAllTests() {
/*...*/
}
}
Note that annotations can be used only in Junit 4+.
I would recommend grouping your tests into a testsuite and doing the initialization from the testsuite. There is a good discussion of the possibilities here http://www.xoriant.com/blog/software-testing-and-qa/using-customized-junit-testsuite-for-testing.html
There are a number of possibilities. The first and easiest is to use #BeforeClass and #AfterClass as has been suggested by Matyas and jFrenetic. This is the easiest way to do it if none of the test classes share setup code.
If the classes share the same setup and teardown, look at using the TestRule, specifically the ExternalResource, which allows you to run code before and after each class. The difference between this and #BeforeClass and #AfterClass is that it uses the same class, and therefore the code can be shared more easily. This can be done for example:
#RunWith(Suite.class)
#SuiteClasses({A.class, B.class, C.class})
public class UsesExternalResource {
public static Server myServer= new Server();
#ClassRule
public static ExternalResource resource= new ExternalResource() {
#Override
protected void before() throws Throwable {
myServer.connect();
};
#Override
protected void after() {
myServer.disconnect();
};
};
}
This works in a test class as well, so you could have different, but shared setups for different classes.
I'm trying to preform setup and teardown for a set of integration tests, using jUnit 4.4 to execute the tests. The teardown needs to be run reliably. I'm having other problems with TestNG, so I'm looking to port back to jUnit. What hooks are available for execution before any tests are run and after all tests have completed?
Note: we're using maven 2 for our build. I've tried using maven's pre- & post-integration-test phases, but, if a test fails, maven stops and doesn't run post-integration-test, which is no help.
Yes, it is possible to reliably run set up and tear down methods before and after any tests in a test suite. Let me demonstrate in code:
package com.test;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
#RunWith(Suite.class)
#SuiteClasses({Test1.class, Test2.class})
public class TestSuite {
#BeforeClass
public static void setUp() {
System.out.println("setting up");
}
#AfterClass
public static void tearDown() {
System.out.println("tearing down");
}
}
So your Test1 class would look something like:
package com.test;
import org.junit.Test;
public class Test1 {
#Test
public void test1() {
System.out.println("test1");
}
}
...and you can imagine that Test2 looks similar. If you ran TestSuite, you would get:
setting up
test1
test2
tearing down
So you can see that the set up/tear down only run before and after all tests, respectively.
The catch: this only works if you're running the test suite, and not running Test1 and Test2 as individual JUnit tests. You mentioned you're using maven, and the maven surefire plugin likes to run tests individually, and not part of a suite. In this case, I would recommend creating a superclass that each test class extends. The superclass then contains the annotated #BeforeClass and #AfterClass methods. Although not quite as clean as the above method, I think it will work for you.
As for the problem with failed tests, you can set maven.test.error.ignore so that the build continues on failed tests. This is not recommended as a continuing practice, but it should get you functioning until all of your tests pass. For more detail, see the maven surefire documentation.
A colleague of mine suggested the following: you can use a custom RunListener and implement the testRunFinished() method: http://junit.sourceforge.net/javadoc/org/junit/runner/notification/RunListener.html#testRunFinished(org.junit.runner.Result)
To register the RunListener just configure the surefire plugin as follows:
http://maven.apache.org/surefire/maven-surefire-plugin/examples/junit.html section "Using custom listeners and reporters"
This configuration should also be picked by the failsafe plugin.
This solution is great because you don't have to specify Suites, lookup test classes or any of this stuff - it lets Maven to do its magic, waiting for all tests to finish.
You can use the #ClassRule annotation in JUnit 4.9+ as I described in an answer another question.
Using annotations, you can do something like this:
import org.junit.*;
import static org.junit.Assert.*;
import java.util.*;
class SomethingUnitTest {
#BeforeClass
public static void runBeforeClass()
{
}
#AfterClass
public static void runAfterClass()
{
}
#Before
public void setUp()
{
}
#After
public void tearDown()
{
}
#Test
public void testSomethingOrOther()
{
}
}
Here, we
upgraded to JUnit 4.5,
wrote annotations to tag each test class or method which needed a working service,
wrote handlers for each annotation which contained static methods to implement the setup and teardown of the service,
extended the usual Runner to locate the annotations on tests, adding the static handler methods into the test execution chain at the appropriate points.
As for "Note: we're using maven 2 for our build. I've tried using maven's pre- & post-integration-test phases, but, if a test fails, maven stops and doesn't run post-integration-test, which is no help."
you can try the failsafe-plugin instead, I think it has the facility to ensure cleanup occurs regardless of setup or intermediate stage status
Provided that all your tests may extend a "technical" class and are in the same package, you can do a little trick :
public class AbstractTest {
private static int nbTests = listClassesIn(<package>).size();
private static int curTest = 0;
#BeforeClass
public static void incCurTest() { curTest++; }
#AfterClass
public static void closeTestSuite() {
if (curTest == nbTests) { /*cleaning*/ }
}
}
public class Test1 extends AbstractTest {
#Test
public void check() {}
}
public class Test2 extends AbstractTest {
#Test
public void check() {}
}
Be aware that this solution has a lot of drawbacks :
must execute all tests of the package
must subclass a "techincal" class
you can not use #BeforeClass and #AfterClass inside subclasses
if you execute only one test in the package, cleaning is not done
...
For information: listClassesIn() => How do you find all subclasses of a given class in Java?
As far as I know there is no mechanism for doing this in JUnit, however you could try subclassing Suite and overriding the run() method with a version that does provide hooks.
Since maven-surefire-plugin does not run Suite class first but treats suite and test classes same, so we can configure plugin as below to enable only suite classes and disable all the tests. Suite will run all the tests.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.5</version>
<configuration>
<includes>
<include>**/*Suite.java</include>
</includes>
<excludes>
<exclude>**/*Test.java</exclude>
<exclude>**/*Tests.java</exclude>
</excludes>
</configuration>
</plugin>
The only way I think then to get the functionality you want would be to do something like
import junit.framework.Test;
import junit.framework.TestResult;
import junit.framework.TestSuite;
public class AllTests {
public static Test suite() {
TestSuite suite = new TestSuite("TestEverything");
//$JUnit-BEGIN$
suite.addTestSuite(TestOne.class);
suite.addTestSuite(TestTwo.class);
suite.addTestSuite(TestThree.class);
//$JUnit-END$
}
public static void main(String[] args)
{
AllTests test = new AllTests();
Test testCase = test.suite();
TestResult result = new TestResult();
setUp();
testCase.run(result);
tearDown();
}
public void setUp() {}
public void tearDown() {}
}
I use something like this in eclipse, so I'm not sure how portable it is outside of that environment
If you don't want to create a suite and have to list all your test classes you can use reflection to find the number of test classes dynamically and count down in a base class #AfterClass to do the tearDown only once:
public class BaseTestClass
{
private static int testClassToRun = 0;
// Counting the classes to run so that we can do the tear down only once
static {
try {
Field field = ClassLoader.class.getDeclaredField("classes");
field.setAccessible(true);
#SuppressWarnings({ "unchecked", "rawtypes" })
Vector<Class> classes = (Vector<Class>) field.get(BlockJUnit4ClassRunner.class.getClassLoader());
for (Class<?> clazz : classes) {
if (clazz.getName().endsWith("Test")) {
testClassToRun++;
}
}
} catch (Exception ignore) {
}
}
// Setup that needs to be done only once
static {
// one time set up
}
#AfterClass
public static void baseTearDown() throws Exception
{
if (--testClassToRun == 0) {
// one time clean up
}
}
}
If you prefer to use #BeforeClass instead of the static blocks, you can also use a boolean flag to do the reflection count and test setup only once at the first call. Hope this helps someone, it took me an afternoon to figure out a better way than enumerating all classes in a suite.
Now all you need to do is extend this class for all your test classes. We already had a base class to provide some common stuff for all our tests so this was the best solution for us.
Inspiration comes from this SO answer https://stackoverflow.com/a/37488620/5930242
If you don't want to extend this class everywhere, this last SO answer might do what you want.