I am working on augmenting a test framework that uses Maven. Due to the nature of the code being tested, it is necessary to run tests in VMs. We are currently doing this in a sort of hack-ish way by running shell scripts that SSH into the VMs via Vagrant and run a list of tests. However, this list of tests has to be updated every time someone adds a new integration test to our tests. Ideally, we'd like to automatically gather the relevant tests that are flagged as Component / Integration tests with #Category JUnit flags in our Java code, and then run these tests within the VMs. It seems like Failsafe has no parameters to run the integration tests outside of the local machine. Is there any way to do this using existing Maven plugins?
Ideally, the flow of things would be as follows:
Discover all component / integration tests using Failsafe.
Pass the list of these tests into a VM
Run the tests on that VM, preferably with vagrant.
The existing plugin for Vagrant in Maven shows how to run a VM during integration tests, but it doesn't make clear how to actually run the integration tests on the VMs within Maven: http://nicoulaj.github.io/vagrant-maven-plugin/examples/running-a-vm-during-integration-tests.html . The plugin hasn't been updated since 2013 either, which isn't very promising.
Related
My question is about the directory location for a docker based acceptance test.
My project is a Spring Boot based command line application which extracts data from a table and builds a spreadsheet. It has unit tests and a JUnit based acceptance test. The JUnit runner for the acceptance test is a standard JUnit runner, not a Spring based runner.
Finally, I have an acceptance test structure which tests the Docker components against a dedicated DB2 instance created specifically for each test. At this point, there's a docker-compose file that:
Launches a DB2 container instance exclusively and solely for this test.
Launches a Flyway migration container to load test data.
Launches a container that does the above mentioned Spring Boot command line application.
After the close of the docker-compose, a comparison is done against the generated spreadsheet and an expected file. If they're byte for byte equivalent, the test is considered passed.
Given the acceptance test above is heavily docker laden and a few steps removed from the Java side, is it still appropriate to put this test under /src/test/acceptance?
There are many approaches to this. In general maven has two plugins: surefire and failsafe. They are very similar in terms of configurations, however surefire is mainly aimed for running unit tests, and failsafe is for integration tests.
So, first off you probably want to configure acceptance tests with a failsafe plugin. You will:
Run them during a different building phase (at least way after the unit tests run)
If your build was broken and some unit tests do not succeed you won't even want to run the acceptance/failsafe tests - it might save some build time.
You will get different reports for integration and unit tests (technically these plugins create different report folders surefire-reports and failsafe-reports)
Now to physically separate the tests you can:
Merely rely on the naming convention. These plugins look for tests named differently, say: SampleTest.class will be run with surefire while SampleTestIT.class will be run with the filesafe plugin. Of course its all customizable at the level of plugibs configurations in the pom file.
Usually unit tests are required to be put to the same package as a real class conceptually. For example: if you have a class Foo in com.myorg.Foo.java, so you place it in src/main/java/com/myorg/Foo.java and the corresponding unit test for it will be in src/main/java/com/myorg/FooTest.java. For integration tests is not usually the case so you can simply create a folder it or something and use run them with different plugins automatically, again because you'll name the tests differently.
Another approach is to separate the folders within the same module, it was already described above. So technically you maintain src/test/java and src/test/resources and next to it you will have something like src/it/java and src/it/resources. Probably you'll still want to use both surefire and failsafe plugin as I've described above. You'll still run both types of the tests in the same maven lifecycle.
The most "radical" approach is to separate the acceptance tests to different maven module. This will give you the ability to run the module with acceptance tests separately in a different build step. This might be handy in CI tool for example. Of course you can achieve a similar effect with properties or with maven profiles.
As you can refer in https://maven.apache.org/guides/introduction/introduction-to-the-standard-directory-layout.html
It is optimal to have separate directories for test. I would suggest put all tests under test and have separate folder, test/acceptance/docker-test or something, but overall it is up to you.
Separate folder does help to run and de-couple different tests.
Could SonarQube, Jacoco, or any other tool automatically run tests whenever I push to my repository, providing me with information if tests passed and code coverage information without running it on the local machine?
I found several post that explains how to run integration tests wising maven and docker. They basically explains hoy to use/setup fabric8 maven-docker-plugin and maven failsafe plugin.
I'm wondering if is possible to use fabric8, for example, with surefire. My specific need is: I need to run my tests (JUnit tests) but first I need/want to start a docker image with MySQL running on it.
I'm not tied to fabric8. If there is another way to start the docker image before start my tests and stop/kill it once tests run finish, that will help me a lot.
Thanks in advance.
Maxi
Yes you can definitely do that. The idea is to use a maven docker plugin such as fabric8 docker plugin and start a container before the test phases and stop it after the test phase.
But note that technically tests that connect to databases are not unit tests, they should be integration tests.
There are many tutorials online to do that such as INTEGRATION TESTING WITH DOCKER AND MAVEN
You can adapt this for unit tests by just changing the phases when the docker plugin runs.
You can change <phase>pre-integration-test</phase> to a phase that starts before the maven test phase such as <phase>generate-test-resources</phase> and stop the container once the test ends for example:
<phase>prepare-package</phase>.
Note that there are no nice phase names as there are for integration test, as it is not ideal to start external services when running unit tests. But anyway it works.
For a complete reference on maven phases check Introduction to the Build Lifecycle
My development environment is Netbeans 7.4 and Maven 3.0.5 (bundled with Netbeans). I'm using JUnit 4.11 to build unit (class names ending with Test) and integration tests (class names ending with IT). I'm running unit tests and excluding integration tests with the "-DskipITs" Maven option. I have a custom action which runs just the integration tests with the failsafe plugin. Both execute successfully. However, I only see the results in the "Test Results" window when running the unit tests. How can I get the integration tests to show in the "Test Results" window? With the integration tests, I'm only seeing the output in the console.
The maven-failsafe-plugin only executes in integration-test and verify (and of course the help) goal while the maven-surefire-plugin runs during test goal.
NetBeans Test Results window only shows the tests that were executed using the 'test' goal.
My solution for this situation is to categorize my integration tests into
testintegration - just lightweight ITs, I run them with surefire (to see them in Test Results window)
testheavy - those that will require me to bootstrap something I run with the fail-safe plugin
I hope you have the option of doing something close to that.
we run our junit tests in Eclipse seems fine, but when we move it to hudson or jenkins, the tests seems hangs there never returns. its definitely the same code, I just wandering whats the difference between running tests in Eclipse and Hudson.
In eclipse you usually just run one test class or one test method. A "fresh" Java Virtual machine is started and stopped for this test.
I suppose that Jenkins runs all tests in one Java Virtual machine. This can make a huge difference.
Try to run all test outside eclipse and outside hudson at once (should be simple if you have a maven project). What is the result? Maybe you can configure your test run to fork the Java VM on each test (this is possible with maven by configuring the surefire plugin).
My guess would be to look at the memory limit. Try to increase the memory limit of the maven process started from Jenkins.
My other guess would be to look whether the Maven build from Jenkins is allowed to execute submodules parallel. If yes, it might cause some race conditions that are hard to find.
Are the tests executed in the same way in Eclipse and in Hudson? My guess is that the tests are executed one by one (or class by class) in Eclipse, and by a build script such as Maven or Ant in Hudson? What happens if you execute the test suite from a terminal on your local machine?