I am trying to get coverage per test feature working with Sonarqube 7.
I am using jacoco-maven-plugin and my tests are running with JUnit 5.
I successfully managed to get global Coverage on Sonarqube dashboard, but I would like to go deeper by being able to see which tests covered which lines of my classes.
I've tried with the given configuration here but without success : I get the following message in the logs 'No information about coverage per test'. I see that this can be obtained by adding a listener org.sonar.java.jacoco.JUnitListener, but it is a JUnit 4 listener, so I guess it is not working because of that.
How can I manage to make the Coverage per test feature work with JUnit 5 ?
I have some problem and solved change version of surefire-plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.2</version>
</plugin>
Related
I have been working with cucumber/Java and JUnit4 (CucumberOptions) for years without trouble running the tests in both IntelliJ and maven command line.
Recently, i have been trying to make the move to JUnit5 and i was able to have all tests running in IntelliJ (only, unfortunately)
My POC project has the following structure:
junit5
-Features (folder with feature files)
-resources (folder with files used in tests)
-src
--test
---java
----stepdefs
-----SetupEnvHook
-----StepDefs
----AllTest (testrunner wip)
----JU4Test (testrunner JUnit4)
----JU5Test (testrunner Junit5)
---resources (test resources)
-junit5.iml
-pom.xml
The JU5Test.java file :
import org.junit.platform.suite.api.ConfigurationParameter;
import org.junit.platform.suite.api.SelectDirectories;
import org.junit.platform.suite.api.Suite;
import stepdefs.SetupEnvHook;
import io.cucumber.java.Before;
import static io.cucumber.core.options.Constants.*;
#Suite
#SelectDirectories("Features")
//#ConfigurationParameter(key = PARALLEL_EXECUTION_ENABLED_PROPERTY_NAME, value = "true")
#ConfigurationParameter(key = PLUGIN_PUBLISH_ENABLED_PROPERTY_NAME, value = "false")
#ConfigurationParameter(key = PLUGIN_PUBLISH_QUIET_PROPERTY_NAME, value = "true")
#ConfigurationParameter(key = PLUGIN_PROPERTY_NAME, value = "json:target/cucumber-reports/cucumber.json")
#ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "stepdefs, my.external.steps.stepdefinition")
public class JU5Test {
#Before
public static void beforeSuite() {
SetupEnvHook.setEnvironment("QA");
}
}
The beforeSuite() method is also used in the JU4Test.
When i set a breakpoint in SetupEnvHook.setEnvironment("QA"); it is completely ignored due to the fact that the Before Annotation is not working, while another breakpoint inside the same
#io.cucumber.java.BeforeAll(order = 9999)
Annotation in SetupEnvHook class is triggered correctly.
My pom file is as follows :
<dependencies>
<dependency>
<groupId>my.external</groupId>
<artifactId>steps</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.9.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
</plugins>
</build>
Please ignore the my external dependancy. This dependancy is related to the stepdefinitions in the test runner file glue property.
I know that group and version values are also missing but these are all fed from the same dependancy in red so as to have more control on the versions everyone uses.
This is all done in Java 8 using
org.apache.maven.plugins:maven-compiler-plugin:3.10.1
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M7
io.cucumber:cucumber-java:7.8.1
io.cucumber:cucumber-junit:7.8.1
io.cucumber:cucumber-junit-platform-engine:7.8.1
org.junit.jupiter:junit-jupiter-api:5.9.1
org.junit.jupiter:junit-jupiter-engine:5.9.1
org.junit.platform:junit-platform-suite-api:1.9.1
org.junit.platform:junit-platform-suite-engine:1.9.1
I already tried using different Annotations not only from io.cucumber.java but also from org.junit (which is basically JUnit4) and org.junit.jupiter.api with no success obviously.
Running through maven command line ends up with :
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project junit5: No tests were executed!
It does not however state that 'no tests were found', had that issue initially and got it solved.
From looking at the error i suspect i may have something missing from the pom.xml surefire plugin but i cannot figure out what. (this pom is the same used to run the JU4Test without issues)
Anyone else have any thoughts on what i can try next? or better yet, the solution for this xD
Edit: remove images
It does not however state that 'no tests were found', had that issue initially and got it solved.
From looking at the error i suspect i may have something missing from the pom.xml surefire plugin but i cannot figure out what. (this pom is the same used to run the JU4Test without issues)
From your description it is impossible to say what is wrong with your project. Your list of depencies includes dependencies not included in your POM.
You may want to consider starting your project from scratch. You can use the https://github.com/cucumber/cucumber-java-skeleton for that.
When i set a breakpoint in SetupEnvHook.setEnvironment("QA"); it is completely ignored due to the fact that the Before Annotation is not working
The reason the #Before annotation is ignored is because you are using a Cucumber annotation on a class that is not part of the glue path.
Though I suspect you are trying to find a mapping for JUnit 4s #BeforeClass. Currently there is not such thing in JUnit 5s Suite Engine. If you need it, you should consider making a pull requests.
Alternatively you could create a package with a single class for each environment and use Cucumbers #BeforeAll hooks to set the environment. Then for each #Suite you configure a different glue path to include those hooks.
Though I think it would be even better to read the target environment from an environment variable and have it default to something sane. You can then use different CI jobs for each environment.
I've been able to configure the project to run test cases in parallel using a TestNG runner; however, there are a handful of scenarios that are not very thread safe. If these test cases were to run in parallel, they'd interfere with each other. Now there are a couple of ways I could make these scenarios thread safe, but I was wondering if there was a way to specify these Cucumber scenarios not to run in parallel.
Is there a specific tag I could configure to tag scenarios not to run in parallel? Specify certain feature files not to run in parallel? I believe I might have come across something like that for JUnit 5, but does this exist with TestNG?
Unlike JUnit 5, TestNG does not provide such fine-grained controls. At best you can create multiple runner classes with a different selection of features and different configurations for parallel/serial execution.
Another thing I just realized was I could make the number of threads an argument; with a default of 1 thread.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven.surefire.version}</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<properties>
<property>
<name>dataproviderthreadcount</name>
<value>1</value>
</property>
</properties>
<includes>
<include>api.automation.tests.runners.TestNGRunnerTest</include>
</includes>
</configuration>
</plugin>
I have one Jenkins build that has all the tests tagged with the same name to run in parallel with however many threads needed.
mvn clean test -Dcucumber.filter.tags="${CucumberFilterTag} and not #opentofix" -Dspring.profiles.active=${Environment} -Ddataproviderthreadcount=4
And in the other build, I have a very similar command (with a different tag) just without the added arguement; to use the default of just one thread. This will run all those tagged tests one at a time.
mvn clean test -Dcucumber.filter.tags="${CucumberFilterTag} and not #opentofix" -Dspring.profiles.active=${Environment}
Either way, the first answer still works, but just thought to add this possibility.
While upgrading to JUnit 5 (version 5.5.2), I have made a strange discovery with the suite functionality: my suites can find and run tests that end with the word "Test" but fail to find tests that don't end in "Test" (in my case, they end in "Base").
In JUnit 4, we used the #Suite.SuiteClasses() annotation to find these tests, but the JUnit 5 #SelectClasses annotation seems to miss these test classes entirely. Even using #IncludeClassNamePatterns({"^Com.*Base.*?$"}) fails to detect the tests, which I found strange (the tests I want to run start with "Com"). After this, I tried the #Tag() annotation, which didn't work either.
I assumed this was because Maven Surefire (version 2.22.2) only detects test classes that start with Test, or end with Test, Tests, or TestCase. So, I tried to include my Base test case:
<includes>
<include>**/*Base.java</include>
<include>**/Test*.java</include>
<include>**/*Test.java</include>
<include>**/*Tests.java</include>
<include>**/*TestCase.java</include>
</includes>
Maven was able to run these Base tests when I built this project, but the test suites still failed to find them.
My code will look similar to the following:
#RunWith(JUnitPlatform.class)
#SelectClasses({
Com_TestOne_Base.class,
Com_TestTwo_Base.class,
Com_TestThree_Base.class,
Com_TestFour_Base.class,
Com_TestFive_Base.class,
Com_TestSix_Base.class,
})
public class Com_Base_Suite {
}
The result of running this suite is a success, but no tests actually run. All these tests have been updated to JUnit 5 and run successfully on their own.
The problem you're running into results from mixing up JUnit 4 and 5. Maven Surefire is able to run JUnit 5 (aka JUnit platform) tests out of the box - given you have the right dependencies in your pom. See e.g. https://github.com/junit-team/junit5-samples/tree/master/junit5-jupiter-starter-maven for a minimal pom.xml.
JUnitPlatform, SelectClasses et al. allow you to run JUnit platform tests through JUnit 4. You'd probably only want to do that if your build tool or IDE do not support the JUnit platform themselves. JUnit 5 does currently not have any explicit support for test suites similar to JUnit 4's #Suite annotation.
I recommend you get rid of Com_Base_Suite alltogether and go with a naming convention that can be configured through Maven's <includes> section.
Latest Jacoco plugin (still in snapshot version, 0.7.10-SNAPSHOT), has a nice new feature to filter out the Lombok generated code (https://github.com/jacoco/jacoco/wiki/FilteringOptions).
All we need to do is add a lombok.config file at the root of the repository with:
lombok.addLombokGeneratedAnnotation=true
When I generate the Jacoco report internally, I see the difference.
However, when my regular quality job executes and publishes the result to Sonar, I get different (ie worse) results.
How come I don't have the same results in my local report and in Sonar? Is there any workaround?
As mentionned here : https://github.com/jacoco/jacoco/pull/513#issuecomment-293176354
filtering is performed at a time of report generation (creation of html, xml, etc), not at a time of collection of execution information (creation of exec file). So that tools that read execution data directly instead of reading of xml (which is a kind of mistake on their side to rely on purely internal intermediate format, but what's done is done) and create their own report (such as SonarQube, Jenkins, etc) will need to update their dependency on JaCoCo once it will be released in order to get filtering for reports. We will notify explicitly downstream projects (in particular all mentioned above) about this when our release will be done. So once again - please be patient. Thank you for your understanding.
I didn't find a way for Sonar to read the end report instead of the exec file, so I guess we need to be patient and wait for the official 0.7.10 jacoco plugin release and then an update on Sonar side !
------ UPDATE May 9th 2018
New versions have been released, and I can confirm it works for me.
Using :
Sonar 6.7
SonarJava plugin 5.1.1.13214
jacoco maven plugin 0.8.1
lombok.addLombokGeneratedAnnotation=true in lombok.config
I now get much better coverage results reported to Sonar, as Lombok generated code is now ignored. It really helps identifying what the "real" uncovered areas are, and whether it's risky or not.
First you have to check your lombok version is at least 1.16.14
pom.xml:
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<scope>provided</scope>
<version>1.16.14</version>
</dependency>
Then you have to check that your Jacoco version is at least 0.8.0
pom.xml:
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.0</version>
<!-- // -->
</plugin>
Then you have to add a lombok.config file in the src folder of your project (not in the resources folder)
lombok.config:
# tells Lombok that this is the root directory and that it shouldn’t search parent directories for more configuration files
config.stopBubbling = true
# tells Lombok to add #lombok.Generated annotation to all generated methods
lombok.addLombokGeneratedAnnotation = true
I have a JUnit4 test suite and I want to execute it under the JUnit Plug-in Test run configuration.
It passes successfully when running through the JUnit Test configuration, but for plug-in conf it fails in different ways.
For example if I use JUnit4TestAdapter it fails with ClassCastException, and if I trying to run it only through the #RunWith annotation it wrotes "No methods found" error. For both implementations I use JUnit4 Test Runner setting inside run configuration.
I use
Eclipse Neon.1a Release (4.6.1)
Jdk 1.8
linking JUnit 4.1 lib to the plugin.
For first case it seems that Eclipse proceed to use the JUnit3 version when executing the suite.
Here it is:
#RunWith(Suite.class)
#SuiteClasses({ DndTest.class })
public class JSTestSuite {
public static Test suite() {
return new JUnit4TestAdapter(JSTestSuite.class);
}
}
And exception is:
java.lang.ClassCastException: junit.framework.JUnit4TestAdapter cannot be cast to junit.framework.Test
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.getTest(RemoteTestRunner.java:403)
While starting the test I have a strange log message in console:
!ENTRY org.eclipse.update.configurator 2017-05-04 17:58:57.279
!MESSAGE Could not install bundle ../../platform/eclipse/plugins/org.eclipse.jdt.junit4.runtime_1.1.600.v20160505-0715.jar No match is available for the required execution environment: J2SE-1.5
I see this lib is on the place, but I can't understand why it failing to be loaded. For JUnit3 Test Runner setting junit3 lib is loaded ok.
There are some bugs related to such issues (like this) but it is really hard to understand what can I do in this case.
For second case I just try to execute simple JUnit4 case without using the JUnit4TestAdapter, but it can't find any methods.
Reloading of eclipse and renaming of the methods didn't help.
What can I do in this case?
I found that there are couple of issues. First I consider one by one to solve the issue.
Issue#1: No match is available for the required execution environment: J2SE-1.5
Issue#2: java.lang.ClassCastException: junit.framework.JUnit4TestAdapter cannot be cast to
junit.framework.Test
Solution for Issue#1:
First solution:
Right-click on your project
Click Properties
Click the "Java Compiler" option on the left menu
Under JDK compliance section on the right, change it to "1.8"
Second solution:
If you use maven in your project, then you can change pom.xml file as below:
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
Resource Link: Maven "build path specifies execution environment J2SE-1.5", even though I changed it to 1.7
Solution for Issue#2:
JUnit runner has a dependency on JUnit 3.8. It won't be used but without it, the whole platform can't be initialized.
So you need 2 versions like the following
Check you have below plugins
org.eclipse.xtext.xbase.junit
org.junit (3.8.2)
org.junit (4.8.2)
How to check for JUnit?
In eclipse, please check in 2 sections.
Check your Project Properties->Java Build Path->Libraries (tab)
Check you Project's Run Configurations->JUnit->Classpath (tab)
How to fix?
To fix the error make sure you have org.junit 3.8 in the target platform!
All credit goes to A Paul.
Resource Link:
https://stackoverflow.com/a/21334162/2293534
Aaron Digulla has commented like below:
Despite the fact that org.eclipse.xtext.junit4 imports org.junit
4.5.0, org.eclipse.xtext.junit (note the missing "4" at the end) seems to have a dependency to JUnit 3.8. After adding the old, outdated
JUnit bundle, the plugin tests started.