Develop Reference

Merging test coverage and test results from multiple Jenkins jobs

Background:
We have a rather large REST API written in Java that we're testing with combination of unit and functional tests. There are many variations that are required when testing it, particularly at the functional level. While the unit tests live in-tree, the functional tests are in a separate code repository.
We are currently using Jacoco for test coverage and TestNG for running our unittests, though I believe answers to my question should be applicable to other tool combinations.
We have several different jobs in Jenkins that are triggered by a check-in to the primary project. These include jobs that run tools like Coverity as well as several different functional test jobs. These jobs are triggered by the initial commit, which is not considered to be "green", until all of the downstream jobs complete successfully.
The Problem:
How do we take coverage reports (like the Jacoco binaries and the TestNG xml files) and combine them to show total code coverage over all of our tests? Specifically, we know how to combine them if they are present in the same job/directory, but these files are spread across multiple Jenkins jobs which may be running at different times.
In my experience, the most commonly accepted way of handling this is to use the
Promoted Builds Plugin to trigger all jobs, then pull their artifacts when it completes down to the triggering job. I don't feel like this scales very well, however, when you have more than one or two jobs that you're attempting to roll-up. This is especially true when you may have more than one variation on the master project (old releases, etc).
I understand it is possible to fingerprint files in Jenkins such that we know that -.jar is the same version used in Jobs A, B, and C. Does a plugin exist that can retrieve all files matching a pattern based on the existence of a different fingerprinted file?
One alternative solution (which would probably be run from an ant/groovy script), is to push test data to a directory somewhere that is tied to a git commit hash, and retrieve all such data in a roll-up job based on the git commit hash of the base project.
Are there any simple ways to do this? Has anyone figured out any better other ways to solve this problem?
Thanks,
Michael

Faced similar issue, tweaked jacoco maven plugin config to merge jacoco results. Basically merged jacoco-unit.exec and jacoco-it.exec into one binary and published that merged result on Jenkins via pipeline step.
pom.xml:
<plugin>
<inherited>false</inherited>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.agent.version}</version>
<executions>
<execution>
<id>merge-results</id>
<phase>post-integration-test</phase>
<goals>
<goal>merge</goal>
</goals>
<configuration>
<fileSets>
<fileSet>
<directory>${project.parent.build.directory}</directory>
<includes>
<include>jacoco-*.exec</include>
</includes>
</fileSet>
</fileSets>
<destFile>${project.parent.build.directory}/jacoco.exec</destFile>
</configuration>
</execution>
</executions>
</plugin>
Jenkinsfile:
echo 'Publish Jacoco trend'
step($class: 'JacocoPublisher',
execPattern: '**/jacoco.exec',
classPattern: '**/classes',
sourcePattern: '**/src/main/java',
)
However you still have to fetch jacoco binaries from several Jenkins builds by another build step or specify their locations explicitly.

Managing TestNg suite from Jenkins parameterized build

I'm running my tests in Jenkins and Maven and have different test suites in several TestNG.xml files.
Now I manage them directly in pom in this way:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire.version}</version>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>Testing_Fuzzy_Logic.xml</suiteXmlFile>
<suiteXmlFile>Testing_ACL.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
But I'd like to be able to select a specific test suite when start a Jenkins build. And I'd like all the suites to be included to the build as default if I did not select a specific one. So my question is:
What is the proper way to write all my TestNG.xml files in Jenkins parameters and send them to pom file to ask Maven use them when running tests?
Something like this:
Jenkins variable:
Set it to pom file somehow:

The issue has been solved by an example above. At first there was an error in file name and it confused me. I fixed it and my example works okay now.
So one of solutions is: a string TEST_NG_XML parameter which is added to parameterized Jenkins build. All test suites are set there and separated by comma by default. When needed, any of them can be removed (See examples above).
But if there are any other solutions, I'll be delighted to see them.

On jenkins, you can pass top level maven goals in following format
test -DargLine="-DTest_NG_XML=src/test/resources/testng.xml"
In this way , you are passing the value of your Test_NG_XML variable while running the maven test phase.In general execute following maven goal on jenkins.
-DargLine="-Dparameter=value"

test -DsuiteXmlFile=src/test/resources/testng1.xml,src/test/resources/testng2.xml
How can one be sure the precedence of execution of test suites passed this way?
I was facing the same issue, I have split it into separate Jenkins builds with each passing separate suite, to be absolutely sure certain suite runs before the other.
I would like to simplify it, possibly passing this way as indicated above... but did you try the order of execution? Thanks

Maven + Clover + Jenkins - How to get Coverage report and non-instrumented artifact in one command

I am trying to run maven clover plugin to generate report as well as generate NON-instrumented artifact.
<plugin>
<groupId>com.atlassian.maven.plugins</groupId>
<artifactId>maven-clover2-plugin</artifactId>
<version>3.1.3</version>
<configuration>
<generatePdf>true</generatePdf>
<generateHtml>true</generateHtml>
<licenseLocation>clover.license</licenseLocation>
<!-- the contextFilters element has to be specified within the reporting section and will not work if you specify it in the build section. -->
<!-- contextFilters>try,static,catch</contextFilters -->
</configuration>
</plugin>
mvn clean clover2:instrument clover2:clover install
If I run above according to clover doc instument goal will run in separate lifecycle and will not affect default buildcycle. So It does but problem is I want to skip test during default build lifecycle.
I tried following but it skipped test for both lifecycle.
mvn clean clover2:instrument clover2:clover install -DskipTests
If above works then I can simple set it up on jenkins withou creating mulitple jobs for multiple maven commands.

It is probably not the best idea to do everything in single cryptic maven command (in the same way it is not the best idea to put all your code in a procedure). Why not splitting the command into several steps or even jobs, which will trigger one another? Moreover from CI point of view different kind of jobs ask different priority to fail fast. I do understand that it is not exactly an answer.

Eclemma always reporting 0% of code coverage

I have a Maven test project for my application.
The JUnit tests run fine, and the code coverage test run too.
But the report always shows 0% of code coverage.
What should i do?

According to the official site, Eclemma is a code coverage plugin for Eclipse, based on JaCoCo library.
As you want to use the same code coverage engine outside eclipse, you should include the plugin Jacoco inside the Maven configuration (pom) of your project, as the following (this code was copied from the Agile Engineering blog):
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.6.0.201210061924</version>
<executions>
<execution>
<id>jacoco-initialize</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>jacoco-site</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
To run the tests just type the following on the command line tool:
mvn clean test
p.s.: you also could use other code coverage plugins like Cobertura or Emma.

Just in case you forgot to do these:
Are you annotating your tests using #Test?
Are you running the class as a JUnit test case or from the coverage button?
I'm not sure what the cause of the problem is, cause it always worked for me. Have you installed it from eclipse itself? Try to uninstall it, and reinstall from eclipse. Here's how to do it just in case:
In Eclipse, Click "Help" > "Install new Software"
Click "Add", and type the following:
Name: EclEmma (or any name you want)
Path: http://update.eclemma.org/
Select EclEmma, and install

Now I realized that you just want to get a report using the tool inside Eclipse...
How is the code coverage in the Eclipse Dialog? Did you tried to use the mouse right click on this dialog to export session (report), or inside File -> Export?

It's a known issue for many years and unfortunately there's no official solution yet for it.
You can see it here, here and here
One not-so-honey solution might be to try using eCobertura (or downgrading eclemma from 2.x to 1.x)

If you are using eclemma, you need to add jacoco dependency. if jacoco has been added and still, you are facing this issue, refer the eclemma faq: "Why does a class show as not covered although it has been executed?"
it says,
First make sure execution data has been collected. For this select the Sessions link on the top right corner of the HTML report and check whether the class in question is listed. If it is listed but not linked the class at execution time is a different class file. Make sure you're using the exact same class file at runtime as for report generation. Note that some tools (e.g. EJB containers, mocking frameworks ) might modify your class files at runtime.
So, Mockito / PowerMockito can cause this problem. In my case, I have added the class in #PrepareForTest(). I was shown that the test case was executed fine without errors but Jacoco did't improve the code coverage in its report.
Finally, removing the class from #PrepareForTest() annotation improved the code coverge. check if you have added it or not and remove it from annotation if added.

I just came across this issue and it was caused by an incorrectly configured classpath. When the unit tests were executed, they were executing against a compiled jar (actual source compiled outside of eclipse) and not my actual source code. After removing the jar from my classpath, the unit tests correctly hit my package source.

I was able to resolve the issue on mine by calling a instance of the class at the top of the test cases.
i.e.
public hotelOccupancy hotel = new hotelOccupancy();
#Test
public void testName() {
// some test here
}
Once I did that all my coverage began working and the issues were resolved.

I'm using eclemma 2.3.2 and it's working perfectly on eclipse
I only need to add these dependencies in my pom.xml
<dependency>
<groupId>org.jboss.arquillian.extension</groupId>
<artifactId>arquillian-jacoco</artifactId>
<version>1.0.0.Alpha6</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jacoco</groupId>
<artifactId>org.jacoco.core</artifactId>
<version>0.7.1.201405082137</version>
<scope>test</scope>
</dependency>
Then I build the project, update maven projects configuration and run coverage plugin as expected

How to tell Maven2 to not test non-#Test-annotated classes in test source folder?

Just got a strange error. Working on project using JUnit and Maven 3.0.3 I've created in my test/src/java folder one test class - ClassifierUtilTest.java, with #Test-annotated methods and stuff, and two utility classes, just for the use in the testing env (one with few static methods for bypassing private visibility scopes and one mock for tested interface).
It works good under Maven 3.0.3 (mvn test), and in Eclipse 3.7 (run as / JUnit test), but when someone else tried to 'mvn test' it with Maven 2.2.1 it failed. Apparently it tried to treat those util classes as test classes and failed due to 'no #Test-annotated methods' and 'more than one constructor'.
It's not JUnit fault (at least it shouldn't be, maven dependency is the same, junit:junit:4.9), so it seems to be strictly maven or maven-surefire-plugin fault. I was wondering if there is some widely-known workaround for Maven 2.2.1 for this problem?

maven-surefire-plugin by default runs all the classes the have Test prefix or suffix (like yours ClassifierUtilTest) and TestCase suffix. Just change the name to ClassifierTestUtil and you'll be fine.
You can also exclude certain files/directories in pom.xml, see Inclusions and Exclusions of Tests:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.11</version>
<configuration>
<excludes>
<exclude>**/ClassifierUtilTest.java</exclude>
</excludes>
</configuration>
</plugin>