is there any reason for a jacoco coverage report to produce different results when running it in a CI environment?
I have the following situation after migrating to Java 17. when I run ./gradlew clean build the report is generated and the jacocoTestCoverageVerification doesnt fail.
But when the same command is executed in a github runner I have a very different result, the coverage report is also created with different coverage values and some classes just fail the coverage verification.
Unfortunately, I cannot provide code snippets but I already tried this:
Run the github runner on my machine, with the same jdk and it produces a different result if I just run the build in the command line :(
add jvmargs noverify for tests tasks
run the github action in a container instead of a self-hosted, same result
other devs have also built the project on different machines without error
run the same build command of the ci
So it seems that something in a github action env produces this behavior.
versions:
jacoco plugin: 0.8.7
gradle 7.3
java 17
The problem here was lombok config files. It turns out that the new version of lombok plugin compatible with java 17 and Gradle 7.3 does not generate the lombok.config files anymore.
In the previous version, these files were generated by io.freefair.lombok plugin, and we had them in .gitignore. Because of that, a local build worked just fine, since the files were still there.
I realized this by looking at the generated reports and I noticed that lombok classes were being analyzed only by the CI.
In other cases, a similar problem could be related to this:
https://www.eclemma.org/jacoco/trunk/doc/classids.html
Related
SonarQube: 8.2.0.32929
sonar-scanner: 3.0.3.778
jacoco: 0.8.4
jdk: 1.8
mvn: 3.6.3
What are you trying to achieve
I am trying to achieve code coverage by using sonar-scanner but I am getting code coverage 0 in sonarqube dashboard.
What have you tried so far to achieve this
I configured the multi-module java project using https://github.com/SonarSource/sonar-scanning-examples/tree/master/sonarqube-scanner-maven/maven-multimodule
and created sonar-project.properties file in base directory with below configuration
sonar.projectKey=org.sonarqube.sonarscanner-maven-aggregate
sonar.projectName=Sonar Scanner Maven Aggregate
sonar.projectVersion=1.0
sonar.language=java
sonar.java.binaries=.
If I use mvn sonar:sonar it works. but with sonar-scanner it is not working.
It works fine with sonarqube 7.8.
Any insight would be appreciated.
I fixed this issue.
while running sonar-scanner command, I added the xmlReportPaths as a define property like sonar-scanner -Dsonar.coverage.jacoco.xmlReportPaths=tests/target/site/jacoco-aggregate/jacoco.xml,../tests/target/site/jacoco-aggregate/jacoco.xml
If your projects are set up as multi-module, you will have to feed the coverage report from each module to the last module that will run as part of the build (probably integration tests modules)
https://community.sonarsource.com/t/coverage-test-data-importing-jacoco-coverage-report-in-xml-format/12151
https://community.sonarsource.com/t/in-sonarqube-8-2-code-coverage-is-always-showing-0/21666
https://docs.sonarqube.org/latest/analysis/coverage/
https://stackoverflow.com/a/15535970
Following the steps mentioned in this example project by Sonar team helped me
https://github.com/SonarSource/sonar-scanning-examples/tree/master/sonarqube-scanner-maven/maven-multimodule
As you already followed these steps, one thing I was missing was the aggregated report was not generated in the last module and I was doing it in one module before that which was our integration test module. But there was one more deployment module in our case
I am writing Cucumber Automated Test Cases in java, using REST Assured.
All the tests run as JUnit tests, & as Gradle tests, & they all pass successfully.
The next step I need to take is to generate a Cucumber Report based on these test results.
I've been told that I need to find a gradle plugin for cucumber.
So far, I've tried putting the following code (from here: https://plugins.gradle.org/plugin/com.github.spacialcircumstances.gradle-cucumber-reporting) into my build.gradle file:
plugins {
id "com.github.spacialcircumstances.gradle-cucumber-reporting" version "0.0.12"
}
But, when I try to refresh the gradle, I get this error:
Sychrnoize Gradle projects with workspace failed due to an error in the referenced Gradle build.
Is this the correct plugin to generate this cucumber report, and if so, why is it not being accepted by the gradle?
I have inherited a codebase whereby we have a maven project of component tests that use junit as the framework that runs the tests.
Everything runs OK, but the problem is that these tests must be compiled into a jar-with-dependencies so that they can be run on a standalone computer with very limited outwards connectivity (e.g does not have access to maven central).
I have managed to create a runnable jar with dependencies which I can run using junit.jar and the command line like so:
java -cp jar-with-dependencies.jar:junit.jar junit.jar org.junit.runner.JUnitCore com.testsuite.AllTests
This works but the problem is that I also need to output the junit results into XML, like maven's surefire plugin does.
My question is, can I run this jar-with-dependencies using maven such that it creates the junit xml reports?
I can successfully run the tests using exec-maven-plugin which essentially runs the previously stated command
I have a Scala project built with SBT. I use Jenkins to do my build, and include the following as a build target:
scoverage:test
This gets me very nice coverage reports for my Scala sources using the Cobertura plugin for Jenkins.
My project has Java files as well, however, and I get no coverage for them. If this were Maven, I'd know what to do. But being SBT, I'm a bit unclear on how I can get coverage for the Java sources and then ensure that Jenkins sees the output and displays it as well.
I do know that the Java tests run using "sbt test" so that's not an issue.
Is this even possible?
I had been introduced to concept of CI lately and was trying to work on jenkins CI. I was stuck up in one thing . How to trigger executable testng files in jenkins CI. For ex locally in our machines we just run testng.xml to execute couple of test cases. In the same way how can we trigger this xml file to run in jenkins CI ?
In most cases with jenkins you wouldn't use an executable. Normally you'd run the wrapper for the tests (Junit/Nunit etc.) which Jenkins is fully capable of running on it's own.
You can use this article to run TestNG tests using Maven:
Running TestNG tests using maven
After configuration is completed just add Invoke top-level Maven targets step to the Build Steps in Jenkins (Maven plugin should be installed). The target should be test in this case.
If you will face with any errors during configuration, try to google them.
If you are not using any build tool like maven or ant, you can invoke it from command line as we'll and specify your suite file. Make sure to set the correct class paths http://testng.org/doc/documentation-main.html#running-testng
You can put this as a build step in Jenkins.
Add a compilation step prior to this step. I haven't ever tried it - have always used ant or maven, but that is where I would start exploring.