SonarQube coverage missing some lines covered by Jacoco report - java

When I run unit tests with Jacoco agent, there is some discrepancy between my local Jacoco report and the coverage on SonarQube. This seems to only be affecting files that contain nested classes. The report generated locally has coverage information for the outer class and all inner classes, but the coverage data on SonarQube only includes the inner classes.
For example, Foo.java contains outer class, Foo, and inner classes, Bar and Baz.
My local report shows instruction coverage of 26% for class Foo, 46% for class Foo.Bar, and 0% for class Foo.Baz; the overall instruction coverage for Foo.java is 30%.
The SonarQube coverage page gives line coverage of 15% for Foo.java. I understand that line coverage does not equal instruction coverage, but I would expect the numbers to be closer. Upon further inspection, I noticed that in the file-based coverage view of Foo.java on SonarQube, all lines in the outer class Foo are marked "Not covered by unit tests" and the only lines that are marked covered are the ones in Foo.Bar that I expected. This difference makes up the approximately 15% gap between the Jacoco report and SonarQube. I don't see any exceptions in the local scanner logs or the server analysis logs.
I am running with JaCoCo 0.7.7.201606060606, Java version 1.8.0_73, and sonar-scanner 2.8 locally. The server is running Java version 1.8.0_66-b17, SonarQube version 5.6.3, and SonarQube Java plugin version 4.2.1.6971.
I would appreciate any suggestions and would be happy to provide more details if that would be helpful.

Comparison of "instructions" with anything else is like comparison of apples and oranges - they don't represent the same thing. Single line of code usually contains many bytecode instructions. And so it is wrong to expect "instruction coverage" to be close to "line coverage", for example: if in total you have 100 instructions in 10 lines and cover 1 line with 20 instructions, then missed instructions 80%, but missed lines 90%.
See http://www.eclemma.org/jacoco/trunk/doc/counters.html about counters that JaCoCo provides. And http://docs.sonarqube.org/display/SONAR/Metric+Definitions about what SonarQube shows. Instructions coverage is presented only in JaCoCo.
And not clear from your question if you see difference between covered lines in Foo.java shown by SonarQube and shown by JaCoCo. If so, then please provide screenshot.

It turns out that the class files that used to run the unit tests (which are also used to generate my local report) differ from the ones used by sonar-scanner. This is because after compilation, unit tests, and local report generation, bnd is run over the class files and rewrites the class files of #Component classes. Because sonar-scanner is run after bnd, it sees different class files. It seems like my problem wasn't inner vs. outer classes, but rather OSGi components vs. non-components; class Foo is an OSGi component, and the inner classes are not.
When I run the scanner over the same classes files used by the Jacoco agent, the line coverage of Foo.java reported by SonarQube is 27% (instead of 15%), and the file-based coverage view matches my local report.

Related

Cobertura showing wrong line coverage

I see a strange cobertura report. It says that line number 89 (see screenshot)
siteMapItems = getItemFromPage(navParentPage, 0);
is executed once, but the method called on this line (getItemFromPage) shows no line coverage. I have also added logs in the method and i also see logs being printed during execution of unit test case.
What could be the reason of this strange report? Its not making sense to me.
Occasionally this kind of desynchronization issues arise from mismatch of java files and class files. Check if cobertura is scanning right class file.

Running Clover coverage reporter inserts new column to CSV test and breaks build

I am trying to integrate Clover with a Java project at work. Clover has been added to the classpath, the plugin has been added to the POM and the license has been set up.
When I run mvn clean clover:setup test clover:aggregate clover:clover Clover starts up correctly. However, tests that generate csv's fail because the CSV output is not what is expected. This is what gets generated for one of the tests:
<ID,NAME,[__CLR4_1_2_TEST_NAME_SNIFFER,"DESCRIPTION,Long","DESCRIPTION,Short",country
001,Fred,com_atlassian_clover.TestNameSniffer$1#e628ccb,Mr. Fred Flinstone,Fred Flinstone,US
002,Bob,com_atlassian_clover.TestNameSniffer$1#e628ccb,Mr. Bob the builder,Bob the builder,UK
003,Tintin,com_atlassian_clover.TestNameSniffer$1#e628ccb,Mr. Tin Tin,Tin Tin,FR
004,,com_atlassian_clover.TestNameSniffer$1#e628ccb,,Rob the Bob,
005,,com_atlassian_clover.TestNameSniffer$1#e628ccb],,Tom the Bob,
>
This is what is expected:
<ID,NAME,["DESCRIPTION,Long","DESCRIPTION,Short",country
001,Fred,Mr. Fred Flinstone,Fred Flinstone,US
002,Bob,Mr. Bob the builder,Bob the builder,UK
003,Tintin,Mr. Tin Tin,Tin Tin,FR
004,,,Rob the Bob,
005,],,Tom the Bob,
Why would Clover add a new column in to the CSV output, is it possible to stop it? I'm using maven 3.3, JDK 8 and Clover version 4.0.2.
I have tried looking for an answer but cannot seem to find anything similar to this issue. When I remove the failing tests Clover works fine.
I'm pretty sure it's connected to the code (or a library) you're using to create that CSV file. Clover on its own won't modify a CSV file. I suppose there's a code which access Java Object fields by reflection. Since class is instrumented by Clover, it adds some instrumentation code (like CLV_TEST_SNIFFER.
There are basically two solutions to your problem:
Modify the code which dumps classes to CSV to exclude static class members (this will effectively exclude Clover objects). Anyway, I believe it's not desired to dump static members to CSV which, as I assume, is a data projection of your Java object (static members don't belong to object but to class). Though, I don't know your full use case, so I may be simply wrong.
You can simply exclude files which cause problems (the ones which are serialised to CSV?) from Clover instrumentation, effectively you'll loose coverage data for those classes.

Cobertura unable to instrument java Interfaces

I am using Cobertura for code coverage of my Java code using maven. When the reports are generated, I found out that there is no coverage generated for Interfaces.java.
While going through the logs, I found below:
[cobertura] INFO [main]
net.sourceforge.cobertura.reporting.html.HTMLReport - Data file does
not contain instrumentation information for the file
com/example/Interface.java. Ensure this class was instrumented, and
this data file contains the instrumentation information.
[file name mocked up]
Please let me know the reason for this. How can I have code coverage for interfaces in java.
Thanks
As commented by Tassos Bassoukos, Interfaces don't have any code, thus there is no possibility of instrumentation. Instead, instrument all classes that implement that interface.
So, cobertura will not be able to show code coverage for interfaces, and skip the interfaces.java files.

How can I prevent anonymous classes in a JaCoCo coverage report?

I current migrate to JaCoCo. In my old system there was one line per source file.
In the code coverage report that I receive from JaCoCo there are a large count of anonymous classes. Is it possible to summarize the result from anonymous classes to its parent class?
I use the current version 0.7.1.
I want one line per source file in the code coverage report.

How do I ignore scanning certain classes using EclEmma and Eclipse?

I am using EclEmma (inside of Eclipse) to scan my JUnit code coverage. This works correctly - however, I do not want EclEmma to scan my src/view folder since it contains Swing code that I consider not worthy of testing.
Is there any way to ignore this folder when EclEmma runs so that it: a) runs faster, and b) does not skew the coverage percentage?
EDIT:
My project's structure is:
src/view
src/model
src/controller
I have tried these (possibly others) with the Path Entries section in the Preferences page:
"src/view"
"src/view/*"
"view"
"view/*"
src/view
These are using the Excludes section in the Preferences page:
*
*View*
*View*.class
src/view/*View*
src/view/*View*.class
They all leave me with the same result of it analysing my entire src folder.
[Edit] The maintainers says you cannot, except one the source directory level: https://github.com/jacoco/eclemma/issues/70
I thought eclemma wasn't excluding files: it is. Just not as I thought.
When you go into excludes in preferences and specify your.classes.here.*, for example, that means those classes won't count towards your getting all your code covered, not that those classes won't be included in what needs to be covered by tests.
Try it and see. Try to exclude a class you know have coverage in it. Once you put that to the excludes preference, on a new coverage run they'll still be there in the coverage window, but they'll come up as 0% and will all be in red.
Rather useless if you ask me. I'm still searching for an adequate solution to exclude classes by name from the classes that need to be covered by tests.
You can specify an exclude field:
Excludes: A list of class names that should be excluded from execution
analysis. The list entries are separated by a colon (:) and may use
wildcard characters (* and ?). (Default: empty)
However, it might be easier to use their options for classpath matching:
Only path entries matching: Comma separated list of strings that must
match with the class path entry. A class path entry matches the
filter, if it contains one of the given strings. (e.g.
"src/main/java", Default: no filter)
See eclemma - how to ignore source about how to ignore src folders.
Also please note their caution,
Warning: If your settings do not match any of the class path entries
in your project(s), every new launch in coverage mode will have an
empty analysis scope.
I have given up on EclEmma because I can't get it to do the things I want it to do, so I use a different method - I'll document it here in case it helps anyone else.
To exclude classes from test, I name all my test classes as *Case.java and then include or exclude them via SuiteClasses. You can read more about that at https://github.com/junit-team/junit4/wiki/Aggregating-tests-in-suites
To measure coverage, I use Maven and Cobertura. This will test just the files specified in my test suites and produce coverage reports accordingly.

Categories

Resources