Develop Reference

Is it possible to generate Cucumber HTML Reports with Karate's JUnit5 fluent API?

Our team is starting a JUnit 5 project with karate tests.
Currently we are using this as a template for our Karate test runner https://github.com/intuit/karate#junit-5-parallel-execution.
It allows us to pass in the "target/surefire-reports" and then before the test finishes we call ReportBuilder.generateReports(). It is basically identical to this code https://github.com/intuit/karate/blob/b50202b3c8a8916a7db0f3d5196d42086ab80a04/karate-junit4/src/test/java/com/intuit/karate/mock/MockServerTest.java.
This works well, but while I was looking at how to set up JUnit 5 I noticed this very slick fluent api https://github.com/intuit/karate#junit-5.
It would be nice to use that syntax, but I can't get the Cucumber report generated like I can with Runner.parallel. I made sure the maven-surefire-plugin was in build.gradle(although I could have messed that up) but it didn't seem to help.
I also tried doing ReportBuilder.generateReports() and the related logic from the parallel execution example in the #AfterAll function, but couldn't get that working either. The errors suggested that the target/surefire-reports folder didn't exist.
Is the cucumber report supported in the second example? If so, is there a trick to getting it setup?

Great question. The reason we de-couple the JUnit execution and the parallel-runner - is JUnit is more useful in development mode, and you expect detailed pass/fail stats in the IDE for example. But this will be an un-necessary overhead in "CI mode".
That said, we have put in some work on making the Parallel runner a fluent interface, so great timing :) You can find an example on line 57 here.
May I request you to try the develop branch and see if you are missing anything ? Building is easy, here are some instructions: https://github.com/intuit/karate/wiki/Developer-Guide

Finding #Ignore'd tests that are now passing

For a decent sized open source project where developers come and go, someone may fix a bug without realizing that someone else a while back committed a disabled unit test (a la #Ignore). We'd like to find the passing tests that are disabled so we can enable them and update the bug tracker, CC list, and anything else downstream.
What is the best way to occasionally run all #Ignore'd tests and identify the ones that are now passing? We are using Java 1.6 with JUnit4, building our project with ant and transitioning to gradle. We use Jenkins for CI.
A few ideas:
Permanently replace all of our #Ignore annotations with a conditional ignore
http://www.codeaffine.com/2013/11/18/a-junit-rule-to-conditionally-ignore-tests/
Run a custom JUnit4 class runner that changes the behavior of #Ignore.
https://stackoverflow.com/a/42520871
Temporarily comment out all #Ignore annotations so that they run. However we'd need a way to negate the failures.

Sorry, this is not a solution, but rather another alternative that has worked for me:
My key point was to not modify existing (1000s) of unit tests. So no broad code changes. No new Annotations, certainly not temporarily.
What I did was override the JUnit #Ignore detection and make that conditional, via classpath prepends: Check in a separate control file if that test/class is listed or disabled. This is based on package/FQCN/method name and Regexp patterns. If covered, run it even though it still has #Ignore in the unchanged original JUNit Test source.
Log the outcome, amend the control file. Rinse and repeat.

Unit tests coverage using Jacoco for test classes written using Powermock

I am trying to get the code coverage report on the sonarqube dashboard on jenkins. The code coverage report is coming up but showing only 4.6% coverage. On investigating I found out that the test classes written using PowerMocks are getting skipped.
On further investigation I found that "JaCoCo doesn't play well with dynamically modified/created classes (this is the way how powermock works). This is a known limitation we can't currently do anything about".
Is there any work around for this so that I can get proper code coverage for test classes written using PowerMocks too.

Simple answer: no, there isn't.
Long answer - boils down to these options:
have look into this Wiki page by the PowerMock team - maybe maybe "offline instrumentation" works out for you.
hope that the corresponding bug gets fixed at some point (I wouldn't hold my breath on that)
get rid of your dependency to PowerMock(ito) - by refactoring and improving your production code
[ I think I evaluated various coverage tools long time ago; and there was one commercial one that claims to work even with PowerMock. But I don't recall any specifics. So I am basically saying: there might be a minuscule chance that another, proprietary coverage tool works with PowerMock ]

I have managed to generate PowerMock coverage with Jacoco, using powermock-module-javaagent.
Just make sure you put powermock agent after jacoco agent:
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<useSystemClassLoader>true</useSystemClassLoader>
<argLine>${jacocoArgLine} -javaagent:${settings.localRepository}/org/powermock/powermock-module-javaagent/${powermock.version}/powermock-module-javaagent-${powermock.version}.jar -noverify</argLine>
...
If you want to see an example, take a look at this project: https://github.com/jfcorugedo/sonar-scanner
Here you can see that sonar takes into account static methods and new statements mocked by PowerMock:
If you want to mock newstatements make sure you use PowerMockRule instead of PowerMockRunner.
Take a look at this test

What works for me is to remove this
#RunWith(PowerMockRunner.class)
And add this in the class
#Rule
public PowerMockRule rule = new PowerMockRule();
Also need to add the dependency for powermockito junit4 rule.
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-module-junit4-rule</artifactId>
<version>2.0.2</version>
<scope>test</scope>
</dependency>
This official page will help more to understand it. PowerMock

How to get code coverage using postman test

We have REST services created in RestEasy and running in wildfly server. We are running Postman test cases to test the Rest URLs.
Is there a way to get a code coverage of the services when we execute postman test suite?
We use SonarQube to analyse the code coverage.

I think no, a similar question was asked here:
Generate Sonar code coverage report from Postman tests
The original poster commented further down:
In fact, after a bit of googling, as a work-around we could use remote
Jacoco agent hooked in the java application server. We'll try to run
jacoco maven goals before and after the tests execution in order to
generate jacoco coverage report. See: link I'll update the post if we
have some progress.
Also, newman seems to have aticket about it:
https://github.com/postmanlabs/newman/issues/408
Though this might help

Karate is the answer to your problem, provided you are willing to switch to another testing framework.
Here is the link to the demo-example which has code-coverage working: https://github.com/intuit/karate/tree/master/karate-demo#code-coverage-using-jacoco. Since Karate is a JVM implementation it is straightforward, and I recommend you keep Karate tests in the same Maven module (or equivalent) for the easiest option. Otherwise it is possible, but just harder - and you will need to fiddle with a Maven profile etc or do some instrumentation synchronization gymnastics.
I guess if you already have a lot of tests in Postman, the advice here may not be practical. But I'm posting this answer for the benefit of others who will come across this question in the future.
If you are lucky, you may be able to quickly port your tests to Karate using the experimental converter built into the UI: https://github.com/intuit/karate/wiki/Karate-UI#postman-import
Perhaps you can contribute to making that feature prod-ready.

There is nothing available that provides code coverage for postman tests.
In the end we chose rest assured and started replacing all postman tests.

testNG tests extending BaseTest

We noticed that when testNG test cases extend TestCase (JUnit) those tests start executing as Junit tests. Also, I should probably mention, the tests are run through Maven.
Is this a bug or a feature? Is it possible to override this behavior and still run those types of tests as TestNG tests? Do you know a link where TestNG talks about this?
thanks.

I didn't think either TestNG or JUnit required any base classes now that both use annotations to specify test methods. Why do you think you need to extend a class? And why on earth would a TestNG class extend the JUnit base class TestCase? Is it any surprise that they run as JUnit tests?
It sounds like neither bug nor feature but user error on your part. I'm not sure what you're getting at here. Why would you do this?
UPDATE: Your question is confusing me. Did you have JUnit tests running successfully that you're not trying to convert to TestNG, or visa versa? I'm having a very hard time understanding what you're trying to achieve here. Leave Maven out of it. It's immaterial whether they're run by you, Ant, or Maven.

Looking at the maven surefire plugin info I can't see any way to select a test for TestNG processing only if it also extends a jUnit 3 class.
IFAIK your best bet is to just work on each class seperately, removing the jUnit references and then retesting. That way you never have the mixture in one class and you should avoid problems. To make the work manageable I would be inclined to do this only when I was changing a test case for some other reason.