I am automating system tests using Silk4J. Silk4J is based on JUnit. In addition I am testing my helper classes using pure JUnit tests.
Now I need to tell Infinitest which tests shall be considered for running immediately. I will choose the unit tests only, not the system tests, of course.
The Infinitest preferences have no options. How to tell Infinitest which tests to run (or which tests not to run)?
I have read Is it possible to specify which tests to choose from?, which seems to be a perfect match, but it didn't work for me. The given link returns HTTP 404. I don't have enough reputation to comment nor to start a bounty for updating the question.
It didn't work because the class name needs to include the package name or .* at the beginning to match any package name.
See Infinitest User Guide for details.
Related
I'm trying to determine which unit tests are executed using JUnit 5 (Jupiter) and which tests are executed with JUnit 4 (Vintage)
I've already tried to see through IntelliJ, but it doesn't show.
Is there any plugin that shows this?
I'm not aware of such a plugin, but that doesn't mean one doesn't exist.
In any case, running your test "suite" via #RunWith(JUnitPlatform.class) should present a hierarchical view in your IDE with the engines as the top-level nodes. Though, Jupiter's dynamic tests will show up as "Unrooted Tests" (at least in Eclipse).
Another option is to execute the JUnit Platform Console Launcher using tree for the details mode. That will present a hierarchical view in the console with the engines as the top-level nodes.
Note, however, that neither of those approaches will provide a test count per test engine. If that's what you're specifically looking for, you'll want to implement your own org.junit.platform.launcher.TestExecutionListener.
You'll find details on all of these options in the JUnit 5 User Guide.
Our team is starting a JUnit 5 project with karate tests.
Currently we are using this as a template for our Karate test runner https://github.com/intuit/karate#junit-5-parallel-execution.
It allows us to pass in the "target/surefire-reports" and then before the test finishes we call ReportBuilder.generateReports(). It is basically identical to this code https://github.com/intuit/karate/blob/b50202b3c8a8916a7db0f3d5196d42086ab80a04/karate-junit4/src/test/java/com/intuit/karate/mock/MockServerTest.java.
This works well, but while I was looking at how to set up JUnit 5 I noticed this very slick fluent api https://github.com/intuit/karate#junit-5.
It would be nice to use that syntax, but I can't get the Cucumber report generated like I can with Runner.parallel. I made sure the maven-surefire-plugin was in build.gradle(although I could have messed that up) but it didn't seem to help.
I also tried doing ReportBuilder.generateReports() and the related logic from the parallel execution example in the #AfterAll function, but couldn't get that working either. The errors suggested that the target/surefire-reports folder didn't exist.
Is the cucumber report supported in the second example? If so, is there a trick to getting it setup?
Great question. The reason we de-couple the JUnit execution and the parallel-runner - is JUnit is more useful in development mode, and you expect detailed pass/fail stats in the IDE for example. But this will be an un-necessary overhead in "CI mode".
That said, we have put in some work on making the Parallel runner a fluent interface, so great timing :) You can find an example on line 57 here.
May I request you to try the develop branch and see if you are missing anything ? Building is easy, here are some instructions: https://github.com/intuit/karate/wiki/Developer-Guide
For a decent sized open source project where developers come and go, someone may fix a bug without realizing that someone else a while back committed a disabled unit test (a la #Ignore). We'd like to find the passing tests that are disabled so we can enable them and update the bug tracker, CC list, and anything else downstream.
What is the best way to occasionally run all #Ignore'd tests and identify the ones that are now passing? We are using Java 1.6 with JUnit4, building our project with ant and transitioning to gradle. We use Jenkins for CI.
A few ideas:
Permanently replace all of our #Ignore annotations with a conditional ignore
http://www.codeaffine.com/2013/11/18/a-junit-rule-to-conditionally-ignore-tests/
Run a custom JUnit4 class runner that changes the behavior of #Ignore.
https://stackoverflow.com/a/42520871
Temporarily comment out all #Ignore annotations so that they run. However we'd need a way to negate the failures.
Sorry, this is not a solution, but rather another alternative that has worked for me:
My key point was to not modify existing (1000s) of unit tests. So no broad code changes. No new Annotations, certainly not temporarily.
What I did was override the JUnit #Ignore detection and make that conditional, via classpath prepends: Check in a separate control file if that test/class is listed or disabled. This is based on package/FQCN/method name and Regexp patterns. If covered, run it even though it still has #Ignore in the unchanged original JUNit Test source.
Log the outcome, amend the control file. Rinse and repeat.
I have written JUnit tests for my class, and would like it to tell me if there is any part of my code that is not unit tested. Is there a way to do this?
Yes, coverage tools like cobertura or emma.
They create reports that show every line in the source code and whether it was executed or not (and aggregated statistics as well).
Of course, they can only show you if the code was run. There is no way to tell if the unit test contained assertions to confirm that the result was correct.
You need some code coverage tools. See here (http://java-source.net/open-source/code-coverage) for some
If you look at the first one I think it does what you need
Cobertura is a free Java tool that calculates the percentage of code accessed by tests. It can be used to identify which parts of your Java program are lacking test coverage. It is based on jcoverage. Features of Cobertura:
Can be executed from ant or from the
command line.
If you use Eclipse, you can also try EclEmma, which shows you which lines of source were covered by your test. This is sometimes more useful than running a coverage tool like Cobertura because you can run a single test from inside Eclipse and then get immediate feedback on what was covered.
Your headline and your actual question differ. The tools mentioned in the other answers can tell you, which part of the code were not tested (=not executed at all). Making "make sure that all parts of code is unit tested" is a different thing. The coverage tools can tell you whether all lines/instructions have been executed, but they don't guarantee that everything is tested functionally (all constellations of data, all execution paths, etc.). This requires some brain power.
In my opinion, test coverage often gives a wrong feeling of safety. E.g. testing trivial getters increases coverage a lot but is rather useless.
If you are using IntelliJ then there is a button titled
"Run With Coverage"
I want to run my unit tests automatically when I save my Eclipse project. The project is built automatically whenever I save a file, so I think this should be possible in some way.
How do I do it? Is the only option really to get an ant script and change the project build to use the ant script with targets build and compile?
Update I will try 2 different approaches now:
Running an additional builder for my project that executes the ant target test (I have an ant script anyway)
ct-eclipse, recommended by Thorbjørn
For sure it it unwise to run all tests, because we can have for example 20.000 tests whereas our change could affect only, let's say 50 of them, among which are tests for the class we have changed and tests for classes that collaborate with our class.
There is an unseful plugin called infinitetest http://improvingworks.com/products/infinitest/ which runs only some tests ( related to class we've changed ) just after we save changes. It also integrate quite nicely with editor ( using annotations ) and problem view - displaying not-passing tests like errors.
Right click on your project > Properties > Builders > New, and there add your ant ant builder.
But, in my opinion, it is unwise to run the unit tests on each save.
See if Eclipse has a plugin for Infinitest.
I'd also consider TestNG as an alternative to JUnit. It has a lot of features that might be helpful in partitioning your unit test classes into shorter and longer running groups.
I believe you are looking for http://ct-eclipse.tigris.org/
I've experimented with the concept earlier, and my personal conclusion was that in order for this to be useful you need a lot of tests which take time. Personally I save very frequently so this would happen frequently, and I didn't find it to be an advantage. It might be different for you.
Instead we bit the bullet and set up a "build server" which watches our CVS repository and builds projects as they change. If the compilation fails or the tests fail we are notified quickly so we can remedy it.
It is as always a matter of taste what works for you. This is what I've found.
I would recommend Inifinitest for the described situation. Infinitest is nowadays a GPL v3 licensed product. Eclipse update site: http://infinitest.github.com
Then you must use INFINITEST. INFINITEST helps you to do Continuous Testing.
Whenever you make a change, Infinitest runs tests for you.
It selects tests intelligently, and only runs the ones you need. It reports unit test failures like compiler errors, and provides additional information that helps you write better tests.