For a customer we need to generate detailed test reports for integration tests which not only show, that everything is green, but also what the test did. My colleagues and I are lazy guys and we do not want to hack spreadsheets or text documents.
For that, I think about a way to document the more complex integration tests with JavaDoc comments on each #Test annotated method and each test class. For the test guys it is a good help to see to which requirement, Jira ticket or whatever the test is linked to and what the test actually tries to do. We want to provide this information to our customer, too.
The big question now is: How can we put the JavaDoc for each method and each test class into the JUnit reports? We use JUnit 4.9 and Maven.
I know, that there is a description for each assertXXX(), but we really would need a nice HTML list as result or a PDF document which lists all classes and there documentation and below that all #Test methods and their description, the testing time, the result and if failed, the reason why.
Or is there another alternative to generate fancy test scripts? (Or should we start an OpenSource project on this!? ;-) )
Update:
I asked another question on how to add a RunListener to Eclipse to have it also report in Eclipse when started there. The proposed solution with a custom TestRunner is another possibility to have the test results report. Have a look: How can I use a JUnit RunListener in Eclipse?
One way to achieve this would be to use a custom RunListener, with the caveat that it would be easier to use an annotation rather than javadoc. You would need to have a custom annotation such as:
#TestDoc(text="tests for XXX-342, fixes customer issue blahblah")
#Test
public void testForReallyBigThings() {
// stuff
}
RunListener listens to test events, such as test start, test end, test failure, test success etc.
public class RunListener {
public void testRunStarted(Description description) throws Exception {}
public void testRunFinished(Result result) throws Exception {}
public void testStarted(Description description) throws Exception {}
public void testFinished(Description description) throws Exception {}
public void testFailure(Failure failure) throws Exception {}
public void testAssumptionFailure(Failure failure) {}
public void testIgnored(Description description) throws Exception {}
}
Description contains the list of annotations applied to the test method, so using the example above you can get the Annotation TestDoc using:
description.getAnnotation(TestDoc.class);
and extract the text as normal.
You can then use the RunListener to generate the files you want, with the text specific to this test, whether the test passed or failed, or was ignored, the time taken etc. This would be your custom report.
Then, in surefire, you can specify a custom listener, using:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.10</version>
<configuration>
<properties>
<property>
<name>listener</name>
<value>com.mycompany.MyResultListener,com.mycompany.MyResultListener2</value>
</property>
</configuration>
</plugin>
This is from Maven Surefire Plugin, Using JUnit, Using custom listeners and reporters
This solution has the disadvantage that you don't have the flexibility of javadoc as far as carriage returns, formatting is concerned, but it does have the advantage that the documentation is in one specific place, the annotation TestDoc.
Have you looked at Maven Sure-fire reports?
You can generate a HTML report from your JUnit Tests.
http://maven.apache.org/plugins/maven-surefire-report-plugin/
I'm not sure how customizable it is though. But it's a good starting point.
I also know that TestNG ( alternative to JUnit ) has some report generating capabilities.
http://testng.org/doc/documentation-main.html#logging-junitreports
I would also recommend log4j
http://logging.apache.org/log4j/1.2/manual.html
you can use jt-report an excellent framework for test reporting.
I have created a program using testNG and iText which outputs the test results in a nice pdf report. You can put a description of your test in the #Test tag, and that can be included in the .pdf report also. It provides the run times of the tests, and for the entire suite. It is currently being used to test webapps with selenium, but that part could be ignored. It also allows you to run multiple test suites in one run, and if tests fail, it allows you to re-run only those tests without having to re-run the entire suite, and those results will be appended to the original results PDF. See below the image for a link to the source if you are interested. I wouldn't mind this becoming an opensource project as I have a good start on it, though I'm not sure how to go about doing that. Here's a screenshot
So I figured out how to create a project on sourceforge. Here's the link sourceforge link
As mentioned above maven is definitely the way to go.. It makes life really easy. You can create an maven project pretty easy using m2eclipse plugin. Once that is done. Just run these commands:
cd <project_dir_where_you_have_pom_file>
mvn site:site
This command will create the style sheets for you. In the same directory run:
mvn surefire-report:report
This will run the test cases and convert the output to html. You can find the output in the 'target/site/surefire-report.html'.
Below is the snippet. As you can see all the test cases (written in JUnit) are shown in the html. Other meta info like total no of test cases ran, how many successful, time taken etc., is also there.
Since I cannot upload image I cant show you the output.
You can go a step further and can give the exact version of the plugin to use like
mvn org.apache.maven.plugins:maven-site-plugin:3.0:site org.apache.maven.plugins:maven-surefire-report-plugin:2.10:report
Maybe it is worth taking a look on "executable specification" / BDD tools like FIT/FitNesse, Concordion, Cucumber, JBehave etc.
With this practice you will have a possibility not only satisfy the customer's requirement formally, but you will be able do bring transparency onto a new level.
Shortly speaking, all these tools allow you (or, better, customer) to define scenarios using natural language or tables, define binding of natural language constructs to real code, and run these scenarios and see if they succeed or fail. Actually you will have a "live" spec which shows what is already working as expected and what is not.
See a good discussion on these tools:
What are the differences between BDD frameworks for Java?
Related
Our team is starting a JUnit 5 project with karate tests.
Currently we are using this as a template for our Karate test runner https://github.com/intuit/karate#junit-5-parallel-execution.
It allows us to pass in the "target/surefire-reports" and then before the test finishes we call ReportBuilder.generateReports(). It is basically identical to this code https://github.com/intuit/karate/blob/b50202b3c8a8916a7db0f3d5196d42086ab80a04/karate-junit4/src/test/java/com/intuit/karate/mock/MockServerTest.java.
This works well, but while I was looking at how to set up JUnit 5 I noticed this very slick fluent api https://github.com/intuit/karate#junit-5.
It would be nice to use that syntax, but I can't get the Cucumber report generated like I can with Runner.parallel. I made sure the maven-surefire-plugin was in build.gradle(although I could have messed that up) but it didn't seem to help.
I also tried doing ReportBuilder.generateReports() and the related logic from the parallel execution example in the #AfterAll function, but couldn't get that working either. The errors suggested that the target/surefire-reports folder didn't exist.
Is the cucumber report supported in the second example? If so, is there a trick to getting it setup?
Great question. The reason we de-couple the JUnit execution and the parallel-runner - is JUnit is more useful in development mode, and you expect detailed pass/fail stats in the IDE for example. But this will be an un-necessary overhead in "CI mode".
That said, we have put in some work on making the Parallel runner a fluent interface, so great timing :) You can find an example on line 57 here.
May I request you to try the develop branch and see if you are missing anything ? Building is easy, here are some instructions: https://github.com/intuit/karate/wiki/Developer-Guide
There are so many posts about running JUnit tests in a specific order and I fully understand:
Tests should not order specific
and that the creators did this with no 1 in mind
But I have test cases that creates a bunch out output files. I need to capability to have one final test that goes and collects these files, zip it and emails it of to someone.
Is there a way to group JUnit tests together for me to have a "wrap up" group that goes and do this? Or is there a better way of doing this?
I am running these from Jenkins as a maven job. I could create another job that does just that based on the previous jobs output but would prefer if I can do it all in one meaning I would be able to run it everywhere even from my IDE.
Maybe the #After and #AfterClass annotations are what you are looking for:
#AfterClass
void cleanupClass() {
//will run after all tests are finished
}
#After
void cleanup() {
//will run after every test
}
However, I would consider handling this through Jenkins if possible. In my opinion the annotations above are for cleaning up any kind of setup that was previously done in order to do the testing.
Sending these files through email does not sound like part of the testing and therefore I would be inclined to keep it separated.
I guess the real problem is that you want the results and output of the tests sent via email.
Your suggestion of using a test for this threw me on the wrong track.
Definitely use some sort of custom Jenkins post hook to do this. There are some fancy plugins that let you code groovy which will do the trick.
Do not abuse a unit test for this. These (should) also run locally as part of builds and you don't want that email being sent every time.
I am making an framework that internally user JUnit and REST Assured. This framework will have the 4 #Test methods for CRUD operations. Whenever the user want to do any operation, he will call only that particular Test method. But at the end of the each operation(say GET or DELETE or any other), it should generate the report.
I tried using surefire-report plugin. As I have read, that will generate report only when we build the project(running all the Test methods).
Is there any mechanism that fulfills my requirement of generation report for individual run also?
Execution will be like : final output will be the jar with individual CRUD facility.API.execute(GET, end_point_name);API.execute(POST, end_point_name,data);Test method get and post is called respectively for the above calls. Report should be generated for both the test cases for normal run as java application.
There are 3 solutions to your problem :
Either you write your logger statement and do proper logging of the events. You can either store it in DEBUG, INFO etc mode for better understanding and more control.
ExtentReports is another way to go :
http://www.ontestautomation.com/creating-html-reports-for-your-selenium-tests-using-extentreports/ refer the above link where they have a provided a detailed way of using the same.
You can also create a separate testng.xml file. Like maintaining a separate suite file this will internally make sure with the help surefire to create a separate reports.
I'm running integration test on a huge code base. In coverage report I want to know which test covered a certain line in code. Is there anyway of doing it with jacoco?
Just in case someone still looking for a solution to this question.
In my case, I have written a small demo using Jacoco to generate a coverage report containing covered line information for each test case. Based on this project structure, I then simply wrote a script to automatically run test cases one by one and collect each coverage report to get information that which lines are cover by each test.
Note that, this solution is not the best solution (it is time-consuming when the number of test cases is large), but it just helps me get the covered line information of test cases with Jacoco.
Please refer to https://github.com/chenliushan/JacocoExample for the demo.
JaCoCo doesn't collect that information, so it can't report it.
Conceivably, you could run each test independently with JaCoCo and generate a coverage report everytime, that way each test shows the exact lines of code it tested. (then you have to wrap this with a custom aggregated report I suppose, where you can navigate from one test to the next).
I get that this might not be practical with a huge codebase and a large number of tests.
Another limitation is that you don't get to "what are ALL the tests that exercised that line of code ?".
As #Rogério remarked, other tools might be able to provide that functionality.
I've recently been lifted out of the .Net world into the Java world and I miss my unit tests.
Using Visual Studio I used NUnit and TestDriven.net to run my unit tests.
What is a comparable system for Java Using Eclipse?
I'm looking specifically for the plugins that will get me going, or a guide on how to do it.
I'm aware that JUnit is what NUnit was initially based on, but I want to know the best way to integrate it into Eclipse as there seem to be a few plugins that do this and I don't have the time to play around with them all.
UPDATE
Okay I didn't know that JUnit was built into the IDE. Are there any plugins that make using JUnit any easier?
Using JUnit with eclipse is actually very easy. Just go to File->New... and select JUnit Test Case. Eclipse will handle adding the JUnit library and all of the imports.
Which version of Eclipse are you using?
For as long as I remember (I've been using Eclipse since early 3.xs), Eclipse supports JUnit out of the box. You just:
Right-click on a project -> Run As -> JUnit Test
Does this not work for you?
I've been using moreUnit for a few years and can't live without its Ctrl+J shortcut to switch between the class and its test case.
I've also found EclEmma useful for finding untested code.
Easier than "Right-click on a project -> Run As -> JUnit Test"? Like you want it bound to a keypress (because it probably is). Lemme check--Yeah, alt-shift-X, then "T". Easy enough?
There is also window/show view/other/java/JUnit that will give you a junit run bar in a window. You can then just hit the run tests button and it will run all the tests in your project/section.
Ctrl-shift-L is great for figuring out keybindings if you are getting to know eclipse.
Also, get VERY familiar wtih ctrl-space, just press it whenever you're in the middle of typing something (seriously, try it with everything!) Also type "sysout[ctrl-space]"
JUnit 4 is actually really easy to use, as long as you're using a project that targets Java 5 or newer, and have added it to the project.
I mean, how much easier can you get than
#Test
public myTest() {
// Test code here
}
There are also #Before, #After, #BeforeClass, #AfterClass, and #Ignore. The *Class methods need to be static methods. #Before runs before each test, #BeforeClass runs before the first test... but keep in mind that JUnit tests can run in any order.
If you're doing database tests, you may want to look into the DBUnit addon, although you need to take some special steps to use it with JUnit 4's native mode.
What do you mean with "make using JUnit any easier"?
I use Ant for runnings tests as a task. Output will be stored into a flat file or a html file. It depends on the TestRunner.
Please specify your question and you'll get answers! :)
fit (http://www.theserverside.com/news/thread.tss?thread_id=39417)
dbunit (http://www.dbunit.org/)
many others
in eclipse, you can right click a package and select run as a junit test.
be careful of http://xunitpatterns.com/test%20fixture%20-%20ambiguous.html. iirc, this boils down to junit creating an instance of each test case before calling setup and nunit just creating one instance.
I've used the testNG which has a plug in for eclipse.