I have a Java project, which consists of loads of maven modules and a considerable amount of unit/integration tests. The project is configured to create test reports via the surefire plugin. Now this plugin basically creates an XML-report per test class and is scheduled to run once a day and executed on Jenkins.
What I want to do is, send the those XML-reports to a test management system (XRAY) in order to make them more visible and manageable. My (naive) approach would be to just just add a post build script on Jenkins and send those reports via curl to the test managements REST-API. This API offers a way to send a single report file at a time. This report file can either be single or nested, i.e. I can basically send both of the following and it works:
Single report
<testsuite>
...
</testsuite>
Aggregated report
<testsuites>
<testsuite ... />
<testsuite ... />
</testsuites>
The REST-API can handle both, that is the IBM JUnit schema and the standard surefire schema
Now to the problem; I obviously want to combine those reports into one to avoid having to make a billion requests to the REST-API. However I can't seem to find an automated way. What I've tried so far is
play around with the surefire plugin to merge the XML reports, but no appropriate option seems to exist
Organise Tests into a (JUnit) test suite, but the output remains an xml report per test
finding alternative plugins/tools that address this issue, no luck
The only other way I can think of is to write a "merge script" myself, possibly using some sort of XSLT-transformation. But I'd rather not. Any help is much appreciated, thanks!
The solution would be to use an external utility for that as surefire seems to not support it.
I've used successfully junit-merge utility, which is an NPM package, as you can see for example in this tutorial.
The usage is pretty straightforward; you just need to specify the output file and the input folder containing the multiple JUnit XML based reports.
junit-merge -o results.xml -d target/surefire-reports/
Related
So this is my situation:
I am fairly new to gitlab-ci. I don't host my own gitlab instance but rather push everything to gitab itself. I am not using and am not familiar with any build tools like Maven. I usually work and run my programms from an IDE rather than the terminal.
This is my problem:
When I push my Java project I want my pipeline to start the Junit tests I wrote. Whereas I've found various simple commands for other languages than Java to run unit tests I didn't come across anything for Junit. I've just found people using Maven, running the test locally and then pushing the test reports to gitlab. Is it even possible to easily run Junit tests on the gitlab server with the pipeline without build tools like Maven? Do I have to run them locally? Do I have to learn to start them with a Java terminal command? I've beeen searching for days now.
The documentation is clear:
To enable the Unit test reports in merge requests, you need to add artifacts:reports:junit in .gitlab-ci.yml, and specify the path(s) of the generated test reports.
The reports must be .xml files, otherwise GitLab returns an Error 500.
You then have various example in Ruby, Gio, Java (Gradle or Maven), and other languages.
But with GitLab 13.12 (May 2021), this gets better:
Failed test screenshots in test report
GitLab makes it easy for teams to set up end-to-end testing with automation tools like Selenium that capture screenshots of failed tests as artifacts.
This is great until you have to sort through a huge archive of screenshots looking for the specific one you need to debug a failing test.
Eventually, you may give up due to frustration and just re-run the test locally to try and figure out the source of the issue instead of wasting more time.
Now, you can link directly to the captured screenshot from the details screen in the Unit Test report on the pipeline page.
This lets you quickly review the captured screenshot alongside the stack trace to identify what failed as fast as possible.
See Documentation and Issue.
My company is using FitPro + FitLibrary to test our applications. Our main test suites are .suite files and they invoke FIT tests pages, which bear the .fit extension (contents
is HTML).
Our application Fixtures are built on top of FitLibraryRunner.jar (a release issued on 01/28/2007), and we have a .bat/.sh script which launches our FIT tests suite using
com.luxoft.fitpro.runner.TestCaseRunner, which is part of the fitpro.jar library.
This setup is convenient for us because we provide the same application but customized for many customers, and all you need to run FIT tests is fitpro.jar among your classpath, and you don't depending upon other stuff.
As FitPro seems to be no longer maintained, our alternative would be to switch to FitNesse.
Now, it would seem as per my understanding that FitNesse does not offer runners that would allow executing FIT tests suites outside of its wiki. Let me precise that the usage of a Wiki is not really useful for us due to us packaging the same libraries but in differents stages of development, and to many customers.
I would like to know if any of you could ever succeeded in launching the FitNesse/SLIM engine outside of the wiki context ? I am looking for a way to invoke a Runner provided with FitNesse, that reads a main suite test file (.suite) and produces an HTML or XML-based report as output, just like the way we do with FitPro.
I am also told that I could not use the .suite and .fit pages we have created with FitPro over the years.
Thanks in advance for any feedback!
J-C
If you don't want to use the FitNesse wiki, then FitNesse doesn't add any value for you. FitNesse renders HTML from its wiki pages and passes them to a test execution engine, like Slim, FitLibrary, fitSharp, etc. FitLibrary and fitSharp can also execute tests sourced from HTML files. I haven't used FitPro but it appears to feed FitLibrary with tests from its own .suite and .fit files. Your best bet may be to use FitLibrary's FolderRunner and organize your suites into folders of HTML files that FolderRunner can process.
I'm trying to figure out how to get cucumber-jvm to produce separate xml report for each feature file. At this moment if I'm running it with mvn I'm getting a single xml, but I wan't a separate file for each.
For each feature file a descendant of org.junit.runners.ParentRunner is being created. You can browse this code here.
Is it possible somehow to tell surefire to create a separate xml for each runner?
More info here: #171
Thank you, guys!
Have you tried to split it with XSLT or other language? Sometimes is better to create an additional task in maven to run after the previous one. If the first XML contains all the data and is possible to split, I would tried it.
I'm looking for some kind of test (mostly jUnit, but not exclusivelly) management. What I have in mind is a tool that would allow to easily enable/disable individual tests based on configuration (xml file, property file or in DB etc.).
I was thinking that there must be some maven plugin that could do that, or some tool. Can you suggest anything?
There are Maven plugins for JUnit and TestNG that allow you to define which tests to run from the command line (or all).
JUnit allows the creation of test suites. With some care, you can tailor your test suites to match your "to be enabled / disabled" sets of tests.
Failing that, you could wrap the test suite idea with something that generates / references an XML file for configuration (if that's really the path you want to take). Considering that it's all in a Java class, you might find it easier to do by another means (perhaps an properties file or just in-class handling).
I've got a Java software that reads settings from properties files and database, reads input files from a directory and creates output files in another directory. It also makes modifications to database.
I need to improve testing of this software from being manual to automatic. Currently the user copies some files to input directory, executes the program and inspects the files in the output director. I'd like to automate this to just running the tests and inspecting the test result file. The test platform would have a expected result file(s) for each input file. The test results should be readable by people that are not programmers :)
I don't want to do this in a jUnit test in the build phase because the tests have to be executed against development and test environments. Is there any tools/platforms that could help me with this or should I build this kind of thing from scratch?
I'd recommend to use TestNG testing framework.
This is functionality testing framework, which provides similar to jUnit functionality, but has a number of features specific to functional testing - like test dependencies, groups etc.
The test results should be readable by
people that are not programmers :)
You can implement your own test listener and use it to build custom test report.