So this is my situation:
I am fairly new to gitlab-ci. I don't host my own gitlab instance but rather push everything to gitab itself. I am not using and am not familiar with any build tools like Maven. I usually work and run my programms from an IDE rather than the terminal.
This is my problem:
When I push my Java project I want my pipeline to start the Junit tests I wrote. Whereas I've found various simple commands for other languages than Java to run unit tests I didn't come across anything for Junit. I've just found people using Maven, running the test locally and then pushing the test reports to gitlab. Is it even possible to easily run Junit tests on the gitlab server with the pipeline without build tools like Maven? Do I have to run them locally? Do I have to learn to start them with a Java terminal command? I've beeen searching for days now.
The documentation is clear:
To enable the Unit test reports in merge requests, you need to add artifacts:reports:junit in .gitlab-ci.yml, and specify the path(s) of the generated test reports.
The reports must be .xml files, otherwise GitLab returns an Error 500.
You then have various example in Ruby, Gio, Java (Gradle or Maven), and other languages.
But with GitLab 13.12 (May 2021), this gets better:
Failed test screenshots in test report
GitLab makes it easy for teams to set up end-to-end testing with automation tools like Selenium that capture screenshots of failed tests as artifacts.
This is great until you have to sort through a huge archive of screenshots looking for the specific one you need to debug a failing test.
Eventually, you may give up due to frustration and just re-run the test locally to try and figure out the source of the issue instead of wasting more time.
Now, you can link directly to the captured screenshot from the details screen in the Unit Test report on the pipeline page.
This lets you quickly review the captured screenshot alongside the stack trace to identify what failed as fast as possible.
See Documentation and Issue.
Related
I created a project using cucumber to perform e2e tests of various apis I consume. I would like to know if I can run these tests through endpoints to further automate the application that was created.
That way I would be able to upload this app and would not need to keep calling locally.
You can do that if you create a Rest API with a get method which executes the test runner when called.
How to run cucumber feature file from java code not from JUnit Runner
But I don't recommend you to do that since what you are trying to achieve seems to me similar to a pipeline definition.
If you're in touch with the developers of these APIs, you can speak with them about including your test cases in their pipeline, since they probably have one in place.
If, for some reason, you still want to trigger your tests remotely and set it up by your own, I would recommend you to start reading about Jenkins. You can host it on any machine and run your tests from there, accessing from any machine to your jenkins instance:
https://www.softwaretestinghelp.com/cucumber-jenkins-tutorial/
If your code is hosted in any platform like github or gitlab, they already have its own way of creating pipelines and you can use it to run your tests. Read about Gitlab pipelines or Github actions.
I am very new to this arena.
I am trying to get the test coverage of my automation test cases written under a completely different repo for the tests using Jacoco.
I want to know if it is possible at first place ? And if it is how to achieve it?
There is separate repo used by the developers for the application source code.
How is it possible to get the test coverage when both source code and tests are in different repos
The unit tests coverage is received by developers.
How can testers get the coverage for their integration tests?
Are you using a CI/CD tool like Jenkins? In that case, you can schedule different builds for different branches in that if you have admin access to the tool.
Edited after seeing the request of John.
Usually, companies will have DevOps admin and other stakeholders of the project who monitor what is happening in each branch. There will be a branching strategy for each Product team. You need to periodically merge contents from developer branch to test branch so that the jacoco test coverage reports don't look confusing to your Dev team members. When, how and what is merged will be decided by the stakeholders and it depends on a lot of facts starting right from the software development process.
If you are following Scrum methodology for software development, at the end of each sprint, developers would give a demo on testable new features or enhancements. Testing team will create test cases based on what is delivered. All these happens in the Sprint review/retrospective/demo meetings.
If you need more information on Jenkins and configuring multiple jobs on them, you need to look at a separate stackexchange forum dedicated to devops. I believe this should be a good place to start for you.
We have REST services created in RestEasy and running in wildfly server. We are running Postman test cases to test the Rest URLs.
Is there a way to get a code coverage of the services when we execute postman test suite?
We use SonarQube to analyse the code coverage.
I think no, a similar question was asked here:
Generate Sonar code coverage report from Postman tests
The original poster commented further down:
In fact, after a bit of googling, as a work-around we could use remote
Jacoco agent hooked in the java application server. We'll try to run
jacoco maven goals before and after the tests execution in order to
generate jacoco coverage report. See: link I'll update the post if we
have some progress.
Also, newman seems to have aticket about it:
https://github.com/postmanlabs/newman/issues/408
Though this might help
Karate is the answer to your problem, provided you are willing to switch to another testing framework.
Here is the link to the demo-example which has code-coverage working: https://github.com/intuit/karate/tree/master/karate-demo#code-coverage-using-jacoco. Since Karate is a JVM implementation it is straightforward, and I recommend you keep Karate tests in the same Maven module (or equivalent) for the easiest option. Otherwise it is possible, but just harder - and you will need to fiddle with a Maven profile etc or do some instrumentation synchronization gymnastics.
I guess if you already have a lot of tests in Postman, the advice here may not be practical. But I'm posting this answer for the benefit of others who will come across this question in the future.
If you are lucky, you may be able to quickly port your tests to Karate using the experimental converter built into the UI: https://github.com/intuit/karate/wiki/Karate-UI#postman-import
Perhaps you can contribute to making that feature prod-ready.
There is nothing available that provides code coverage for postman tests.
In the end we chose rest assured and started replacing all postman tests.
I'm going to execute some Selenium tests after each Bamboo build. As far as I see, the best way is to store them in a separate repo and use specific project (or stage in an existing one) to run these tests. But there is an issue, I can't figure out. I'm using deployment plans to deliver product after build to development environment, so I'd like my tests to be executed, only if the deployment was successful. Does anybody know how to properly express this in Bamboo triggers' terms? Thanks you.
It's rather a confusing and complicated process. As we all know selenium needs a live website to point to in order to execute the tests. There are several ways to accomplish this using Bamboo. I assume you already have the build pipeline set up for automatic deployment. Depending on what you want and how you deploy several agents can be used to execute the tests. Another way is to use Selenium Grid. You want to trigger the selenium task after the deployment happen using several slaves. A grid creates the Hub and Slaves relationship and tells the hub to execute the tests accordingly. Here is some info about the plug-in that can be used to trigger Selenium Testng tests. And, of course, as yo said, you want to create the selenium task to be dependent on the deployment so if the deployment fails then the test will not run. Hope this help!
We have a huge java based application , which is there since few years.We also have a large set of block box test cases with QA team to carry out regression testing.
There is an initiative being taken in our project to improve the quality of the application and on the same lines we have to measure the code which is getting covered by these black box test cases.
I know that we can have a code coverage report through code coverage tools like EMMA,Code Cover,Cobertura ,these tools work along with White Box Unit test cases(i.e JUnit test cases).
I want to know whether any of these tools can be used to generate similar code coverage reports when black box test cases are executed on the application.
With regards to this I have done some google search and found out that the application code can be "Instrumented" and it is possible to generate code coverage reports.
Now what I am trying to do is to
1.Instrument the code in Eclipse using "Code pro" eclipse plugin,
2.Once the code is instrumented ,will generate the jar file of the instrumented code and deploy the same on the test Environment (Unix box).
Now the question is, whether I am going in right direction?
How and where the code coverage reports will be generated when black box testing is being done on the instrumented code on server(not local machine).
Take a look at jacoco
http://www.eclemma.org/jacoco/trunk/doc/mission.html
This uses a java agent and can instrument your code at runtime
You can use jacoco for this, set the jvm under to test to run with the tcpserver option, run tests and then connect to it using the tcpclient option. If you want to collect coverage separately for n runs then you can connect to it over jmx and call reset