I want to integrate TestRail with automated tests where I want to run test cases from testRail which calls the test cases in Jenkins and then writes the result back to TestRail itself.
In order to run test cases directly from TestRail, you'll need to first set up a UI script and a trigger script. The UI script will add a button that allows a user to kick the tests, and the trigger script will actually send some command to start the tests. I would note that you'll need to call out to some external test runner that can actually kick and run the tests.
In order to get the results back into TestRail you'll need to leverage TestRail's API. This will allow you to post your results back into the desired run. You can associate your results via test case ID's and place them in either a new or existing run.
Related
I created a project using cucumber to perform e2e tests of various apis I consume. I would like to know if I can run these tests through endpoints to further automate the application that was created.
That way I would be able to upload this app and would not need to keep calling locally.
You can do that if you create a Rest API with a get method which executes the test runner when called.
How to run cucumber feature file from java code not from JUnit Runner
But I don't recommend you to do that since what you are trying to achieve seems to me similar to a pipeline definition.
If you're in touch with the developers of these APIs, you can speak with them about including your test cases in their pipeline, since they probably have one in place.
If, for some reason, you still want to trigger your tests remotely and set it up by your own, I would recommend you to start reading about Jenkins. You can host it on any machine and run your tests from there, accessing from any machine to your jenkins instance:
https://www.softwaretestinghelp.com/cucumber-jenkins-tutorial/
If your code is hosted in any platform like github or gitlab, they already have its own way of creating pipelines and you can use it to run your tests. Read about Gitlab pipelines or Github actions.
I have some features to test using Gherkin and Cucumber. The thing is that the execution is random, and since, for example, the first scenario is creating elements on the page, second one is looking for them and third moving them, all test are crashing cause the execution is going like: nº9 firts, then 8, then 2, then...
I am not using execution tags, or if I use them, I'm using it above "Feature:" to make sure all scenarios are running
Anyone could bring some light here?
General consensus within the test automation community is that your automated tests should be able to run independently. That is, tests should be runnable in any given order and the result of a test should not depend on the outcome of one or more previous tests. Try changing the architecture of your test cases.
It is possible to run tests in specific order using JUnit or TestNG.
https://www.ontestautomation.com/running-your-tests-in-a-specific-order/
How can I have a karate setup so that I can run a bunch of tests when running locally and a subset with running in pre-prod?
When I run the tests locally, I spin up a mock server and set it up using Background. In pre-prod, no mock server is required, so I would like to skip the Background execution.
Also, I was not able to use the #Before annotation to start my cucumber Test Runner.
Use tags. Refer to the documentation: https://github.com/intuit/karate#cucumber-tags
#preprod
Scenario: some scenario
Personally I prefer the approach where you spin up mock servers from your JUnit test classes, and there are a lot of examples, like this one: example
But you can do this also, refer the docs on conditional logic:
* eval if (karate.env == 'preprod') karate.call('mock-start.feature')
I was not able to use the #Before annotation
That's not really helpful, please follow the instructions here: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
I have around 200 testNG test cases can be executed through maven by and suite.xml file. But I want to convert these test cases into web service or any other similar webservice so that anybody can call any test case from there machine and will be able to know whether that particular functionality is working fine at that moment.
But what if no one calls the test webservices for longer time? You won't know the state of your application, if you have any failures/regressions.
Instead, you can use
continuous integration to run the tests automatically on every code push; see Jenkins for a more complete solution; or, more hacky, you can create your own cron job/demon/git hook on a server to run your tests automatically
a maven plugin that displays the results of the last execution of the automated tests; see Surefire for a html report on the state of the last execution of each test
We have a large number of complex integration tests that run for several hours.
How do I receive TestNG XML reports while the tests are running, not after the run?
You can build a TestNG listener which extends org.testng.TestListenerAdapter and override its org.testng.TestListenerAdapter#onFinish wherein you can build the logic to push the results of a <test> tag after its run to a data source of your own. You can also try making it more real-time by building an implementation of the listener interface org.testng.IInvokedMethodListener and within org.testng.IInvokedMethodListener#afterInvocation check if a method is a test method and if yes, start recording the results to a data source of your choice.