I have a framework that uses TestNG with Cucumber and runs in parallel.
Now i need some of my tests to be run sequentially.
There are some cases to do such trick with exclude option in TestNG xml. But, since im using Cucumber, i have only one #Test method, that parses cucumber features and runs its step definitions so i can't modify any particular #Test.
I think i can do what i need by creating only one step definition for sequential step, then synchronized it with some static lock. But in my opinion this is way too ugly and unconvenient to use and read. But i hope there is some more clear and easy to achieve way.
Any suggestions would be appreciated
Related
I have some features to test using Gherkin and Cucumber. The thing is that the execution is random, and since, for example, the first scenario is creating elements on the page, second one is looking for them and third moving them, all test are crashing cause the execution is going like: nº9 firts, then 8, then 2, then...
I am not using execution tags, or if I use them, I'm using it above "Feature:" to make sure all scenarios are running
Anyone could bring some light here?
General consensus within the test automation community is that your automated tests should be able to run independently. That is, tests should be runnable in any given order and the result of a test should not depend on the outcome of one or more previous tests. Try changing the architecture of your test cases.
It is possible to run tests in specific order using JUnit or TestNG.
https://www.ontestautomation.com/running-your-tests-in-a-specific-order/
How can I have a karate setup so that I can run a bunch of tests when running locally and a subset with running in pre-prod?
When I run the tests locally, I spin up a mock server and set it up using Background. In pre-prod, no mock server is required, so I would like to skip the Background execution.
Also, I was not able to use the #Before annotation to start my cucumber Test Runner.
Use tags. Refer to the documentation: https://github.com/intuit/karate#cucumber-tags
#preprod
Scenario: some scenario
Personally I prefer the approach where you spin up mock servers from your JUnit test classes, and there are a lot of examples, like this one: example
But you can do this also, refer the docs on conditional logic:
* eval if (karate.env == 'preprod') karate.call('mock-start.feature')
I was not able to use the #Before annotation
That's not really helpful, please follow the instructions here: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
There are so many posts about running JUnit tests in a specific order and I fully understand:
Tests should not order specific
and that the creators did this with no 1 in mind
But I have test cases that creates a bunch out output files. I need to capability to have one final test that goes and collects these files, zip it and emails it of to someone.
Is there a way to group JUnit tests together for me to have a "wrap up" group that goes and do this? Or is there a better way of doing this?
I am running these from Jenkins as a maven job. I could create another job that does just that based on the previous jobs output but would prefer if I can do it all in one meaning I would be able to run it everywhere even from my IDE.
Maybe the #After and #AfterClass annotations are what you are looking for:
#AfterClass
void cleanupClass() {
//will run after all tests are finished
}
#After
void cleanup() {
//will run after every test
}
However, I would consider handling this through Jenkins if possible. In my opinion the annotations above are for cleaning up any kind of setup that was previously done in order to do the testing.
Sending these files through email does not sound like part of the testing and therefore I would be inclined to keep it separated.
I guess the real problem is that you want the results and output of the tests sent via email.
Your suggestion of using a test for this threw me on the wrong track.
Definitely use some sort of custom Jenkins post hook to do this. There are some fancy plugins that let you code groovy which will do the trick.
Do not abuse a unit test for this. These (should) also run locally as part of builds and you don't want that email being sent every time.
I'm new to BDD and particularly Cucumber.
Can I get a features and its steps from a variable? Also, I want to get a feature and its steps from a test tracker (TestRail) before running tests by the special selection of this tests, and put it in a list, then one by one get a scenario and run it.
Is there such a possibility? Should I use Cucumber or another framework for this?
No, you can't define a Cucumber scenario in code (or at least not in a supported way). But if you were going to write code to get a scenario and its steps from your test tracker and run it, you could equally well write code to put the scenario and its steps in files and run the scenario with the cucumber executable.
I don't know of a Java testing framework in which you can define tests dynamically. You could do that in Ruby with RSpec or (less cleanly) minitest. But I don't know whether a Ruby test framework would be acceptable, or whether it would be OK for the people writing entries in your test tracker to have to read and/or write RSpec examples. (It seems strange to have Cucumber step definitions in a test tracker, too; having features in a test tracker seems more reasonable, aside from the question of how to run them.)
I have a JUnit 4 test suite with BeforeClass and AfterClass methods that make a setup/teardown for the following test classes.
What I need is to run the test classes also by them selves, but for that I need a setup/teardown scenario (BeforeClass and AfterClass or something like that) for each test class. The thing is that when I run the suite I do not want to execute the setup/teardown before and after each test class, I only want to execute the setup/teardown from the test suite (once).
Is it possible ? Thanks in advance.
I don't know of any standard way to do this with JUnit. The reason for it, as you probably already know, is that your test cases should run independently of each other. This concerns the "normal" setup/teardown methods which run before and after each test method. Class setup and teardown is a bit different though - although I would still prefer running my tests independently and staying out of the trouble zone.
However, if you really are convinced of what you are doing, you could use a global flag to signal whether or not the class setup/teardown is to run, and to check for its state in the class setup/teardown methods. In your test suite, you could include a special class as the very first one, which does nothing more than execute the setup and set the global flag to indicate to the real test cases that their class setup/teardown methods must not be run. Similarly, a special last class in the suite can execute the teardown code. The caveat is that I am afraid JUnit does not guarantee the order of execution of test classes inside a suite, although most probably it does execute them in the specified order - but this is just an implementation detail. Try it out, it may work for you - but there is no guarantee it will always do what you expect.
If you have jUnit 4.7+ I recommend looking into the new feature called Rules (which are explained in this blog post). They might not be exactly what you want, but they are probably the best you get with jUnit.
Supposedly TestNG has better test grouping possibilities, but I haven't really looked into it myself yet.
No, there's no standard way to do this in JUnit, though you could hack something up as Péter Török suggested.
Note however that you are more or less abusing JUnit in doing this. The whole point of unit tests it that they are independent of each other. This is because dependencies between tests create a total maintenance nightmare (tests failing because the run in the wrong order).
So I'd advise you to strongly consider if it's not better to just always run the setup...