I want to execute testcases in below order.
Example: execution starts with 3rd scenario of 2nd feature then 4th scenario of 6th feature and so on.
Can we do this customization using TestNG/cucumber options/java or any other tool?
Is it possible using hooks or cli options --order ?
You could probably achieve this with use of #tags above the above the line that has the Scenario keyword. Then execute multiple test runs but using the tag options as parameters passed to the build as mentioned here.
However, this approach does seem a little odd. If the reasons for doing this are to force an order due to tests leaving data cached in the running application that require state from the other one then this is not good practice. Each Scenario should be completely runnable in isolation and without any prior one that might set up state.
Related
I have a Feature file and the background step is pretty simple. However, the setup done in that step needs to be working with two different types of values and it is applicable to all the scenarios within that FF. Is there a way we can make this background dynamic?
Example: I want to do as below:
Background:
Given hospital configuration is done using '<some config>'
|some config|
| abc |
|xyz |
There are three ways I can think of to run a feature file twice, one with one configuration and the other with another configuration.
Duplicate the feature file and change the background so that it specifically loads the particular configuration.
Before dismissing this solution look at some of the positives.
its very simple
its easy to document (just put comments in the preamble about the duplication)
it supports customization specific to a particular profile.
The last point is worth expanding on. If the behavior is identical with each profile then what is the point of the profile. Basically you are saying it has no effect at all. If it has not effect at all then why are you testing it. If the behavior varies between profiles then you really want to explore and express those differences.
Run the feature twice by using an external setting.
This basically comes down to having Cucumber pull the setting from the environment and running cucumber twice, once for the first setting, and then again for the second setting.
So something like
SETTING=abc cucumber ...
SETTING=xyz cucumber ...
Use an around hook to run the internals of a scenario twice.
Here you are putting a tag on the feature and making a custom hook to run the scenarios twice. This is equivalent to the second solution but you are embedding the mechanism inside a single run of the features.
In ruby this would look something like
# run_twice_hook.rb
Around(#run_twice) do |scenario, block|
load_first_setting
block.call
load_second_setting
block.call
end
I think this solution comes closest to what the OP wants.
I also think this is the worst solution, because its technical, tricky, hidden and embedded inside the feature run. Its expensive to implement and expensive to maintain, and it probably hides an underlying business problem (mentioned in the first solution)
So please use this solution with a great deal of caution. In general with cuking simple is best, and a bit of repetition is preferable to technical complexity to avoid repetition.
I have a question:) How does it work karate if it comes to parallel execution?
We use karate in specific way where under the hood we hava a bunch of java code. I'm curious if variables declared as ThreadLocal in our java code is really thread save?
Is each test run in separate thread or there is another way of running test simultaneously?
The problem we faced is that it looks like a least two of running tests have access to ThreadLocal variable which should be isolated from each other.
Could you explain?
If you use a ThreadLocal you are on your own :) The short answer is Karate will create a thread-pool for Scenario execution, and each Scenario can go onto any of these threads. Maybe you need to read this section on how you can force some Scenario-s to run in sequence - but you may still have problems if the ones that ran first do not "clean up".
EDIT: I also think it should NOT be possible for 2 Scenarios to be on the same thread at the same time and if you are seeing this - it is a bug, please help us replicate it (see last line of my answer).
Karate's multi threading is battle-tested and we would not be able to claim that you can do the Gatling integration if this was not the case. Maybe you should just trust Karate for doing all the heavy lifting you need ? For example look at "hooks": https://github.com/intuit/karate#hooks
And of course, if you really do think there's an issue, follow this process please: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
I have some features to test using Gherkin and Cucumber. The thing is that the execution is random, and since, for example, the first scenario is creating elements on the page, second one is looking for them and third moving them, all test are crashing cause the execution is going like: nº9 firts, then 8, then 2, then...
I am not using execution tags, or if I use them, I'm using it above "Feature:" to make sure all scenarios are running
Anyone could bring some light here?
General consensus within the test automation community is that your automated tests should be able to run independently. That is, tests should be runnable in any given order and the result of a test should not depend on the outcome of one or more previous tests. Try changing the architecture of your test cases.
It is possible to run tests in specific order using JUnit or TestNG.
https://www.ontestautomation.com/running-your-tests-in-a-specific-order/
I'm new to BDD and particularly Cucumber.
Can I get a features and its steps from a variable? Also, I want to get a feature and its steps from a test tracker (TestRail) before running tests by the special selection of this tests, and put it in a list, then one by one get a scenario and run it.
Is there such a possibility? Should I use Cucumber or another framework for this?
No, you can't define a Cucumber scenario in code (or at least not in a supported way). But if you were going to write code to get a scenario and its steps from your test tracker and run it, you could equally well write code to put the scenario and its steps in files and run the scenario with the cucumber executable.
I don't know of a Java testing framework in which you can define tests dynamically. You could do that in Ruby with RSpec or (less cleanly) minitest. But I don't know whether a Ruby test framework would be acceptable, or whether it would be OK for the people writing entries in your test tracker to have to read and/or write RSpec examples. (It seems strange to have Cucumber step definitions in a test tracker, too; having features in a test tracker seems more reasonable, aside from the question of how to run them.)
I am using Maven and TestNG.
How to distinguish at runtime when a particular method is being called by a TestNG/JUnit test-case or by the main java code
Several comments are alluding to this, however it's generally extremely poor practice to build in statements that work one way under test, and another way when the app is running standalone. This increases the probability that the app will pass tests, but fail in production.
Instead, you should look at why you're wanting to make this distinction. In general, it will be for the sake of some dependent object, or due to input of one variety or another. In these cases, it's better to engineer the class to accept dependent objects to be inserted into it via configuration and under test, the only thing that changes is the configuration. The class under test should not distinguish the dependant classes from one another. Instead, just work with their interfaces, so you can create mock classes for testing.
When accepting input, redirect the input source to take scripted input.
For databases, redirect to an in-memory DB which is configured for the test, etc.
You will find this approach will VASTLY improve the quality of the code you write, and decrease the probability of bugs sneaking past your unit tests.
At runtime, your code is never running unit tests. Unless you invoke them explicitly from your code, which you should never do.
Unit tests are only run manually, or during the test phase of the maven lifecycle.