Context:
Currently I'm working with a Selenium based system in Java that runs tests using JUnit and Maven. Through the Maven Surefire plugin, I'm able to run tests in parallel. I have ensured the following things -
ThreadLocal is used for singleton objects
Separate WebDriver per thread
Using explicit waits (e.g. (new WebDriverWait(webdriver, timeout)).until(ExpectedConditions.________(_____)); )
Problem:
However, when running tests in parallel, I'm getting TimeoutExceptions at WebDriverWait explicit waits. This can occur at any places in the test that use explicit waits. These timeout exceptions do not occur when the tests are running sequentially.
Question:
I would like to know whether any of you have encountered this situation and how you go about solving this problem. Other relevant information and feedback are welcomed too.
Thanks in advance! If you need any supplemented resources such as sample code, I'm happy to provide.
Firstly I am not sure how to properly use multi-thread with JUnit, last time I tried I had no success, anyway, I have had better results with TestNG. Other that that, things are similar to yours, basically from maven (surefire) I am calling testng.xml, reference.
Now, webdriver, out-of-the-box, is not thread-safe. Threads can get mixed-up and all kind of "near-to-impossible-to-debug" stuff can happen. Anyway, lately WebDriver people have tried to tackle this problem and we now have ThreadGuard class available (source). That, according to docs:
Multithreaded client code should use this to assert that it accesses
webdriver in a thread-safe manner.
So in your case you can simply use it like (from top of my head, sorry for typos):
ThreadLocal<WebDriver> driverStore = new ThreadLocal<>();
WebDriver driver = ThreadGuard.protect(new FirefoxDriver());
driverStore.set(driver);
I have had success using this setup.
Related
I have a question:) How does it work karate if it comes to parallel execution?
We use karate in specific way where under the hood we hava a bunch of java code. I'm curious if variables declared as ThreadLocal in our java code is really thread save?
Is each test run in separate thread or there is another way of running test simultaneously?
The problem we faced is that it looks like a least two of running tests have access to ThreadLocal variable which should be isolated from each other.
Could you explain?
If you use a ThreadLocal you are on your own :) The short answer is Karate will create a thread-pool for Scenario execution, and each Scenario can go onto any of these threads. Maybe you need to read this section on how you can force some Scenario-s to run in sequence - but you may still have problems if the ones that ran first do not "clean up".
EDIT: I also think it should NOT be possible for 2 Scenarios to be on the same thread at the same time and if you are seeing this - it is a bug, please help us replicate it (see last line of my answer).
Karate's multi threading is battle-tested and we would not be able to claim that you can do the Gatling integration if this was not the case. Maybe you should just trust Karate for doing all the heavy lifting you need ? For example look at "hooks": https://github.com/intuit/karate#hooks
And of course, if you really do think there's an issue, follow this process please: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
I'm refactoring an automation tests framework implemented with Java, Selenium, TestNG and Maven. I'm currently trying to reduce the execution time of a test case which takes more than one hour for completing.
Reviewing the code I realized that I could execute some steps concurrently in order to reduce execution time. Although I know that WebDriver class isn't thread safe, therefore I'd like to open three new instances of the web browser with the same previously page loaded in order to collect the required data concurrently.
Question
How could I achieve this?
I created a java wrapper to feed jmeter. I have implemented java classes with selenium that are invoked by the wrapper and perform GUI tests.
I activated the headless option.
launching tests with a single user from jmeter all works correctly.
trying to launch two users tests fail.
can you help me to understand why?
Most probably you missed an important bit: each Selenium session need to have a separate URL and Selenium server needs to be running on a different port. So make sure to amend your "wrapper" to be aware of multiple WebDriver instances and to kick off a separate instance of Selenium server (or standalone client) for each JMeter thread (virtual user).
Unfortunately we cannot help further without seeing your code, just keep in mind that your wrapper needs to be thread-safe. Also pay attention to jmeter.log file - normally it should contain enough information to get to the bottom of your test failure.
P.S. Are you aware of WebDriver Sampler plugin? It's designed in line with JMeter threads model and you should be able to kick off as many browsers as your machine can handle. If you for some reason it doesn't fit your needs you can at least take a look into the source code to get an idea with regards to what you need to change in your "wrapper"
I have developed a micro-framework in Java which does the following function:
All test cases list will be in a MS-Access database along with test data for the Application to be tested
I have created multiple classes and each having multiple methods with-in them. Each of these methods represent a test-case.
My framework will read the list of test cases marked for execution from Access and dynamically decide which class/method to execute based on reflection.
The framework has methods for sendkeys, click and all other generic methods. It takes care of reporting in Excel.
All this works fine without any issue.
Now I am looking to run the test cases across multiple machines using Grid. I read in many sites that we need a framework like TestNG to have this in Grid. But I hope that it could be possible to integrate Grid in my own framework. I have read many articles and e-books which does not explain the coding logic for this.
I will be using only windows 7 with IE. I don't need cross browser/os testing.
I can make any changes to the framework to accomplish this. So please feel free.
In the Access DB which I mentioned above, I will have details about test case and the machine in which the test case should run. Currently users can select the test cases they want to run locally in the Access DB and run it.
How will my methods(test scripts) know which machine its going to be executed? What kind of code changes I should do apart from using RemoteWebDriver and Capabilities?
Please let me know if you need any more information on my code or have any question. Aslo kindly correct me if any of my understanding on Grid is wrong.
How will my methods know which machine it is going to be executed? - You just need to know one machine with a grid setup - the ip of your hub machine. The hub machine will decide where to send the request to from the nodes that are registered with, depending upon the capabilities you specify while instantiating the driver. When you initialize the RemoteWebDriver instance, you need to specify the host (ip of your hub). I would suggest to keep the hub ip as a configurable property.
The real use of the grid is for parallel remote execution. So how do you make your tests run in parallel is a thing that you need to decide. You can use a framework like Testng which provides parallelism with simple settings. You might need to restructure your tests to accomodate testng. The other option would be to implement multithreading yourself to trigger your tests in parallel. I would recommend testng based on my experience since it provides many more capabilities apart from parallelism. You need to take care that each instance of driver is specific to your thread and not a global variable.
All tests can hit the hub and the hub can take care of the rest.
It is important to remember that Grid does not execute your tests in parallel for you. It is the job of your framework to divide tests across multiple threads and collate the results . It is also key to realise that when running on Grid, the test script still executes in the machine the test was started on. Grid provides a REST API to open and interact with browsers, so your test will be using this rather than opening a browser locally. Any other non-selenium code will be executed within the context of the original machine not machine where the browser has been opened (e.g. File System access is not where the browser has opened). Any use of static classes and globals in your framework may also cause issues as each test will acces these concurrently. Your code must be thread safe.
Hopefully this hasn't put you off using Grid. It is an awesome tool and really easy to use. It is the parallel execute which is hard and frameworks such as TestNG provide this out of the box.
Good luck with your framework.
I am a developer temporarily tasked with helping the QA team with test automation using JUnit 4 and Selenium WebDriver. I am new to testing and test automation.
Looking at various WebDriver examples, a common pattern is to instantiate an implementation of WebDriver (like FirefoxWebDriver) in the #Before method, use the instance in #Test method to interface with the browser and driver.quit() in #After.
So if there are 5 #Test methods, the browser will be opened, test app initialized and browser closed 5 times.
My question is why open, initialize and close required for each test case? My guess is to prevent one test case failure from having a negative side effect on other tests. Is this correct? Are there any other reasons?
Great Question!
Coming from the OTHER side, i'm a QA automation test engineer, working with the Dev team on automation...
As #Prateek and myself have pointed out, there are a couple reasons why.
The two most obvious:
The main purpose of testing is to localize the error. If all the tests would run in one go and it fails, you would be unable to tell where it failed. ~Prateek
Multithreading is a reason why as well. Picture your company having thousands of regression tests that need to be ran. Having each test isolated, makes it so they are completely independent of other tests, making you able to potentionally run them all at the same time (assuming your app can handle it) which .. say each test takes 1 minute to run, 1x1000=1000 minutes to run each test sequentially as opposed to finishing them all in just 1 minute!
So you are correct in your assumtion here -
My guess is to prevent one test case failure from having a negative side effect on other tests. Is this correct?
But of course, there are more reasons than that alone.
There is one more reason, each time WebDriver is instantiated, a new profile is created which clears browser cache..Currently there is no other way to clear browser cache..