Develop Reference

Selenium WebDriver: How to make multithreaded method openNewTab (WindowType.TAB) using static driver?

I'm writing search engine. For control http errors I use Selenium WebDriver. Created a class HTTPExceptoinController for this:
public final class HTTPExceptoinController {
private static HTTPExceptoinController instance;
private static String originalWindow;
private static ChromeOptions options;
private static WebDriver driver;
private HTTPExceptoinController() {}
public static synchronized HTTPExceptoinController getInstance() {
if (instance==null) {
instance = new HTTPExceptoinController();
setController();
}
return instance;
}
In the same class I made a method "openNewTab". How can I make each thread close its own tab without synchronization?
public synchronized String openNewTab(String link) {
driver.switchTo().newWindow(WindowType.TAB);
String page=null;
try {
driver.get(link);
String currentwindow = driver.getWindowHandle();
driver.switchTo().window(currentwindow);
WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(3));
wait.until(ExpectedConditions.visibilityOfElementLocated(By.tagName("a")));
page = driver.getPageSource();
driver.manage().deleteAllCookies();
driver.close();
} catch (Exception e) {
driver.close();
driver.switchTo().window(originalWindow);
}
driver.switchTo().window(originalWindow);
return page;
}
I tried to do this with ConcurrentHashMap; put in "mapWindow" WindowHandle and System.currentTimeMillis(), but it didn't work.
private static void stopwatch() {
Thread worker = new Thread(new Runnable() {
public void run() {
while(Status.isRun()) {
Set<String> allWindows = driver.getWindowHandles();
Iterator<String> i = allWindows.iterator();
while(i.hasNext()){
String childwindow = i.next();
if(mapWindow.containsKey(childwindow)){
long timeout = mapWindow.get(childwindow);
timeout=timeout+5000;
if(System.currentTimeMillis()>=timeout) {
mapWindow.remove(childwindow);
driver.switchTo().window(childwindow);
((JavascriptExecutor) driver).executeScript("window.stop;");
driver.close();
}
}
try { Thread.sleep(1000);}
catch (InterruptedException e) {}
}
}}});
worker.start();
}

Selenium: How to avoid StaleElementReferenceException when looping through a set of elements?

I have a page that contains a bunch of tables. I loop through the tables in the outer loop and then loop through each row in the table in the inner loop. It all works fine. But some of the pages have a Next button. When I add code to click that after completing the page, then I start getting StaleElementReferenceException while looping through the rows of a table.
Here is the code:
WebDriverWait wait1 = new WebDriverWait(driver, 10000);
WebElement maxPage = null;
WebElement auctionsWaitingDiv = driver.findElement(By.cssSelector("div[class='Head_W']"));
if (auctionsWaitingDiv.isDisplayed() == false) return properties;
try {
maxPage = wait1.until(ExpectedConditions.visibilityOfElementLocated(By.id("maxWA")));
} catch (TimeoutException ex) {
return properties;
}
Integer maxPageNo = 1;
if (!maxPage.getText().isEmpty())
maxPageNo = Integer.parseInt(maxPage.getText());
for (int i = 1; i <= maxPageNo; i++) {
driver.findElement(By.cssSelector("div[id='Area_W']")); //only look at Auctions Waiting section
WebDriverWait wait2 = new WebDriverWait(driver, 10000);
List<WebElement> tables = null;
try {
tables = wait2.until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.cssSelector("table[class='ad_tab']")));
} catch (TimeoutException ex) {
System.out.println("table not found in allotted time");
return properties;
} catch (StaleElementReferenceException ex) {
System.out.println("returning due to StaleElementReferenceException");
return properties;
}
for (WebElement table: tables) {
List<String> propAttributes = new ArrayList<>();
// StaleElementReferenceException: The element reference of
// <table class="ad_tab"> is stale; either the element is no
// longer attached to the DOM, it is not in the current
// frame context, or the document has been refreshed
List<WebElement> rows = table.findElements(By.cssSelector("tr"));
String parcelLink = "";
for (WebElement row : rows) {
WebElement key = row.findElement(By.cssSelector("th"));
WebElement val = row.findElement(By.cssSelector("td"));
String keyVal = key.getText() + val.getText();
propAttributes.add(keyVal);
if (key.getText().equals("Parcel ID:")) {
WebElement a = val.findElement(By.cssSelector("a"));
parcelLink = a.getAttribute("href");
}
}
}
driver.findElement(By.xpath(".//*[#class='PageRight']")).click(); //click the "Next" button
}
What I don't understand is why the stale element is happening at all? The page is not changing during the loop and I've waited until all elements have been fetched. How to avoid the StaleElementReferenceException?
Edit: The last stack trace shows it is happening in this line:
List<WebElement> rows = table.findElements(By.cssSelector("tr"));
and the error message above it shows:
SEVERE: null
org.openqa.selenium.StaleElementReferenceException: The element reference of <table class="ad_tab"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed

The StaleElementReferenceException is thrown whenever you want to access an element reference which is not available anymore. This happens when the element is no longer attached to the DOM or if the page was updated.
The solution for this is just searching for the element again whenever this happens.
You could adapt all your tests or page objects. Or you write your own RobustWebDriver and RobustWebElement which refreshes the element if a SERE is thrown.
RobustWebDriver:
public class RobustWebDriver implements WebDriver {
private WebDriver originalWebDriver;
public RobustWebDriver(WebDriver webDriver) {
this.originalWebDriver = webDriver;
}
#Override
public void get(String url) {
this.originalWebDriver.get(url);
}
#Override
public String getCurrentUrl() {
return this.originalWebDriver.getCurrentUrl();
}
#Override
public String getTitle() {
return this.originalWebDriver.getTitle();
}
#Override
public List<WebElement> findElements(By by) {
List<WebElement> elements = new ArrayList<>();
for (WebElement element : this.originalWebDriver.findElements(by)) {
elements.add(new RobustWebElement(element, by, this));
}
return elements;
}
#Override
public WebElement findElement(By by) {
return new RobustWebElement(this.originalWebDriver.findElement(by), by, this);
}
#Override
public String getPageSource() {
return this.originalWebDriver.getPageSource();
}
#Override
public void close() {
this.originalWebDriver.close();
}
#Override
public void quit() {
this.originalWebDriver.quit();
}
#Override
public Set<String> getWindowHandles() {
return this.originalWebDriver.getWindowHandles();
}
#Override
public String getWindowHandle() {
return this.originalWebDriver.getWindowHandle();
}
#Override
public TargetLocator switchTo() {
return this.originalWebDriver.switchTo();
}
#Override
public Navigation navigate() {
return this.originalWebDriver.navigate();
}
#Override
public Options manage() {
return this.originalWebDriver.manage();
}
}
RobustWebElement:
public class RobustWebElement implements WebElement {
private WebElement originalElement;
private RobustWebDriver driver;
private By by;
private static final int MAX_RETRIES = 10;
public RobustWebElement(WebElement element, By by, RobustWebDriver driver) {
this.originalElement = element;
this.by = by;
this.driver = driver;
}
#Override
public void click() {
int retries = 0;
while (retries < MAX_RETRIES) {
try {
this.originalElement.click();
return;
} catch (StaleElementReferenceException ex) {
refreshElement();
}
retries++;
}
throw new StaleElementReferenceException(
String.format("Element is still stale after %s retries.", MAX_RETRIES));
}
#Override
public void sendKeys(CharSequence... keysToSend) {
int retries = 0;
while (retries < MAX_RETRIES) {
try {
this.originalElement.sendKeys(keysToSend);
return;
} catch (StaleElementReferenceException ex) {
refreshElement();
}
retries++;
}
throw new StaleElementReferenceException(
String.format("Element is still stale after %s retries.", MAX_RETRIES));
}
// TODO add other unimplemented methods with similar logic.
private void refreshElement() {
this.originalElement = driver.findElement(by);
}
And then you just need to wrap your WebDriver into the RobustWebDriver and you are ready to go:
WebDriver driver = new RobustWebDriver(new ChromeDriver());
EDIT:
Of course you need to take care of scrolling up and down by yourself.

Well after tearing my hair out for a day, I finally realized what was happening. It should have been obvious to me. When the "Next" button is clicked, it takes some time for the new page to load. By simply adding a delay, the new DOM is loaded and processing begins on it, not again on the previous one!
driver.findElement(By.xpath(".//*[#class='PageRight']")).click();
try {
Thread.sleep(4000); //provide some time for the page to load before processing it
} catch (InterruptedException ex) {
Logger.getLogger(RealAuction.class.getName()).log(Level.SEVERE, null, ex);
}
Now it runs to completion with no StaleElementReferenceException.

Proper way to pass a webdriver to another class

I want to pass my WebDriver to another class instead of passing it to the individual methods within that class. That would mean passing it to the constructor of the class when I create an instance of it. Here is my code, and my issue further below -
public class StepDefinitions{
public static WebDriver driver = null;
CustomWaits waits;
#Before("#setup")
public void setUp() {
driver = utilities.DriverFactory.createDriver(browserType);
System.out.println("# StepDefinitions.setUp(), driver = " + driver);
waits = new CustomWaits(driver);
}
}
public class CustomWaits {
WebDriver driver;
public CustomWaits(WebDriver driver){
this.driver = driver;
}
public boolean explicitWaitMethod(String id) {
boolean status = false;
try {
WebDriverWait wait = new WebDriverWait(driver, 30);
WebElement element = wait.until(ExpectedConditions.visibilityOfElementLocated(By.id(id)));
status = element.isDisplayed();
} catch (NullPointerException e){
e.printStackTrace();
}
return status;
}
}
The error I am getting in is NullPointerException when a method of that class is called within an #Given, #When, etc. This is a scope issue I cannot resolve.
Feature File:
#test
Feature: Test
#setup
Scenario: Navigate to Webpage and Assert Page Title
Given I am on the "google" Website
Then page title is "google"
Here is the step definition:
#Given("^element with id \"([^\"]*)\" is displayed$")
public void element_is_displayed(String link) throws Throwable {
if (waits.explicitWaitMethod(link)) {
// This is where waits becomes null when I put a breakpoint
driver.findElement(By.id(link)).isDisplayed();
} else {
System.out.println("Timed out waiting for element to display");
}
}

I would do something like this.
public class StepDefinitions{
public StepDefinitions() {
driver = utilities.DriverFactory.createDriver(browserType);
System.out.println("# StepDefinitions.setUp(), driver = " + driver);
waits = new CustomWaits(driver);
}
public static WebDriver driver = null;
public static CustomWaits waits;
#Given("^element with id \"([^\"]*)\" is displayed$")
public void element_is_displayed(String link) throws Throwable {
if (waits.explicitWaitMethod(link)) {
// This is where waits becomes null when I put a breakpoint
driver.findElement(By.id(link)).isDisplayed();
} else {
System.out.println("Timed out waiting for element to display");
}
}
}
public class CustomWaits {
private static WebDriver driver;
public CustomWaits(WebDriver driver){
this.driver = driver;
}
public boolean explicitWaitMethod(String id) {
boolean status = false;
try {
WebDriverWait wait = new WebDriverWait(driver, 30);
WebElement element = wait.until(ExpectedConditions.visibilityOfElementLocated(By.id(id)));
status = element.isDisplayed();
} catch (NullPointerException e){
e.printStackTrace();
}
return status;
}
}

Java Spring, waiting for AsyncHandlers

apologies for the basic question; I'm new to the Java world and the spring framework. I've built a little example application that makes a bunch of async requests to an external service and returns a list of the responses ('metrics'), but I need to make my application wait until all the responses have come back. Right now I have a (don't hate me) Thread.sleep while I let the results come back, but obviously this is very nasty. Can anyone suggest a better way of architecting this?
Calling class:
#Service
public class MetricService {
#Autowired
private MetricProcessor processor;
private LinkedBlockingQueue<Metric> queue;
#Scheduled(fixedDelay = 60000)
public void queryExternalService() {
List<Metrics> metrics = new ArrayList<>();
metrics = processor.getMetrics();
//this is horrible and I'm a horrible human being
try {
Thread.sleep(10000); //wait for the requests to come back
}
catch (Exception e) {
e.printStackTrace();
}
queue.addAll(metrics);
}
}
Class:
#Component
public class MetricProcessor {
#Autowired
private AsyncClient externalClient;
public List<Metrics> getMetrics() {
List<Metrics> returnObj = new Arraylist<>();
for(Blah blah : bleh) {
Request request = new Request("abc");
externalClient.getMetricAsync(request, new AsyncHandler<request, result>() {
#Override
public void onError(Exception e) {
System.out.println("Error");
}
#Override
public void onSuccess(Request request, Result result) {
returnObj.add(new Metric(result.getKey(), result.getValue()));
}
});
}
return returnObj;
}
}
Any help would be greatly appreciated!

Try a Future.
In MetricService:
public void queryExternalService() {
Future<List<Metrics>> metricsFuture = processor.getMetrics();
try {
queue.addAll(metricsFuture.get(60, TimeUnit.SECONDS));
} catch (InterruptedException | ExecutionException | TimeoutException e) {
e.printStackTrace();
}
}
So notice instead of the desired List, your processor provides a reference to a Future which may fulfil that request later:
public Future<List<Metrics>> getMetrics() {
MetricsFuture metricsFuture = new MetricsFuture();
// Need to ask for the metrics to be built
metricsFuture.buildMetrics();
return metricsFuture;
}
private static class MetricsFuture extends AbstractFuture<List<Metrics>> {
// Assuming the requests are asynchronous, this should be a thread-safe list
List<Metrics> returnObj = new CopyOnWriteArrayList<>();
void buildMetrics() {
for(Blah blah : bleh) {
final Request request = new Request("abc");
externalClient.getMetricAsync(request, new AsyncHandler<request, result>() {
#Override
public void onError(Exception e) {
onError(request, e);
}
#Override
public void onSuccess(Request request, Result result) {
addMetrics(new Metrics(result.getKey(), result.getValue()));
}
});
}
}
void onError(Request request, Exception e) {
// Is any error a total failure? This allows us to terminate waiting
setException(e); // alternative we could remove request or keep a list of errors
System.out.println("Error");
}
void addMetrics(Metrics metric) {
returnObj.add(metric);
// Once we have received the expected number of results we can pass that prepare that
// as a result of this future.
if(returnObj.size() == bleh.size()) {
set(returnObj);
}
}
}

Unexpected behaviour with Java timer task

My Timer task is not functioning as it's supposed to. I have scheduled it to repeat a specific task every 3 seconds but this is not happening.
As per Java documentations:
schedule(TimerTask task, long delay,long period) .
Schedules the specified task for repeated fixed-delay execution, beginning after the specified delay. Subsequent executions take place at approximately regular intervals separated by the specified period.
public class Tester {
public static void main(String[] args) {
log.info("Schedule task");
Timer time = new Timer();
TesterClient tc = new TesterClient();
time.schedule(tc, 0, 3000);
}
}
public class TesterClient extends TimerTask {
public void init() {
System.out.println("New Task!!!!");
}
#Override
public void run() {
init();
}
}
And yet i only get one "New Task!!!!" printed in console
Am i missing something here?
Thanks
Update:
I will try to paste in here every piece of code that is relevant and goes from top to bottom in terms of execution.
Start:
public class Tester {
public static Logger log = Logger.getLogger("com.orderlysoftware.orderlycalls.manager.ManagerClient");
public static Timer time = new Timer();
public static void main(String[] args) {
log.info("Creating service");
Service.serviceInit();
log.info("Initializing TesterClient for scheduled task");
TesterClient tc = new TesterClient();
time.schedule(tc, 0, 3000);
}
public static ManagerSettings managerSettings() {
ManagerSettings managerSettings = new ManagerSettings();
managerSettings.setName("managerClient");
managerSettings.setHost("77.237.251.152");
managerSettings.setPort(5038);
managerSettings.setUsername("orderlystats");
managerSettings.setPassword("orderlystats");
return managerSettings;
}
}
Service class method:
static ExecutorService executorService;
{
serviceInit();
}
//public static ClassLoader loader;
public static void serviceInit(){
if(executorService!=null) {
return;
}
executorService= Executors.newCachedThreadPool();
try {
ThreadPoolExecutor tpe=(ThreadPoolExecutor)executorService;
tpe.setMaximumPoolSize(100000);
} catch (Exception ex) {
System.out.println(ex);
}
}
package com.orderlysoftware.testing;
import java.io.IOException;
import java.util.TimerTask;
import java.util.concurrent.ExecutorService;
import java.util.logging.Logger;
import com.orderlysoftware.orderlycalls.OrderlyCalls;
import com.orderlysoftware.orderlycalls.asterisk.manager.ManagerClient;
import com.orderlysoftware.orderlycalls.asterisk.manager.action.ManagerResponse;
import com.orderlysoftware.orderlycalls.asterisk.manager.event.ManagerEvent;
import com.orderlysoftware.orderlycalls.asterisk.manager.event.ManagerEventListener;
import com.orderlysoftware.orderlycalls.base.Service;
public class TesterClient extends TimerTask {
public static Logger log = Logger.getLogger("com.orderlysoftware.orderlycalls.manager.ManagerClient");
public static ExecutorService es = Service.getExecutorService();
public ManagerClient mc;
public void init() {
log.info("run check method to see if Manager Client is running");
boolean running = check();
log.info("checker status is : " + running);
while(running) {
try {
Thread.sleep(3000);
startCall();
} catch (InterruptedException e) {
log.info("Sleep interrupted");
}
}
}
public boolean check() {
log.info("ManagerClient is: " + mc);
if(mc == null) {
log.info("Initialize service");
mc = (ManagerClient)OrderlyCalls.createService(ManagerClient.class, Tester.managerSettings());
log.info("Initialize ManagerClient");
mc.init();
log.info("Service created. ManagerClient initialized : "+ mc);
}
if(!mc.isConnected()) {
log.info("ManagerClient is not connected");
return false;
}
log.info("Check if ManagerClient is connected AND running");
if(mc.isConnected() && !mc.isRunning()) {
log.info("Manager Client is connected but NOT running");
return false;
}
if(mc.isConnected() && mc.isRunning()) {
log.info("ManagerClient is connected and running");
return true;
}
return false;
}
private void startCall() {
log.info("Adding listener to the call");
addListenerToCall(mc);
int testID = 0;
ManagerResponse response = null;
try {
response = mc.originate("Local/1001#main", "1001", "main", "1", null, null, 2500, "1002", "testID=" + (testID++), "1", true);
log.info("Manager response is: " + response);
if(response == null) {
mc.shutdown();
throw new IOException("Null response for originate.");
}
if(!response.getValue("Response").equals("Success")) {
mc.shutdown();
throw new IOException("Originate returned " + response.getValue("Response") + ": " + response.getValue("Message"));
}
} catch (IOException e) {
log.info("IO Exception" + e.toString());
}
}
public void addListenerToCall(ManagerClient mc) {
try {
// Add event listener
log.info("Adding ManagerEventListener to ManagerClient: " + mc);
mc.addManagerEventListener(new ManagerEventListener() {
#Override
public void handleManagerEvent(ManagerEvent event) {
if("OriginateResponse".equals(event.getType())) {
handleOriginateResponse(event);
}
}
});
} catch (IOException e) {
log.info("IO Exception : " + e);
}
}
protected void handleOriginateResponse(ManagerEvent event) {
try {
// do something here
Thread.sleep(1000);
} catch (InterruptedException e) {
log.info("sleep interupted" + e);
}
}
#Override
public void run() {
log.info("New Task!!!!!!!!!!");
init();
}
}

It works for me - but I suspect the problem is that you're letting the Timer get garbage collected:
After the last live reference to a Timer object goes away and all outstanding tasks have completed execution, the timer's task execution thread terminates gracefully (and becomes subject to garbage collection). However, this can take arbitrarily long to occur.
As noted in comments, I believe that "outstanding tasks" means "tasks that have already been started" rather than just "ones which would be scheduled". The docs are unclear, however, and I may be mistaken.
If you prevent garbage collection (e.g. by keeping a reference to the Timer in a static variable) then I think you'll see it keep going forever...

Your program works fine for me too. The issue got reproduced with following change in your program:
import java.util.*;
public class Tester {
public static void main(String[] args) {
System.out.println("Schedule task");
Timer time = new Timer();
TesterClient tc = new TesterClient();
time.schedule(tc, 0, 3000);
}
}
class TesterClient extends TimerTask {
public void init() {
System.out.println("New Task!!!!");
}
#Override
public void run() {
init();
this.cancel(); //-------This causes hang in execution after printing once
}
}
But, not sure what could have caused this to happen without cancel() in your program.