Develop Reference

How to load lazy content on Linkedin search page using selenium

Summary
I am trying to scrape all first connections' profile links of an account on LinkedIn search page. But since the page loads the rest of the content dynamically (as you scroll down) I can not get the 'Next' page button which is at the end of the page.
Problem description
https://linkedin.com/search/results/people/?facetGeoRegion=["tr%3A0"]&facetNetwork=["F"]&origin=FACETED_SEARCH&page=YOUR_PAGE_NUMBER
I can navigate to the search page using selenium and the link above. I want to know how many pages there are to navigate them all just changing the page= variable of the link above.
To implement that I wanted to check for the existence of Next button. As long as there is next button I would request the next page for scraping. But if you do not scroll down till the bottom of the page -which is where the 'Next' button is- you can not find the Next button nor you can find the information about other profiles because they are not loaded yet.
Here is how it looks when you do not scroll down and take a screenshot of the whole page using firefox screenshot tool.
How I implemented
I can fix this by hard coding a scroll down action into my code and making the driver wait for visibilityOfElementLocated. But I was wondering whether there is any other way better than my approach. And if by the approach the driver can not find the Next button somehow the program exits with the exit code 1.
And when I inspect the requests when I scroll down the page, it is just requests for images and etc as you can see below. I couldn't figure out how the page loads more info about profiles as I scroll down the page.
Source code
Here is how I implemented it in my code. This app is just a simple implementation which is trying to find the Next button on the page.
package com.andreyuhai;
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class App
{
WebDriver driver;
public static void main( String[] args )
{
Bot bot = new Bot("firefox", false, false, 0, 0, null, null, null);
int pagination = 1;
bot.get("https://linkedin.com");
if(bot.attemptLogin("username", "pw")){
bot.get("https://www.linkedin.com/" +
"search/results/people/?facetGeoRegion=" +
"[\"tr%3A0\"]&origin=FACETED_SEARCH&page=" + pagination);
JavascriptExecutor js = (JavascriptExecutor) bot.driver;
js.executeScript("scrollBy(0, 2500)");
WebDriverWait wait = new WebDriverWait(bot.driver, 10);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//button[#class='next']/div[#class='next-text']")));
WebElement nextButton = bot.driver.findElement(By.xpath("//button[#class='next']/div[#class='next-text']"));
if(nextButton != null ) {
System.out.println("Next Button found");
nextButton.click();
}else {
System.out.println("Next Button not found");
}
}
}
}
Another tool for that which I wonder about : LinkedIn Spider
There is this chrome extension called linkedIn Spider
This also does exactly what I am trying to achieve but using JavaScript I guess, I am not sure. But when I run this extension on the same search page. This does not do any scrolling down or loading other pages one by one extract the data.
So my questions are:
Could you please explain me how LinkedIn achieves this? I mean how does it load profile information as I scroll down if not making any request or etc. I really don't know about this. I would appreciate any source links or explanations.
Do you have any better (faster I mean) idea to implement what I am trying to implement?
Could you please explain me how LinkedIn Spider could be working without scrolling down and etc.

I have checked the div structure and the way linkedin is showing the results. So, if you hit the url directly and check the by following xpath: //li[contains(#class,'search-result')] You would find out that all the results are already loaded on the page, but linkedin are showing only 5 results in one go and on scrolling, it shows the next 5 results, however all the results are already loaded on the page and can be found out by the mentioned xpath.
Refer to this image which highlights the div structure and results when you find the results on entering the xpath on hitting the url: https://imgur.com/Owu4NPh and
Refer to this image which highlights the div structure and results after scrolling the page to the bottom and then finding the results using the same xpath: https://imgur.com/7WNR830
You could see the result set is same however there is an additional search-result__occlusion-hint part in the < li > tag in the last 5 results and through this linkedin is hiding the next 5 results and showing only the first 5 results on the first go.
Now comes the implementation part, i have checked "Next" button comes only when you scroll through whole results on the page, so instead of scrolling to a definite coordinates because that can be changed for different screensizes and windows, you can take the results in a list of webelement and get it's size and then scroll to the last element of that list. In this case, if there are total 10 results then the page will be scrolled to the 10th results and if there are only 4 results then the page will be scrolled to the 4th result and after scrolling you can check if the Next button is present on the page or not. For this, you can check the list size of the "Next" button web element list, if the list size is greater than 0, it means the next button is present on the page and if its not greater than 0, that means the Next button is not present on the list and you can stop your execution there.
So to implement it, i have taken a boolean which has an initial value as true and the code will be run in a loop till that boolean becomes false and it will become false when the Next button list size becomes equal to 0.
Please refer to the code below:
public class App
{
WebDriver driver;
// For initialising javascript executor
public Object executeScript(String script, Object... args) {
JavascriptExecutor exe = (JavascriptExecutor) driver;
return exe.executeScript(script, args);
}
// Method for scrolling to the element
public void scrollToElement(WebElement element) {
executeScript("window.scrollTo(arguments[0],arguments[1])", element.getLocation().x, element.getLocation().y);
}
public static void main(String[] args) {
// You can change the driver to bot according to your usecase
driver = new FirefoxDriver();
// Add your direct URL here and perform the login after that, if necessary
driver.get(url);
// Wait for the URL to load completely
Thread.sleep(10000);
// Initialising the boolean
boolean nextButtonPresent = true;
while (nextButtonPresent) {
// Fetching the results on the page by the xpath
List<WebElement> results = driver.findElements(By.xpath("//li[contains(#class,'search-result')]"));
// Scrolling to the last element in the list
scrollToElement(results.get(results.size() - 1));
Thread.sleep(2000);
// Checking if next button is present on the page
List<WebElement> nextButton = driver.findElements(By.xpath("//button[#class='next']"));
if (nextButton.size() > 0) {
// If yes then clicking on it
nextButton.get(0).click();
Thread.sleep(10000);
} else {
// Else setting the boolean as false
nextButtonPresent = false;
System.out.println("Next button is not present, so ending the script");
}
}
}
}

What I have observed was, the content is already loaded in the page and it will be displayed to us when we do the scroll down.
But if we inspect the 'Next >' button by loading the page manually by using the class name 'next' for example like below,
//button[#class='next']
we cannot locate it until we do the scroll down because it is not visible to us. But by using the below XPath, we can identify all the profile links count irrespective of whether they are displayed or not?
//h3[contains(#class, 'search-results__total')]/parent::div/ul/li
As you want to fetch all the profile links from the page, we can use the above XPath help to do that. We will get the links count using the above XPath then we will scroll into each element view at a time and then we will fetch the profile links on the way like below :
// Identifying the all the profile links
List<WebElement> totalProfileLinks = driver.findElements(By.xpath("//h3[contains(#class, 'search-results__total')]/parent::div/ul/li"));
// Looping for getting the profile link
for(int i=1;i<totalProfileLinks.size();i++) {
// Scrolling so that it will be visible
((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", totalProfileLinks.get(i));
// Fetching the anchor node
final WebElement link = driver.findElement(By.xpath("(//h3[contains(#class, 'search-results__total')]/parent::div/ul/li//div[contains(#class, 'search-result__info')]//a)["+i+"]"));
// Avoiding the StaleElementReferenceException
new FluentWait<WebDriver>(driver).withTimeout(1, TimeUnit.MINUTES).pollingEvery(1, TimeUnit.SECONDS).ignoring(StaleElementReferenceException.class).until(new Function<WebDriver, WebElement>() {
public WebElement apply(WebDriver arg0) {
return link;
}
});
// Fetching and printing the link from anchor node
System.out.println(link.getAttribute("href").trim());
}
So, if we want to click on the 'Next >' button first we need to check if its present or not(As we have scrolled while fetching the profile links, the 'next' button also get displyed). We can use the help of `driver.findElements();` method to get the matches of that element count and can store it in some List(Because it returns List of WebElements) like below :
List<WebElement> nextButton = driver.findElements(By.className("next"));
The benefit of using the above technique is, the script won't fail if there are no element matches also and we will have an empty list if there are no matches.
Then we can use the size() method of the List interface to get the matches count like below :
int size = nextButton.size();
And if the size is more than 0 then that element is present otherwise not, we can check that condition like below :
if(size > 0) {
nextButton.get(0).click(); // Do some operation like clicking on it
System.out.println("=> 'Next >' button is there and clicked on it...");
} else {
System.out.println("=> 'Next >' button is NOT there...");
}
As the content is loaded and the element is view-able, we will use the JavaScriptExecutor which will locate and clicks on it.
Wrap the above code in while loop and check the presence of the 'Next >' button every time after clicking on the previous 'Next >' button like below :
boolean next = true;
while(next) {
// Checking 'Next >' button is there or not in the page
List<WebElement> nextButton = driver.findElements(By.className("next"));
// If the 'Next >' button is there then clicking on it otherwise stopping the execution
if(nextButton.size() > 0) {
doClickUsingJSE(nextButton.get(0));
System.out.println("=> 'Next >' button is there and clicked on it...");
} else {
next = false;
System.out.println("=> 'Next >' button is NOT there so stopping the execution...");
}
Thread.sleep(1000);
}
Loop will break if the 'if' condition fails in the above code because 'next' will becomes 'false'. And if we use the Fluent Wait then it will help us in avoiding some 'Exceptions' like 'WebDriverException' and 'StaleElementReferenceException'. So I have written one separate method which will wait for an element by avoiding some exceptions and clicks on it if the conditions get satisfied.
Check the code below :
private static void doClickUsingJSE(final WebElement element) {
// Using the Fluent Wait to avoid some exceptions like WebDriverException and StaleElementReferenceException
Wait<WebDriver> wait = new FluentWait<WebDriver>(driver).withTimeout(1, TimeUnit.MINUTES).pollingEvery(1, TimeUnit.SECONDS).ignoring(WebDriverException.class, StaleElementReferenceException.class);
WebElement waitedElement = wait.until(new Function<WebDriver, WebElement>() {
public WebElement apply(WebDriver driver) {
return element;
}
});
wait.until(ExpectedConditions.visibilityOf(waitedElement));
wait.until(ExpectedConditions.elementToBeClickable(waitedElement));
// Clicking on the particular element using the JavaScriptExcecutor
((JavascriptExecutor) driver).executeScript("arguments[0].click();", waitedElement);
}
As I mentioned about the JavaScriptExecutor earlier, I have included use of that also in the above method only.
Try the below end to end working code :
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.Keys;
import org.openqa.selenium.StaleElementReferenceException;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebDriverException;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.FluentWait;
import org.openqa.selenium.support.ui.Wait;
import com.google.common.base.Function;
public class BasePage
{
// Declaring WebDriver
private static WebDriver driver;
private static void doClickUsingJSE(final WebElement element) {
// Using the Fluent Wait to avoid some exceptions like WebDriverException and StaleElementReferenceException
Wait<WebDriver> wait = new FluentWait<WebDriver>(driver).withTimeout(1, TimeUnit.MINUTES).pollingEvery(1, TimeUnit.SECONDS).ignoring(WebDriverException.class, StaleElementReferenceException.class);
WebElement waitedElement = wait.until(new Function<WebDriver, WebElement>() {
public WebElement apply(WebDriver driver) {
return element;
}
});
wait.until(ExpectedConditions.visibilityOf(waitedElement));
wait.until(ExpectedConditions.elementToBeClickable(waitedElement));
// Clicking on the particular element using the JavaScriptExcecutor
((JavascriptExecutor) driver).executeScript("arguments[0].click();", waitedElement);
}
public static void main( String[] args ) throws Exception
{
System.setProperty("webdriver.chrome.driver", "C:\\NotBackedUp\\chromedriver.exe");
// Initializing the Chrome Driver
driver = new ChromeDriver();
// Launching the LinkedIn site
driver.get("https://linkedin.com/search/results/people/?facetGeoRegion=[\"tr%3A0\"]&facetNetwork=[\"F\"]&origin=FACETED_SEARCH&page=YOUR_PAGE_NUMBER");
// You can avoid this and it to your convience way
// As there are no connections in my page, I have used like this
//------------------------------------------------------------------------------------
// Switching to the login from - iframe involved
driver.switchTo().frame(driver.findElement(By.className("authentication-iframe")));
// Clicking on the Sign In button
doClickUsingJSE(driver.findElement(By.xpath("//a[text()='Sign in']")));
// Entering the User Name
WebElement element = driver.findElement(By.id("username"));
doClickUsingJSE(element);
element.sendKeys("something#gmail.com");
// Entering the Password
element = driver.findElement(By.id("password"));
doClickUsingJSE(element);
element.sendKeys("anything"+Keys.ENTER);
// Clicking on the People drop down
Thread.sleep(8000);
element = driver.findElement(By.xpath("//span[text()='People']"));
doClickUsingJSE(element);
// Selecting the All option
Thread.sleep(2000);
element = driver.findElement(By.xpath("//ul[#class='list-style-none']/li[1]"));
element.click();
// Searching something in the LinkedIn search box
Thread.sleep(3000);
element = driver.findElement(By.xpath("//input[#role='combobox']"));
doClickUsingJSE(element);
element.sendKeys("a"+Keys.ENTER);
Thread.sleep(8000);
//------------------------------------------------------------------------------------
boolean next = true;
while(next) {
// Identifying the all the profile links
List<WebElement> totalProfileLinks = driver.findElements(By.xpath("//h3[contains(#class, 'search-results__total')]/parent::div/ul/li"));
// Looping for getting the profile link
for(int i=1;i<totalProfileLinks.size();i++) {
// Scrolling so that it will be visible
((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", totalProfileLinks.get(i));
// Fetching the anchor node
final WebElement link = driver.findElement(By.xpath("(//h3[contains(#class, 'search-results__total')]/parent::div/ul/li//div[contains(#class, 'search-result__info')]//a)["+i+"]"));
// Avoiding the StaleElementReferenceException
new FluentWait<WebDriver>(driver).withTimeout(1, TimeUnit.MINUTES).pollingEvery(1, TimeUnit.SECONDS).ignoring(StaleElementReferenceException.class).until(new Function<WebDriver, WebElement>() {
public WebElement apply(WebDriver arg0) {
return link;
}
});
// Fetching and printing the link from anchor node
System.out.println(link.getAttribute("href").trim());
}
// Checking 'Next >' button is there or not in the page
List<WebElement> nextButton = driver.findElements(By.className("next"));
// If the 'Next >' button is there then clicking on it otherwise stopping the execution
if(nextButton.size() > 0) {
doClickUsingJSE(nextButton.get(0));
System.out.println("=> 'Next >' button is there and clicked on it...");
} else {
next = false;
System.out.println("=> 'Next >' button is NOT there so stopping the execution...");
}
Thread.sleep(1000);
}
}
}
I hope it helps... Happy Coding...

Scrolling to a WebElement and clicking on it

I am new to automation and am practicing on the flipkart website.
On the page:
http://www.flipkart.com/mobiles/pr?sid=tyy,4io&otracker=clp_mobiles_CategoryLinksModule_0-2_catergorylinks_11_ViewAll
... when I try to click an element that is not in view of the page by scrolling to it, I get the exception: Element is not clickable
Below is the code:
WebElement mobile = driver.findElement(By.xpath ("//a[#title='Apple iPhone 6S (Silver, 128 GB) ']"));
JavascriptExecutor jse = (JavascriptExecutor) driver;
jse.executeScript("arguments[0].scrollIntoView();", mobile);
mobile.click();
I believe this issue is occurring because of the header available in flipkart: even though the window is getting scrolled to that particular element, the header is covering the element so it's not possible to click on it.
Can anyone help resolve this?

you can try like this
Case where you want to click on a element that is not in view of the page (without scrolling) try below
public static void main(String[] args) throws InterruptedException {
WebDriver driver = new FirefoxDriver();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.get(
"http://www.flipkart.com/mobiles/pr?sid=tyy,4io&otracker=clp_mobiles_CategoryLinksModule_0-2_catergorylinks_11_ViewAll");
driver.manage().window().maximize();
// Take everything on the page in list first .
List<WebElement> completecalContent = driver.findElements(By.xpath("//*[#class='fk-display-block']"));
System.out.println(completecalContent.size());
// printing all elements
for (int i = 0; i < completecalContent.size(); i++) {
System.out.println("Print complete Content : " + completecalContent.get(i).getText());
if (completecalContent.get(i).getText().equals("Apple iPhone 5S (Space Grey, 16 GB)")) {
// move to a specific element
((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView();",
completecalContent.get(completecalContent.size() - 1));
// move slightly up as blue header comes in the picture
((JavascriptExecutor) driver).executeScript("window.scrollBy(0,-100)");
// then click on the element
completecalContent.get(i).click();
}
}
}
Case where you want to scroll then in that case update above code with these lines.
A. if you want to scroll to the bottom of the page then
((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight)");
B. if u want to scroll to a specific element then try this
WebElement element = driver.findElement(By.xpath("xpath to element"));
((JavascriptExecutor) driver).executeScript(
"arguments[0].scrollIntoView();", element);
C. if you want to scroll on the basis of coordinates then try this
((JavascriptExecutor) driver).executeScript("window.scrollBy(0,500)");

Instead of scrolling up to web element you can try scrolling to little bit down in page like
JavascriptExecutor jse = (JavascriptExecutor)driver;
jse.executeScript("scroll(250, 0)"); //x value '250' can be altered
Else you can try scrolling to element which is good enough above to required element. It means in code you tried instead of taking required webelement just scroll upto web element above the required so that the header does not cover required element.
Thank You,
Murali

Hey if you are not certain about the element's position on the page, you can find the co-ordinates at run time and then execute your text.
You can get elements co-ordinate by using Point
Point point = element.getLocation();
int xcord = point.getX();
int ycord = point.getY();
You can also get the dimensions of a webelement like its Height and Width using Dimension
Once you have the x and y co-ordinates and you have its dimensions. You can write your code to scroll till that particular co-ordinates on the page.
Hope it helps!

Selenium tests running too fast, not waiting for the radio button to be selected

I have a Selenium Grid and WebDriver 2.48.2 test that runs too fast. The majority of the time the test stops because a radio button isn't selected prior to a button being pressed.
The radio buttons are setup using JavaScript based on a JSON file to create any number of them within a section on the fly. Once the continue button is clicked that section is destroyed and a new one created with new radio buttons.
I tried an implicit wait with no success.
driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);
The only solution that is working for me is a delay to allow enough time for the radio button to be clicked.
driver.findElement(By.id("radiobuttonid")).click();
Thread.sleep(delay);
But in my opinion this isn't an ideal solution, it's possible the delay may not be long enough or that it's too long, wasting time; there could be any number of radio buttons so the time will increase exponentially.
I've tried setting up various explicit waits with different expected conditions but with no success.
I've tried waiting for the radio button to be created since it may not exist (presenceOfElementLocated & elementToBeClickable). I've tried waiting for it to be selected (elementToBeSelected).
I'm having trouble finding exactly what the expected conditions are supposed to do since the descriptions are brief and open to misinterpretation.
Ideally, I want the test to continue soon after the radio button is clicked. If possible, what's the best way to do this?
EDIT
L.Bar's suggestion below didn't work for me, but it was very helpful to determine that the radio buttons exist, just that the continue button is being clicked before the radio button has a chance to be selected.
EDIT 2
This is just to expand on Jeremiah correct answer. I put the code into a method to make it reusable.
private static Predicate<WebDriver> forceSelectionOfElement (final String id)
{
return new Predicate<WebDriver>()
{
#Override
public boolean apply(WebDriver arg0)
{
WebElement element = arg0.findElement(By.id(id));
boolean rval = element.isSelected();
if (!rval) {
element.click();
}
return rval;
}
};
}
Usage
WebDriverWait wait = new WebDriverWait(driver, 5);
wait.until(forceSelectionOfElement("q___01_02_01___1___a"));
Remember to import this namespace, it took me an embarrassingly long time to figure out I imported the wrong one :)
import com.google.common.base.Predicate;

I've had a similar problem, but in my case my 'click' event was lost somewhere. My test would proceed, the click code would trigger, but the state of the element would never visibly change and the test would fail. I ended up leveraging the looping behavior of the WebDriverWait by adding a custom predicate that is a bit more persistent about the selected state.
WebDriverWait wait = new WebDriverWait(driver, 5);
final By lookup = By.id("radio");
LOG.debug("Wait for radio to be clickable.");
wait.until(ExpectedConditions.elementToBeClickable(lookup)); //I assume this implies visibility.
LOG.debug("Element clickable. Proceeding into click behavior.");
wait.until(new Predicate<WebDriver>() {
#Override
public boolean apply(WebDriver arg0) {
LOG.debug("Resolving 'myId' Element");
WebElement radio = arg0.findElement(lookup);
boolean rval = radio.isSelected();
LOG.debug("Element Resolved and has a state of " + (rval ? "selected" : "not selected"));
if (!rval) {
LOG.debug("Clicking on the Element!");
radio.click();
}
//If we return false we will loop. So the first time through we let this click the radio for us.
//Second time through we should find that the element is clicked. If it's not we click it again until it represents the state we're wanting.
return rval;;
}});
The custom predicate will
Resolve the WebElement
Capture the current Selected status
Click the WebElement if it is not selected
Return the current selected status.
So this should not allow the test to proceed until the clicked state of the element is what I want it to be. So far it's done basically what I've wanted.
It's also worth noting that Thread.sleep is seen as not the best practice for selenium interactions. This is what the ExplicitWait concept is meant to account for. I've linked one of the latest topics in which I'm aware of this being discussed.
Thread.sleep works but implicit wait, webdriverwait and fluent wait does not?
Best of Luck!

My way to do it:
public bool IsElementPresent(By by, IWebDriver driver)
{
try
{
driver.FindElement(by);
return true;
}
catch (NoSuchElementException)
{
return false;
}
}
public bool IsElementPresent(By by, IWebDriver driver, int sec)
{
bool itemExist = false;
itemExist = IsElementPresent(by, driver);
while (!itemExist && sec >= 0)
{
Thread.Sleep(1000);
itemExist = IsElementPresent(by, driver);
sec--;
}
if (sec == -1)
return false;
else
return true;
}
Calling to test if exist
//Checking if element exist 10 sec
if(IsElementPresented(By.Id("radiobuttonid"),driver,10))
{
//element exist click on it
}
else
{
//didn't exist.
}

You stated that you explored ExpectedConditions, have you tried ExpectedConditions.elementToBeSelected()? You should be able to detect if the radio button is selected and then move forward. Try something like this
WebDriverWait wait = new WebDriverWait(driver, 10);
WebElement radio = driver.findElement(By.id("radiobuttonid"));
radio.click();
wait.until(ExpectedConditions.invisibilityOfElementLocated(By.id("ID of message element")));
// do stuff
EDIT
Changed code after more info from OP

How to handle StaleElementReferenceException

I have a scenario where I am attempting to loop through a number of elements on a bar chart, until I find the "rect" tag name. When I click on the "rect" tag name, the individual bar is selected from the chart, and I am redirect to another page. Please see below for an image of the bar chart I am working with:
http://imgur.com/xU63X1Z
For reference, the bar chart I am working with is the top right-hand side. The test I want to execute is to click on the first bar in the chart; doing so, will redirect me to an appropriate page. In order to do this I have written the following code in Selenium Webdriver using Eclipse (Java):
WebElement deliveredChartDailyFocus = driver.findElement(By.id("delivered-chart-daily"));
deliveredChartDailyFocus.click();
List<WebElement> children = deliveredChartDailyFocus.findElements(By.tagName("rect"));
Iterator<WebElement> iter = children.iterator();
while (iter.hasNext()) {
WebElement we = iter.next();
if(we.isDisplayed()){
we.click();
}
Everything appears to work well in that the above code hits the “rect” element and redirects me to the appropriate page. However, when I hit the page I then get an error as the code is still looking for the “rect” value which isn’t on the new page.
You’ll notice that there is a “break” line missing from the above…..this is because, upon debugging the code I found that when iterating through the loop, the click event does not kick in until the 3rd iteration, I’m assuming because the “rect” element is not visible? Therefore, if I put in a “break” statement it exits out of the loop after the first iteration, and therefore I never get to the part where I carry out the “click” event to navigate to the new page.
Essentially, what I’m after is a way of being able to loop as many times as necessary until the appropriate “rect” element can be located. Upon clicking on that, I am redirected to the new page….only at that point do I want the loop to exit, so that the “NoSuchElementException error is not displayed.
If any more details are required please let me know, would really appreciate any guidance on this.

Once you're on the new page all those rect elements are gone. Exercising any references to those rect elements will trigger this StaleElementReferenceException.
So don't reference those elements after the click. Iterate to the first displayed rect element then stop iterating.
WebElement deliveredChartDailyFocus = driver.findElement(By.id("delivered-chart-daily"));
deliveredChartDailyFocus.click();
// Get a list of all the <rect> elements under the #delivered-chart-daily element
List<WebElement> children = deliveredChartDailyFocus.findElements(By.tagName("rect"));
WebElement elementToClick = null; // variable for the element we want to click on
for (WebElement we : children) // loop through all our <rect> elements
{
if (we.isDisplayed())
{
elementToClick = we; // save the <rect> element to our variable
break; // stop iterating
}
}
if (elementToClick != null) // check we have a visible <rect> element
{
elementToClick.click();
}
else
{
// Handle case if no displayed rect elements were found
}

Andy, the issue here is DOM refresh. You can not simply get the collection of IWebElements and iterate through and click back and forth. You can find the count of the elements and every time you come to the page find the element you want to click dynamically. See this for implementation
public void ClickThroughLinks()
{
_driver.Navigate().GoToUrl("http://www.cnn.com/");
//Maximize the window so that the list can be gathered successfully.
_driver.Manage().Window.Maximize();
//find the list
By xPath = By.XPath("//h2[.='The Latest']/../li//a");
var linkCollection = _driver.FindElements(xPath);
for (int i = 0; i < linkCollection.Count; i++)
{
//wait for the elements to be exist
new WebDriverWait(_driver, TimeSpan.FromSeconds(10)).Until(ExpectedConditions.ElementExists(xPath));
//Click on the elements by index
if (i<=3)
{
_driver.FindElements(xPath)[i].Click();
}
else
{
break;
}
_driver.Navigate().Back();
_driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(10));
}
}

How to scroll using jscroll pane in webdriver java

An element not coming into view as the pane remains constant. I believe if I scroll down the pane maybe it will solve that issue?
So basically I want to scroll down a div to an element so as to get into visibility.
What piece of code would I need to add to my webdriver java?
I tried with the following without success:
WebDriver driver = null;
JavascriptExecutor js = (JavascriptExecutor) driver;
js.executeScript("$('#fed-panel').data('jsp').scrollToBottom()");

I use the following in my code to deal with some Ajax objects that appear only when one scrolls down. You can replace the 'this.objectExists' function below (which is something I implemented) with ObjectVisible.
public void scrollDown(String identifier, String m){
Boolean readyStateComplete = false;
int i=0;
while (!readyStateComplete) {
JavascriptExecutor js = (JavascriptExecutor)driver;
js.executeScript("window.scrollTo(0,Math.max(document.documentElement.scrollHeight," +
"document.body.scrollHeight,document.documentElement.clientHeight));");
covlog.logIssue(CovLogger.DEBUG, "scrollDown","Scrolling for the " + ++i + " times");
String tmp = js.executeScript("return document.readyState").toString();
readyStateComplete = tmp.contentEquals("complete");
//In case the object doesn't exists, break out of the loop
//the object might represent a spinner that shows that more
// records are downloaded.
if ((i%24)==0){ //24 is page size
if (!this.objectExists(identifier, m)){
readyStateComplete = true;
}
}
}
}