Extract text and web links with the selenium WebDriver - java

I'm studying selenium and I want to extract the texts and links from Sympla's events, but when I click on the "more events" button, I can't extract the next events, it is always extracting the same initial events from the page.
Complete class for easy reproduction.
public static void main(String[] args) throws InterruptedException {
WebDriverManager.firefoxdriver().setup();
WebDriver driver = new FirefoxDriver();
driver.manage().window().maximize();
driver.get("https://www.sympla.com.br/eventos?ts=online_mais-de-3-mil-eventos-online");
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
// If have captcha, close the page and exit.
boolean captcha = driver.getPageSource().contains("Não sou um robô");
if (captcha == true) {
System.out.println("O Captcha apareceu, acabou a brincadeira!");
driver.close();
driver.quit();
}
// load more button
WebElement CarregarMais = driver.findElement(By
.xpath("//button[#id='more-events']"));
// Number of events counter
List<WebElement> eventos = (List<WebElement>) driver.findElements(By
.cssSelector("div.event-name.event-card"));
System.out.println("Number of links: " + eventos.size());
// Number of links counter
List<WebElement> eventos_link = (List<WebElement>) driver
.findElements(By.cssSelector("a.sympla-card.w-inline-block"));
// iterating over the button more events
for (int j = 0; j < eventos.size(); j++) {
CarregarMais.click();
#SuppressWarnings("deprecation")
WebDriverWait wait = new WebDriverWait(driver, 10);
WebElement element = wait.until(ExpectedConditions
.elementToBeClickable(By
.xpath("//button[#id='more-events']")));
// Iterating over event links
for (int i = 0; i < eventos_link.size(); i++) {
System.out.println(i + " " + eventos.get(i).getText() + " - "
+ eventos_link.get(i).getAttribute("href"));
Thread.sleep(500);
}
}
}

It's because you don't read the links again. With every click on the button a new page is created, so you need to read them again.
Furthermore you would need to store the last fetched link.
So after waiting for the button to be clickable again you need to reread eventos and eventos_link. And maybe you use a global variable like lastFetchedLinkIndex.
This would be my approach (adjusted your code):
WebDriverManager.firefoxdriver().setup();
WebDriver driver = new FirefoxDriver();
driver.manage().window().maximize();
driver.get("https://www.sympla.com.br/eventos?ts=online_mais-de-3-mil-eventos-online");
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
// If have captcha, close the page and exit.
boolean captcha = driver.getPageSource().contains("Não sou um robô");
if (captcha == true) {
System.out.println("O Captcha apareceu, acabou a brincadeira!");
driver.close();
driver.quit();
}
// load more button
WebElement CarregarMais = driver.findElement(By
.xpath("//button[#id='more-events']"));
// Number of events counter
List<WebElement> eventos = (List<WebElement>) driver.findElements(By
.cssSelector("div.event-name.event-card"));
System.out.println("Number of links: " + eventos.size());
// Number of links counter
List<WebElement> eventos_link = (List<WebElement>) driver
.findElements(By.cssSelector("a.sympla-card.w-inline-block"));
int lastEventScraped = 0;
// iterating over the button more events
for (int j = 0; j < eventos.size(); j++) {
CarregarMais.click();
#SuppressWarnings("deprecation")
WebDriverWait wait = new WebDriverWait(driver, 10);
WebElement element = wait.until(ExpectedConditions
.elementToBeClickable(By
.xpath("//button[#id='more-events']")));
eventos = (List<WebElement>) driver.findElements(By
.cssSelector("div.event-name.event-card"));
eventos_link = (List<WebElement>) driver
.findElements(By.cssSelector("a.sympla-card.w-inline-block"));
// Iterating over event links
for (int i = lastEventScraped; i < eventos_link.size(); i++, lastEventScraped++) {
System.out.println(i + " " + eventos.get(i).getText() + " - "
+ eventos_link.get(i).getAttribute("href"));
Thread.sleep(500);
}
}

Related

How to select multiple checkboxes in selenium(java)?

I am trying to select all checkboxes whose value is matched to my value.
Code is working fine when the web page has no vertical scroll. But if web page has some more data then the checkbox is not selected as I want.
Here is my code-
List<WebElement> rselect = tagdis1.findElements(By.className("row-selection-checkbox"));
System.out.println("Row selection Size- " + rselect.size());
List<WebElement> record = driver.findElements(By.id("$ctrl.item.id"));
System.out.println("Size- " + record.size());
int DocNameCount = 0;
for (int j = 0; j < record.size(); j++) {
String Pname = record.get(j).getText();
System.out.println("Pdf name- " + Pname);
if (Pname.equals(docName + ".pdf")) {
// here total 4 records i get but able to click only on 3 records
System.out.println(j + " " + Pname);
rselect.get(j).click();
Thread.sleep(2000);
}
}
Please use scroll option in your operation,
/*
* By
* scroll to the element and wait
*/
public void scroll(By element){
JavascriptExecutor js = (JavascriptExecutor) driver;
js.executeScript("arguments[0].scrollIntoView(true);",driver.findElement(element));
log.info("Scrolling down");
}

How to solve this problem using if-else loop in java selenium?

If variable value between j =<10 then page through error "Sorry ! user already..."
and then I try to assign another random value to variable k and fill up form again.
I want to execute driver.findElement(By.xpath("//*[contains(text(),'Sorry ! ')]")).isDisplayed() step only if I get error. Else just print success message.
Problem here is even if I get success message I'm getting element not found error due to driver.findElement(By.xpath("//*[contains(text(),'Sorry ! ')]")).isDisplayed().
I also tried implementing boolean Sorry_Error = driver.findElement(By.xpath("//*[contains(text(),'Sorry ! ')]")).isDisplayed(); but I think it is similar logic just different way of implementation. if there is no error then loop should go for else block but insted it searching .isDisplayed xpath and throwing error.
Isn't it correct way to handle user already exist error or is there any other logic.
This is success message
This is error message
package com.self.learn;
import java.util.Random;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.testng.annotations.Test;
public class testandquiz {
public static WebDriverWait wait;
#Test
public void Simple_Test() throws InterruptedException {
System.setProperty("webdriver.chrome.silentOutput", "true");
System.setProperty("webdriver.chrome.driver", "src/main/resources/chromedriver.exe");
WebDriver driver = new ChromeDriver();
wait = new WebDriverWait(driver, 10);
driver.manage().deleteAllCookies();
driver.get("https://www.testandquiz.com/user-registration");
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
((JavascriptExecutor) driver).executeScript("scroll(0,300)");
WebElement Fname = driver.findElement(By.id("first_name"));
WebElement Lname = driver.findElement(By.id("last_name"));
WebElement User_name = driver.findElement(By.id("user_name"));
WebElement email_address = driver.findElement(By.id("email_address"));
WebElement password = driver.findElement(By.id("password"));
WebElement re_password = driver.findElement(By.id("re_password"));
WebElement city = driver.findElement(By.id("city"));
WebElement state = driver.findElement(By.id("state"));
WebElement country = driver.findElement(By.id("country"));
WebElement pin_code = driver.findElement(By.id("pin_code"));
WebElement contact_no = driver.findElement(By.id("contact_no"));
Random randomGenerator = new Random();
int j = 10;
int k = 99999;
int i = randomGenerator.nextInt(j);
Fname.sendKeys("Test" + i + "");
Lname.sendKeys("Test" + i + "");
User_name.sendKeys("Test" + i + "");
email_address.sendKeys("Test" + i + "");
password.sendKeys("Test" + i + "");
re_password.sendKeys("Test" + i + "");
city.sendKeys("Test" + i + "");
state.sendKeys("Test" + i + "");
country.sendKeys("Test" + i + "");
pin_code.sendKeys("Test" + i + "");
contact_no.sendKeys("1234567890");
driver.findElement(By.xpath("//button[contains(text(),'Sign Up')]")).click();
if (driver.findElement(By.xpath("//*[contains(text(),'Sorry ! ')]")).isDisplayed()) {
WebElement Fname1 = driver.findElement(By.id("first_name"));
WebElement Lname1 = driver.findElement(By.id("last_name"));
WebElement User_name1 = driver.findElement(By.id("user_name"));
WebElement email_address1 = driver.findElement(By.id("email_address"));
WebElement password1 = driver.findElement(By.id("password"));
WebElement re_password1 = driver.findElement(By.id("re_password"));
WebElement city1 = driver.findElement(By.id("city"));
WebElement state1 = driver.findElement(By.id("state"));
WebElement country1 = driver.findElement(By.id("country"));
WebElement pin_code1 = driver.findElement(By.id("pin_code"));
WebElement contact_no1 = driver.findElement(By.id("contact_no"));
Fname1.clear();
Fname1.sendKeys("Test" + k + "");
Lname1.clear();
Lname1.sendKeys("Test" + k + "");
User_name1.clear();
User_name1.sendKeys("Test" + k + "");
email_address1.clear();
email_address1.sendKeys("Test" + k + "");
password1.clear();
password1.sendKeys("Test" + k + "");
re_password1.clear();
re_password1.sendKeys("Test" + k + "");
city1.clear();
city1.sendKeys("Test" + k + "");
state1.clear();
state1.sendKeys("Test" + k + "");
country1.clear();
country1.sendKeys("Test" + k + "");
pin_code1.clear();
pin_code1.sendKeys("Test" + k + "");
contact_no1.clear();
contact_no1.sendKeys("1234567890");
driver.findElement(By.xpath("//button[contains(text(),'Sign Up')]")).click();
}
else {
String Message = driver.findElement(By.xpath("//div[contains(text(),'verification mail')]")).getText();
System.out.println(Message);
}
}
}
I don't know how to implement this logic.
You won't necessarily need a wait here... depends on if javascript updates the DOM. If it's after a standard page load, then just use:
List<WebElement> elements = driver.findElements(By.xpath("//*[contains(text(), 'Sorry')]");
if (elements.size() > 0)
{
// sorry found
}
else
{
// sorry not found
}
Else use try/catch with a wait:
try {
WebDriverWait wait = new WebDriverWait(driver, 5);
WebElement element = wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//*[contains(text(), 'Sorry')]"));
// sorry found
}
catch (Exception ex)
{
//sorry not found
}
Use same logic for error text.

how to write multiple web table with same header into same excel sheet using selenium?

i am trying to write multiple web table(on each page i.e pagination) into single excel sheet having same header..and able to write 9 rows of first table , but for second instance index need to start from 10, ..how to do that?..below are the sample code.. any help will be appreciated..thanks in advance..
though i am getting output on console..
public class DataScraper {
public static void main(String[] args) throws InterruptedException {
WebDriver driver = new ChromeDriver();
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.get("https://etrain.info/in?PAGE=LIST--RAJ-TRAINS--1");
// *[#id="lowerdata"]/table/tbody/tr[1]/td/table/tbody/tr[5]/td[1]/a
// *[#id="lowerdata"]/table/tbody/tr[1]/td/table/tbody/tr[6]/td[1]/a
// *[#id="lowerdata"]/table/tbody/tr[1]/td/table/tbody/tr[29]/td[1]/a
String beforeXpath_TrainNoRow = "//td[#id='content']//tr[";
String afterXpath_TrainNoRow = "]//td[1]";
// div[#id='sublowerdata']//tr[3]//td[1]
// div[#id='sublowerdata']//tr[11]//td[1]
String beforeXpath_No = "//div[#id='sublowerdata']//tr[";
String afterXpath_No = "]//td[1]";
// int rowCount = TrainList.size();
Xls_Reader reader = new Xls_Reader(
"C:\\Selenium_Automation\\Projects\\DataDriven_FW\\src\\com\\testdata\\TrainSchedule.xlsx");
if (!reader.isSheetExist("Rajdhani")) {
reader.addSheet("Rajdhani");
reader.addColumn("Rajdhani", "IslNo");
// reader.addColumn("TrainSearch", "TrainName");
}
for (int i = 5; i <= 30; i++) {
String actualXpath_TrainNoRow = beforeXpath_TrainNoRow + i + afterXpath_TrainNoRow;
WebElement TrainNo = driver.findElement(By.xpath(actualXpath_TrainNoRow));
TrainNo.click();
Thread.sleep(5000);
List<WebElement> rows = driver.findElements(By.xpath("//table[#id='schtbl']//tr"));
System.out.println("total rows is " + rows.size());
int rowCount = rows.size();
for (int j = 3; j <= rowCount - 1; j++) {
String actualXpath_No = beforeXpath_No + j + afterXpath_No;
String SrNo = driver.findElement(By.xpath(actualXpath_No)).getText();
int islNo = Integer.parseInt(SrNo);
System.out.println(islNo);
reader.setCellData("Rajdhani", "IslNo", j, SrNo);
}
driver.navigate().back();
driver.navigate().refresh();
// table[#id='schtbl']//tr -- table rows inner
}
}
}
expected output should be from 1 to 9 in each rows appending next instance output i.e 1 2..9 1 2..9 butt i am getting only 1 to 9 for first instance.

3rd page is not click able in pagination

I have used below code for pagination, 2nd page is able to click but next pages are not able to click
List<WebElement> pagination = driver.findElements(By.tagName("i"));
List<WebElement> pagination1 = driver.findElements(By.xpath(".//[#id='ctl00_cpHFooter_PageUC1_rptrPager_ctl05_lnkNext']/i"));
pagination.size();
System.out.println("Total pages :" + pagination.size());
if(pagination .size()>0)
{
System.out.println("pagination exists");
//click on pagination link
for(int i=0; i<pagination1.size(); i++)
{
pagination1.get(i).click();
JavascriptExecutor jse1 = (JavascriptExecutor)driver;
jse1.executeScript("scroll(0, 2000);");
}
}
else
{
System.out.println("pagination not exists");
}
You have to start from page 2 and use some wait or sleep()
List<WebElement> pagination = driver.findElements(By.tagName("i"));
List<WebElement> pagination1 = driver.findElements(By.xpath(".//[#id='ctl00_cpHFooter_PageUC1_rptrPager_ctl05_lnkNext']/i"));
pagination.size();
System.out.println("Total pages :" + pagination.size());
if(pagination .size()>0)
{
System.out.println("pagination exists");
//click on pagination link
for(int i=2; i<pagination1.size(); i++)
{
pagination1.get(i).click();
Thread.sleep(5000);
JavascriptExecutor jse1 = (JavascriptExecutor)driver;
jse1.executeScript("scroll(0, 2000);");
}
}
else
{
System.out.println("pagination not exists");
}

After selecting the second option from dropdown, the WebElement is still showing the record of first option

There is a dropdown list where each selection has a different URL under the dropdown buttons. Suppose when I select first option then it shows 10 hyperlink and select the second option it shows 5 hyperlinks, etc.
Problem - When I select the second option, it is still showing 10 hyperlinks instead of 5 and shows
org.openqa.selenium.StaleElementReferenceException: Element not found
in the cache - perhaps the page has changed since it was looked up
Select select = new Select(selectdropdown);
List<WebElement> options = select.getOptions();
int isize = options.size();
for (int i = 0; i < isize; i++)
{
String value = select.getOptions().get(i).getText();
driver.manage().timeouts().implicitlyWait(100, TimeUnit.SECONDS);
WebElement WebElementer = driver.findElement(By.xpath("//*[#id='content-inner']"));
List<WebElement> elementList = new ArrayList<>();
elementList = WebElementer.findElements(By.cssSelector("a[href]"));
System.out.println("Total number of links found" + elementList.size());
System.out.println("to check wheather link is working or not");
for (WebElement element : elementList)
{
try
{
System.out.println("URL: " + element.getAttribute("href").trim() + " returned "
+ islinkBroken(new URL(element.getAttribute("href").trim())));
}
catch (Exception exp)
{
System.out.println("At " + element.getAttribute("innerHTML")
+ " Exception occured -> " + exp.getMessage());
}
}
}
where you selecting the element ?? (C# syntax example)
IList<IWebElement> accountsDDL = driver.FindElements(By.XPath("//select[#id='yourSelectId']/option"));
for (int i = 1; i < accountsDDL.Count; i++)
{
new SelectElement(driver.FindElement(By.Name("yourSelectId"))).SelectByText(accountsDDL[i].Text); // Selecting the element
}
In java
I spent a little time cleaning up your code and added a few things. See if this works. As Leon said, I think one of the issues was that you didn't have code that actually changed the selected option.
Select select = new Select(selectdropdown);
for (int i = 0; i < select.getOptions().size(); i++)
{
select.selectByIndex(i); // you were missing this line?
// String value = select.getFirstSelectedOption().getText(); // this variable is never used
// driver.manage().timeouts().implicitlyWait(100, TimeUnit.SECONDS); // this doesn't do what you think it does
// I think this next line should work. I combined the two locators into one.
List<WebElement> elementList = driver.findElements(By.cssSelector("#content-inner a[href]"));
System.out.println("Total number of links found" + elementList.size());
System.out.println("to check wheather link is working or not");
for (WebElement element : elementList)
{
try
{
String href = element.getAttribute("href").trim();
System.out.println("URL: " + href + " returned " + islinkBroken(new URL(href)));
}
catch (Exception exp)
{
System.out.println("At " + element.getAttribute("innerHTML") + " Exception occured -> " + exp.getMessage());
}
}
}
Suggestion: It might be useful to you to add the selected option text to your exception message.

Categories

Resources