Develop Reference

How to scrape selected table columns and write them in CVS in Java Selenium

My object is to scrape data by using Java Selenium. I am able to load selenium driver, connect to the website and fetch the first column then go to the next pagination button until its become disable and write it to the console. Here is what I did so far:
public static WebDriver driver;
public static void main(String[] args) throws Exception {
System.setProperty("webdriver.chrome.driver", "E:\\eclipse-workspace\\package-name\\src\\working\\selenium\\driver\\chromedriver.exe");
System.setProperty("webdriver.chrome.silentOutput", "true");
driver = new ChromeDriver();
driver.get("https://datatables.net/examples/basic_init/zero_configuration.html");
driver.manage().window().maximize();
compareDispalyedRowCountToActualRowCount();
}
public static void compareDispalyedRowCountToActualRowCount() throws Exception {
try {
Thread.sleep(5000);
List<WebElement> namesElements = driver.findElements(By.cssSelector("#example>tbody>tr>td:nth-child(1)"));
System.out.println("size of names elements : " + namesElements.size());
List<String> names = new ArrayList<String>();
//Adding column1 elements to the list
for (WebElement nameEle : namesElements) {
names.add(nameEle.getText());
}
//Displaying the list elements on console
for (WebElement s : namesElements) {
System.out.println(s.getText());
}
//locating next button
String nextButtonClass = driver.findElement(By.id("example_next")).getAttribute("class");
//traversing through the table until the last button and adding names to the list defined about
while (!nextButtonClass.contains("disabled")) {
driver.findElement(By.id("example_next")).click();
Thread.sleep(1000);
namesElements = driver.findElements(By.cssSelector("#example>tbody>tr>td:nth-child(1)"));
for (WebElement nameEle : namesElements) {
names.add(nameEle.getText());
}
nextButtonClass = driver.findElement(By.id("example_next")).getAttribute("class");
}
//printing the whole list elements
for (String name : names) {
System.out.println(name);
}
//counting the size of the list
int actualCount = names.size();
System.out.println("Total number of names :" + actualCount);
//locating displayed count
String displayedCountString = driver.findElement(By.id("example_info")).getText().split(" ")[5];
int displayedCount = Integer.parseInt(displayedCountString);
System.out.println("Total Number of Displayed Names count:" + displayedCount);
Thread.sleep(1000);
// Actual count calculated Vs Dispalyed Count
if (actualCount == displayedCount) {
System.out.println("Actual row count = Displayed row Count");
} else {
System.out.println("Actual row count != Displayed row Count");
throw new Exception("Actual row count != Displayed row Count");
}
} catch (Exception e) {
e.printStackTrace();
}
}
I want to:
scrape more than one column or may be selected columns for example on this LINK name, office and age column
Then want to write these columns data in CSV file
Update
I tried like this but not running:
for(WebElement trElement : tr_collection){
int col_num=1;
List<WebElement> td_collection = trElement.findElements(
By.xpath("//*[#id=\"example\"]/tbody/tr[rown_num]/td[col_num]")
);
for(WebElement tdElement : td_collection){
rows += tdElement.getText()+"\t";
col_num++;
}
rows = rows + "\n";
row_num++;
}

Scraping:
Usually when I want to gather list elements I will select by Xpath instead of CssSelector. The structure of how to access elements through the Xpath is usually more clear, and depends on one or two integer values specifying the element.
So for your example where you want to find the names, you would find an element by the Xpath, the next element in the list's Xpath, and find the differing value:
The first name, 'Airi Satou' is found at the following Xpath:
//*[#id="example"]/tbody/tr[1]/td[1]
Airi's position has the following Xpath:
//*[#id="example"]/tbody/tr[1]/td[2]
You can see that across rows the Xpath for each piece of information differs on the 'td' markup.
The next name in the list, 'Angela Ramos' is found:
//*[#id="example"]/tbody/tr[2]/td[1]
And Angela's position is found:
//*[#id="example"]/tbody/tr[2]/td[2]
You can see that the difference in the column is controlled by the 'tr' markup.
By iterating over values of 'tr' and 'td' you can get the whole table.
As for writing to a CSV, there are a some solid Java libraries for writing to CSVs. I think a straightforward example to follow is here:
Java - Writing strings to a CSV file
UPDATE:
#User169 It looks like you're gathering a list of elements for each row in the table. You want to gather the Xpaths one by one, iterating over the list of webElements that you found originally. Try this, then add to it so it will get text and save it to an array.
for (int num_row = 1; num_row < total_rows; num_row++){
for (int num_col = 1; num_col < total_col; num_col++){
webElement info = driver.findElement(By.xpath("//*[#id=\"example\"]/tbody/tr[" + row_num + ']/td[' + col_num + "]");
}
}
I haven't tested it so it may need a few small changes.

selenium logic to select 5th element of a dropdown and check for duplicate element is present in the same drop down or not

Select s = new Select(driver.findElement(By.id("<< id exp>>")));
s.selectByIndex(5);
this is the piece of code to find 5th element of a dropdown ,what about next

Using this you can get all the duplicate options from the dropdown.
List<WebElement> options = new Select(driver.findElement(By.id("-id-"))).getOptions();
for(WebElement e: options) {
System.out.println(e.getText());
}
Set<String> optionNames = new HashSet<String>(options.size());
for(WebElement element: options) {
if(optionNames.add(element.getText())==false) {
System.out.println("Duplicate option:" +element);
}
else {
System.out.println("No duplicate options found");
}
}

Shadow Root - click in a href under several shadow roots

I have a list of links inside of several shadowRoots. Already solved this problem.
public WebElement expandRootElement(WebElement element) {
WebElement ele = (WebElement) ((JavascriptExecutor) driver).executeScript("return arguments[0].shadowRoot",element);
return ele;
}
WebElement root5_adminPanel = shadowRoot4_MduiContainerChild2.findElement(By.cssSelector("#layout > border-layout > ng-view > admin-panel"));
WebElement shadowRoot5_AdminPanel= expandRootElement(root5_adminPanel);
WebElement root6_breadCrumb = shadowRoot5_AdminPanel.findElement(By.cssSelector("#layout > border-layout > breadcrumb"));
WebElement shadowRoot6_breadCrumb = expandRootElement(root6_breadCrumb);
WebElement root6_domainPanel = shadowRoot5_AdminPanel.findElement(By.cssSelector("#layout > border-layout > ng-view > gdsr-domain-panel"));
WebElement shadowRoot6_domainPanel = expandRootElement(root6_domainPanel);
WebElement root7_selectDomain = shadowRoot6_domainPanel.findElement(By.cssSelector("#domainContainer > domain-panel-item.ng-binding.last"));
WebElement shadowRoot7_selectDomain = expandRootElement(root7_selectDomain);
When I reach this shadowRoot7, I have a list of items with the same name, which I already created a List to fix it.
List<WebElement> rows_table = shadowRoot6_domainPanel.findElements(By.cssSelector("#domainContainer > domain-panel-item:nth-child(n)"));
(They are around 45 items)
This will select all of them, in this case all the domain-panel-item rows.
My problem is that each domain-panel-item still contain another shadowRoot (the same path for all of them) an i would like to select a random item, not the first or last one, for example, the item number 43.
enter image description here
My solution was this one but it doesn't work because it doesnt access to the link that i want:
public void clickSelectedDomain(String domain) {
List<WebElement> rows_table = shadowRoot6_domainPanel.findElements(By.cssSelector("#domainContainer > gdsr-domain-panel-item:nth-child(n)"));
int rows_count = rows_table.size();
for (int row=0; row<rows_count; row++) {
if(rows_table.get(row).getAttribute("href").contains(domain)) {
rows_table.get(row).click();
}
}
}
Some have an idea how to fix this?

You solved the problem by calling recursively executeScript() in order to get the imbricated Shadow DOMs but actually you could have just called executeScript() once, and inside got the Shadow DOMs successively.
driver.executeScript( function ()
{
var root1 = document.querySelector( 'selector string 1' ).shadowRoot
var root2 = root1.querySelector( 'selector string 2' ).shadowRoot
var root3 = root2.querySelector( 'selector string 3' ).shadowRoot
...
return foundElement
}
Anyways, in the for() {} loop, you should extract the ultimate Shadow DOM one last time, and then select the <a> element to check its content.

I am getting StaleElementReferenceException: element is not attached to the page document

HTMLCODE
I am getting StaleElementReferenceException: element is not attached to the page document. I went through some of the solutions that are already there in StackOverflow. It did not work and it continues to throw the same error. Here is the code I am using which is throwing the stale reference error
WebElement table2 = driver.findElement(By.cssSelector("body > div:nth-child(74) > div.sp-palette-container"));
List<WebElement> allrows2 = table2.findElements(By.tagName("div"));
for(WebElement row2: allrows2){
List<WebElement> cells = row2.findElements(By.tagName("span"));
for(WebElement cell:cells){
if (cell.getAttribute("title").equals("rgb(0, 158, 236)")) {
cell.click();
}
}
}

Because clicking the found cell lead some HTML changes on the current page , due to this changes selenium will treat the page(after click) is an "new" page (even though not redirect to another page actually).
In the next iteration of the loop, the loop still refer to element belongs to "previous" page, this is the root cause of "StateElementReference" exception.
So you need to find those elements again on the "new" page to change the reference of element comes from "new" page.
WebElement table2 = driver.findElement(By.cssSelector("body > div:nth-child(74) > div.sp-palette-container"));
List<WebElement> allrows2 = table2.findElements(By.tagName("div"));
int rowSize, cellSize = 0;
rowSize = allrows2.sie();
for(int rowIndex=0;rowIndex<rowSize;rowIndex++){
WebElement row2 = allrows2.get(rowIndex);
List<WebElement> cells = row2.findElements(By.tagName("span"));
cellSize = cells.size();
for(int cellIndex=0;cellIndex<cellSize;cellIndex++){
WebElement cell = cells.get(cellIndex);
if (cell.getAttribute("title").equals("rgb(0, 158, 236)")) {
cell.click();
// find cells again on "new" page
cells = row2.findElements(By.tagName("span"));
// find rows again on "new" page
allrows2 = table2.findElements(By.tagName("div"));
}
}
}

If your usecase is to click() on the elements with title as rgb(0, 158, 236) you can use the following code block :
String baseURL = driver.getCurrentUrl();
List<WebElement> total_cells = driver.findElements(By.xpath("//div[#class='sp-palette-container']//div//span"));
int size = total_cells.size();
for(int i=0;i<size;i++)
{
List<WebElement> cells = driver.findElements(By.xpath("//div[#class='sp-palette-container']//div//span"));
if (cells.get(i).getAttribute("title").contains("rgb(0, 158, 236)"))
{
cells.get(i).click();
//do your other tasks
driver.get(baseURL);
}
}

Use a "break" after clicking on the element found. The exception occurs because, after clicking on your element, the loop continues.
WebElement table2 = driver.findElement(By.cssSelector("body > div:nth-child(74) > div.sp-palette-container"));
List<WebElement> allrows2 = table2.findElements(By.tagName("div"));
for(WebElement row2: allrows2){
List<WebElement> cells = row2.findElements(By.tagName("span"));
for(WebElement cell:cells){
if (cell.getAttribute("title").equals("rgb(0, 158, 236)")) {
cell.click();
break;
}
}
}

How can I verify the selected item in the list?

I need to verify that the selected item in the list is "to-date". When I try to use getAttribute, I get output null for the other items, and true for the selected item, but I don't see their names printed in the console. There are 4 items in the list, and all I see is null,null,null,true. But my main goal is to verify the selected item in the list. Thanks for the help!
The HTML page:
This is my code below:
WebElement defaultItem = driver.findElement(By.xpath("//*[#id='myAddNewCampaignForm']/ol/li[16]/select"));
List<WebElement> allOptions = defaultItem.findElements(By.tagName("option"));
System.out.println("total items " + allOptions.size());
for (int i=0; i<allOptions.size(); i++) {
System.out.println(allOptions.get(i).getText());
System.out.println(allOptions.get(i).getAttribute("selected"));
}

Import this
import org.openqa.selenium.support.ui.Select
Use Select class to control drop down objects.
WebElement defaultItem = driver.findElement(By.xpath("//*[#id='myAddNewCampaignForm']/ol/li[16]/select"));
Select select = new Select(defaultItem);
List<WebElement> selectedOptions = select.getAllSelectedOptions();
for(WebElement option : selectedOptions){
System.out.println(option.getText());
}