Develop Reference

How to scrape selected table columns and write them in CVS in Java Selenium

My object is to scrape data by using Java Selenium. I am able to load selenium driver, connect to the website and fetch the first column then go to the next pagination button until its become disable and write it to the console. Here is what I did so far:
public static WebDriver driver;
public static void main(String[] args) throws Exception {
System.setProperty("webdriver.chrome.driver", "E:\\eclipse-workspace\\package-name\\src\\working\\selenium\\driver\\chromedriver.exe");
System.setProperty("webdriver.chrome.silentOutput", "true");
driver = new ChromeDriver();
driver.get("https://datatables.net/examples/basic_init/zero_configuration.html");
driver.manage().window().maximize();
compareDispalyedRowCountToActualRowCount();
}
public static void compareDispalyedRowCountToActualRowCount() throws Exception {
try {
Thread.sleep(5000);
List<WebElement> namesElements = driver.findElements(By.cssSelector("#example>tbody>tr>td:nth-child(1)"));
System.out.println("size of names elements : " + namesElements.size());
List<String> names = new ArrayList<String>();
//Adding column1 elements to the list
for (WebElement nameEle : namesElements) {
names.add(nameEle.getText());
}
//Displaying the list elements on console
for (WebElement s : namesElements) {
System.out.println(s.getText());
}
//locating next button
String nextButtonClass = driver.findElement(By.id("example_next")).getAttribute("class");
//traversing through the table until the last button and adding names to the list defined about
while (!nextButtonClass.contains("disabled")) {
driver.findElement(By.id("example_next")).click();
Thread.sleep(1000);
namesElements = driver.findElements(By.cssSelector("#example>tbody>tr>td:nth-child(1)"));
for (WebElement nameEle : namesElements) {
names.add(nameEle.getText());
}
nextButtonClass = driver.findElement(By.id("example_next")).getAttribute("class");
}
//printing the whole list elements
for (String name : names) {
System.out.println(name);
}
//counting the size of the list
int actualCount = names.size();
System.out.println("Total number of names :" + actualCount);
//locating displayed count
String displayedCountString = driver.findElement(By.id("example_info")).getText().split(" ")[5];
int displayedCount = Integer.parseInt(displayedCountString);
System.out.println("Total Number of Displayed Names count:" + displayedCount);
Thread.sleep(1000);
// Actual count calculated Vs Dispalyed Count
if (actualCount == displayedCount) {
System.out.println("Actual row count = Displayed row Count");
} else {
System.out.println("Actual row count != Displayed row Count");
throw new Exception("Actual row count != Displayed row Count");
}
} catch (Exception e) {
e.printStackTrace();
}
}
I want to:
scrape more than one column or may be selected columns for example on this LINK name, office and age column
Then want to write these columns data in CSV file
Update
I tried like this but not running:
for(WebElement trElement : tr_collection){
int col_num=1;
List<WebElement> td_collection = trElement.findElements(
By.xpath("//*[#id=\"example\"]/tbody/tr[rown_num]/td[col_num]")
);
for(WebElement tdElement : td_collection){
rows += tdElement.getText()+"\t";
col_num++;
}
rows = rows + "\n";
row_num++;
}

Scraping:
Usually when I want to gather list elements I will select by Xpath instead of CssSelector. The structure of how to access elements through the Xpath is usually more clear, and depends on one or two integer values specifying the element.
So for your example where you want to find the names, you would find an element by the Xpath, the next element in the list's Xpath, and find the differing value:
The first name, 'Airi Satou' is found at the following Xpath:
//*[#id="example"]/tbody/tr[1]/td[1]
Airi's position has the following Xpath:
//*[#id="example"]/tbody/tr[1]/td[2]
You can see that across rows the Xpath for each piece of information differs on the 'td' markup.
The next name in the list, 'Angela Ramos' is found:
//*[#id="example"]/tbody/tr[2]/td[1]
And Angela's position is found:
//*[#id="example"]/tbody/tr[2]/td[2]
You can see that the difference in the column is controlled by the 'tr' markup.
By iterating over values of 'tr' and 'td' you can get the whole table.
As for writing to a CSV, there are a some solid Java libraries for writing to CSVs. I think a straightforward example to follow is here:
Java - Writing strings to a CSV file
UPDATE:
#User169 It looks like you're gathering a list of elements for each row in the table. You want to gather the Xpaths one by one, iterating over the list of webElements that you found originally. Try this, then add to it so it will get text and save it to an array.
for (int num_row = 1; num_row < total_rows; num_row++){
for (int num_col = 1; num_col < total_col; num_col++){
webElement info = driver.findElement(By.xpath("//*[#id=\"example\"]/tbody/tr[" + row_num + ']/td[' + col_num + "]");
}
}
I haven't tested it so it may need a few small changes.

Select a textbox at the location where i have a match for the header data and data in first column

This is how my page looks like on which I need to work.
The DOM looks like this
Scenario :- I need to traverse the table and where the header data(BH001 etc) and first column data(ABC etc) matches the data input by the user, I need to click on the textbox corresponding to it.
I have written the below specified code but its not working :-
public static void getMarksBox(WebDriver driver, String user, String taskCode) {
UserData userNm = TestData.findUserById(user);
String userName = userNm.getName();
WebElement table = WaitUtils.waitForElement(driver, By.cssSelector("table.eds-o-table.cvr-c-table--marksbook"));
List<WebElement> tableCols = table.findElements(By.cssSelector("td.eds-o-table__cell"));
int columnIndex = -1;
for(int i=1; i<tableCols.size();i++)
{
if(userName.equals(tableCols.get(i).findElement(By.cssSelector(".v-label-cvr-c-data-nav-link")).getText()))
{
columnIndex = i;
break;
}
}
List<WebElement> tableRows = table.findElements(By.cssSelector("tr.eds-o-table__row"));
List<WebElement> tableHeaders = tableRows.get(1).findElements(By.cssSelector(".v-label-cvr-u-margin-right--sm"));
WebElement textBox = table.findElement(By.cssSelector(".v-textfield"));
for(WebElement header :tableHeaders)
{
if(taskCode.equals(header.getText()))
{
textBox = tableRows.get(columnIndex);
textBox.click();
WaitUtils.sleepInSeconds(5);
break;
}
}
}

As #supputuri suggested, you can find the matched row or cell directly via XPath/Css selector to avoid complex loop to reduce execution time.
public static void getMarksBox(WebDriver driver, String user, String taskCode) {
UserData userNm = TestData.findUserById(user);
String userName = userNm.getName();
WebElement table = WaitUtils.waitForElement(driver,
By.cssSelector("table.eds-o-table.cvr-c-table--marksbook"));
WebElement matchedRow = table.findElement(By.xpath(
String.format("./tobdy/tr[td[1][normalize-space(.)='%s']]", userName)))
WebElement matchedTextBox = matchedRow.findElement(
By.cssSelector("./td:nth-child(2) input.v-textfield-eds-c-input"))
matchedTextBox.click()
// or you can directly find the matchedTextBox in one findElement
String xpath = String.format(
"./tobdy/tr[td[1][normalize-space(.)='%s']]" +
"/td[2]//input[contains(#class,'v-textfield-eds-c-input')]", userName)
WebElement matchedTextBox = table.findElement(By.xpath(xpath))
matchedTextBox.click()
}

Use this below xpath to access the inputbox directly, rather doing the loops that you have written in the above method.
//td[position()=count(//th[contains(.,'First Name')]/preceding-sibling::th)+1 and normalize-space(.)='ABC']/ancestor::tr//td[position()=count(//th[contains(.,'BH001')]/preceding-sibling::th)+1]//input[contains(#class,'v-textfield-eds-c-input')]
Here is the general notation:
//td[position()=count(//th[contains(.,'your reference column name')]/preceding-sibling::th)+1 and normalize-space(.)='reference value']/ancestor::tr//td[position()=count(//th[contains(.,'target column name')]/preceding-sibling::th)+1]//input[contains(#class,'v-textfield-eds-c-input')]

How to handle table rows and columns

I am using Java and Selenium to write a test. In my target application DOM, there are many tables that I need to go through. I used to use things like:
WebElement mytable = driver.findElement(By.xpath(".//table/tbody"));
List<WebElement> rows_table = mytable.findElements(By.tagName("tr"));
int rows_count = rows_table.size();
for (int row=0; row<rows_count; row++){
List<WebElement> Columns_row = rows_table.get(row).findElements(By.tagName("td"));
int columns_count = Columns_row.size();
System.out.println("Number of cells In Row "+row+" are "+columns_count);
for (int column=0; column<columns_count; column++){
String celtext = Columns_row.get(column).getText();
System.out.println("Cell Value Of row number "+row+" and column number "+column+" Is "+celtext);
}
System.out.println("--------------------------------------------------");
}
But I was looking for something that can handle tables in an easier way, so I searched and found:
1) selenium.getTable
2) GetTable(ElementFinder finder, JavascriptLibrary js)
but I couldn't find any good code sample for them.
Long story short I was wondering if there is any better way rather than finding .//tr or .//td to handle rows and columns in a table?

getTable is not implemented in webdriver. But you can try this.
WebDriver driver = new ChromeDriver();
Selenium selenium = new WebDriverBackedSelenium(driver, "your website url");
selenium .getTable("xpath");
selenium.getTable("table.1.2");
getTable(tableCellAddress) Arguments:
tableCellAddress - a cell address, e.g. "foo.1.4"
Returns: the text from the specified cell
Gets the text from a cell of a table. The cellAddress syntax tableLocator.row.column, where row and column start at 0.
Add a supress warning for the method.

Similar Question's answer replied in the following thread:
Extracting text from webtable selenium webdriver
HashMap<Integer, HashMap<String, String>> tableData = new HashMap<Integer, HashMap<String, String>>();
HashMap<String, String> tableCellDataMap = null;
WebElement table = driver.findElement(By.id("<table_id>"));
/**
* Call this method to get Table Cell Data by passing WebElement table
* #Params table
*
* #author Fayaz
**/
public static HashMap<String, String> getTableData(WebElement table) {
List<WebElements> tableColHeaders = table.findElements(By.tagName("th"));
List<WebElements> tableRows = table.findElements(By.tagName("tr"));
// Start rowIndex with '1' because in the zeroth index we get 'th' tags
for(int rowIndex = 1; rowIndex < tableRows.size(); rowIndex++) {
List<WebElement> tableCells = tableRows.get(rowIndex).findElements(By.tagName("td"));
for(int cellIndex = 0; cellIndex < tableCells.size(); cellIndex++) {
String tableCellData = tableCells.get(cellIndex).getText();
String tableColName = tableColHeaders.get(cellIndex).getText();
tableCellDataMap = new HashMap<String, String>();
tableCellDataMap.put(tableColName, tableCellData);
}
tableData.put(rowIndex, tableCellDataMap);
}
return tableCellDataMap;
}
Actually, it can even be further modularize to make Table utility class with methods like, getTableHeader(), getRowSize(), getColSize(), getData(int rowIndex), getData(int colIndex)
If we have multiple tables, then loop the tables as well

Getting issue while trying to select rows in sequence via Selenium Webdriver

I have table in website. Table allows to select multiple rows by pressing Shift key + Down arrow keys.
I am trying to perform same using selenium webdriver but it's not selecting rows one by one, it select row then unselect it and goes to next....
My Code :
List<WebElement> TRcount = driver.findElements(By.tagName("tr"));
int x;
for(x=0;x<TRcount.size();x++)
{
Actions rows = new Actions(Base.getdriver());
rows.keyDown(TRcount.get(x),Keys.SHIFT).keyUp(TRcount.get(x+1), Keys.SHIFT).build();
rows.build().perform();
TRcount.get(x).click();
}

You pressing keyDown and keyUp. Try
Actions rows = new Actions(Base.getdriver());
rows.keyDown(Keys.SHIFT).perform();
for(x = 0 ; x < TRcount.size() ; x++)
{
TRcount.get(x).click();
}
rows.keyUp(Keys.SHIFT).perform();
By the way, perform() is doing build(), no need to call them both.

I believe this should be:
List<WebElement> TRcount = driver.findElements(By.tagName("tr"));
int x;
Actions rows = new Actions(Base.getdriver());
rows = rows.keyDown(Keys.SHIFT).build();
for(x=0;x<TRcount.size();x++)
{
rows = rows.sendKeys(TRcount.get(x),Keys.DOWN).build();
}
rows = rows.keyUp(Keys.SHIFT).build();
rows.build().perform();
If you have public URL to replicate this then We could try it more easily.

java html parser multi page table

i am using Jsoup as html parser to get all the details from the table in this website. With the code below am only able to get the data on the first page only. Any advise?
public static void main(String[] args) {
String html = "http://www.fifa.com/worldranking/rankingtable/index.html#";
try {
Document doc = Jsoup.connect(html).get();
Elements tableElements = doc.select("table");
Elements tableHeaderEles = tableElements.select("thead tr th");
System.out.println("headers");
System.out.print("row");
for (int i = 0; i < tableHeaderEles.size(); i++) {
System.out.print(tableHeaderEles.get(i).text() + " | ");
}
System.out.println();
Elements tableRowElements = tableElements.select(":not(thead) tr");
for (int i = 0; i < tableRowElements.size(); i++) {
Element row = tableRowElements.get(i);
System.out.print("row");
Elements rowItems = row.select("td");
for (int j = 0; j < rowItems.size(); j++)
{
System.out.print(rowItems.get(j).text() + " | ");
}
System.out.println();
}
} catch (IOException e) {
e.printStackTrace();
} }

JSoup is a HTML parser, but looking at the website is using javascript to load the table. So you will need to click into it.
You could use HTMLUnit or Selenium for navigate and JSoup to parse the HTML.
I hope it helps.
Edit:
Looking better in the code of the page. I think that it could be useful :
http://www.fifa.com/worldranking/rankingtable/gender=m/rank=100/confederation=0/page=0/_ranking_table.html
I change the values of the URL, look that the rank u can increase (is the date of the ranking) and the important one would be the page. You could load all the ranking increasing the page parameter. Then just parsing it with JSoup would be enough.
For example the last ranking would be:
http://www.fifa.com/worldranking/rankingtable/gender=m/rank=237/confederation=0/page=1/_ranking_table.html
Then you could increase the parameter page=2, then 3, ... till 7
Cheers.