Develop Reference

How to replace a space with a word while extract the data from PDF using PDFBox

I want to replace any empty column with a word; for example, the word BLK while extract Pdf data.
the below tables are the example of the expected table and actual result.
Original Table
+--------------------------------------+
|# |NAME |TEL |GENDER |
|---------------------------|----------|
|1 |JOHN |096587498 |M |
|2 |VILLA | |F |
+--------------------------------------+
Expected Result
# NAME TEL GENDER
1 JOHN 096587498 M
2 VILLA BLK F
Actual Result
# NAME TEL GENDER
1 JOHN 096587498 M
2 VILLA F
The actual result is from the class PDFTextStripper.
capture of pdf

The PDFTextStripper does not see the graphical lines in the PDF, it merely sees text characters. Thus, in your line #2 it sees "2", "Villa", and "F" with gaps in-between. With this class alone, therefore, you won't get what you want.
In general you have the following options using PDFBox:
You can first try and recognize the table cell regions in your PDF by parsing the vector graphics instructions of the page and then extract text cell by cell.
This answer provides a proof-of-concept for this. Beware: This answer focuses on the example document provided by the OP of that question. In particular it expects the lines to be drawn as thin filled rectangles; for a generic solution, the code collecting the table lines needs to be extended to also recognize lines drawn otherwise.
This approach obviously requires table rows and columns to be divided by lines (or by extension alternatively by background colors or something similar); this is not always the case.
In case of your example document the code works out of the box:
[A1] #
[A2] Name
[A3] Tel
[A4] Gender
[B1] 1
[B2] John
[B3] 096875959
[B4] M
[C1] 2
[C2] Villa
[C3]
[C4] F
(output of ExtractBoxedText test testExtractBoxedTextsTestWPhromma)
You can extract the text attempting to reflect the layout of the PDF. If you know the general layout of the table in question (column n goes from here to there...), you can derive the table cell contents.
This answer provides a proof-of-concept for the layout-aware text extraction. Beware, the code is PDFBox 1.8.x based, some adaptions might be necessary.
This approach requires knowledge of the table column layout; this is not always given.
In case of your example document the code works out of the box:
# Name Tel Gender
1 John 096875959 M
2 Villa F
(output of ExtractTextWithLayout test testExtractTestWPhromma)
For tagged PDFs you can try to extract the text including the tagging which reflects the table structure (if properly tagged).
As your example document is tagged, I'll show a quick & dirty proof-of-concept for this below.
This approach requires the PDF to be properly tagged; this is not always the case.
Extraction of content with tags
If your PDF is properly tagged, you can extract the content including the markup tags like this:
PDDocument document = PDDocument.load(SOURCE);
Map<PDPage, Map<Integer, PDMarkedContent>> markedContents = new HashMap<>();
for (PDPage page : document.getPages()) {
PDFMarkedContentExtractor extractor = new PDFMarkedContentExtractor();
extractor.processPage(page);
Map<Integer, PDMarkedContent> theseMarkedContents = new HashMap<>();
markedContents.put(page, theseMarkedContents);
for (PDMarkedContent markedContent : extractor.getMarkedContents()) {
theseMarkedContents.put(markedContent.getMCID(), markedContent);
}
}
PDStructureNode root = document.getDocumentCatalog().getStructureTreeRoot();
showStructure(root, markedContents);
(ExtractMarkedContent test testExtractTestWPhromma)
using these two helper methods
void showStructure(PDStructureNode node, Map<PDPage, Map<Integer, PDMarkedContent>> markedContents) {
String structType = null;
PDPage page = null;
if (node instanceof PDStructureElement) {
PDStructureElement element = (PDStructureElement) node;
structType = element.getStructureType();
page = element.getPage();
}
Map<Integer, PDMarkedContent> theseMarkedContents = markedContents.get(page);
System.out.printf("<%s>\n", structType);
for (Object object : node.getKids()) {
if (object instanceof COSArray) {
for (COSBase base : (COSArray) object) {
if (base instanceof COSDictionary) {
showStructure(PDStructureNode.create((COSDictionary) base), markedContents);
} else if (base instanceof COSNumber) {
showContent(((COSNumber)base).intValue(), theseMarkedContents);
} else {
System.out.printf("?%s\n", base);
}
}
} else if (object instanceof PDStructureNode) {
showStructure((PDStructureNode) object, markedContents);
} else if (object instanceof Integer) {
showContent((Integer)object, theseMarkedContents);
} else {
System.out.printf("?%s\n", object);
}
}
System.out.printf("</%s>\n", structType);
}
void showContent(int mcid, Map<Integer, PDMarkedContent> theseMarkedContents) {
PDMarkedContent markedContent = theseMarkedContents != null ? theseMarkedContents.get(mcid) : null;
List<Object> contents = markedContent != null ? markedContent.getContents() : Collections.emptyList();
StringBuilder textContent = new StringBuilder();
for (Object object : contents) {
if (object instanceof TextPosition) {
textContent.append(((TextPosition)object).getUnicode());
} else {
textContent.append("?" + object);
}
}
System.out.printf("%s\n", textContent);
}
(ExtractMarkedContent helper methods)
The output for your example PDF
is
<null>
<Document>
<Table>
<THead>
<TR>
<TH>
<P>
#
</P>
</TH>
<TH>
<P>
Name
</P>
</TH>
<TH>
<P>
Tel
</P>
</TH>
<TH>
<P>
Gender
</P>
</TH>
</TR>
</THead>
<TBody>
<TR>
<TH>
<P>
1
</P>
</TH>
<TD>
<P>
John
</P>
</TD>
<TD>
<P>
096875959
</P>
</TD>
<TD>
<P>
M
</P>
</TD>
</TR>
<TR>
<TH>
<P>
2
</P>
</TH>
<TD>
<P>
Villa
</P>
</TD>
<TD>
<P>
</P>
</TD>
<TD>
<P>
F
</P>
</TD>
</TR>
</TBody>
</Table>
<P>
</P>
</Document>
</null>
You recognize the empty cell:
<TD>
<P>
</P>
</TD>
This proof-of-concept extracts to the standard output. You obviously can alternatively collect the data in a string builder or stream, or you can fill the <Table> data immediately into custom structures, they after all already come separated in cells.
Beware: This only is a proof-of-concept. Where the code outputs data like this System.out.printf("?%s\n", ...);, some specific handling may be required. Also other border conditions likely are not adequately considered. (Actually I only implemented it to properly extract the contents of your example PDF.)

How can I get the XPATH directly from accessed element using different selector [duplicate]

I would like to find an element using a differect cssSelector / tag / ClassName amd them get it's xpath value ( to be more specific, I have a website where when a day changes, one of the classes change it's class) here is what do I meean:
<tr>
<td> 1.1.2019 </td>
<td> 2.1.2019 </td>
<td class="active"> 3.1.2019 </td>
<td> 4.1.2019 </td>
</tr>
<tr>
<td> </td>
<td> 10 </td>
<td> </td> #Here
<td> </td>
</tr>
I want to according to where is that "active class", click the table under it. ny idea how to do so ?
short version of what I want :
Find element using cssSelector
Get this element's Xpath <- the problem
click it using edited xpath
I want to GET XPATH OF LOCATED ELEMENT , not to locate it using Xpath

You can find the index by locating all the <td> elements in the first row and check wich one has the index
List<WebElement> columns = driver.findElements(By.xpath("//tr[td[#class='active']]/td")); # just an example, can be any other locator
int index = 0;
for (int i = 0 ; i < columns.getSize() ; i++) {
String attribute = columns.get(i).getAttribute("class")
if (attribute != null && attribute.equals("active")) {
index = i + 1;
}
}

Find a table row number using selenium webdriver

I've got a table and need to find a specific row number. In this case, I'm interested in the 2nd row (2nd tr).
Here's the HTML:
<table>
<thead>
<tbody>
<tr class="classes mico_models_classes_7 listRowWhite">
<tr class="classes mico_models_classes_8 listRowDark">
<td style="width:130px;">9:00am - 10:30am</td>
<td class="noprint enrolledFull" style="width:70px;height:40px;text-align:center;"> </td>
<td style="width:200px;">
<a class="eventName" data-eventdescription="Very long spin class" data-eventname="Spinning 90 min" href="javascript://">Spinning 90 min</a>
<br>
with
<a class="classesEmployeeName" title="John Doe" href="javascript://" data-employeeid="5117">John Doe</a>
</td>
<td style="text-align:right;width:72px;">0 of 20</td>
<td>Spin Class</td>
</tr>
<tr class="classes mico_models_classes_9 listRowWhite">
<tr class="classes mico_models_classes_10 listRowDark">
</tbody>
</table>
Unfortunately, the following returns null for dataRowIndex:
String classTittle = "Spinning 90 min";
String dataRowIndex = driver.findElement(By.cssSelector("[data-eventname='" + classTitle + "']")).getAttribute("rowIndex");

You can solve this by iterating the tr HTML-elements while counting them. You can also access them while iterating.
List<WebElement> element = driver.findElements(By.cssSelector("tr"));
int row = 0;
for( WebElement w : element){
String elemText = w.getText();
System.out.println(elemText);
String clickText = "Spinning 90 min";
if(elemText.contains(clickText)){
w.click(); //do something with the element
System.out.println("Text in row " + row + " is " + clickText + " so i clicked it!");
}
System.out.println("this was row " + row + "\n");
row++;
}
Will yield:
this was row 0
9:00am - 10:30am Spinning 90 min
with John Doe 0 of 20 Spin Class
Text in row 1 is Spinning 90 min so i clicked it!
this was row 1
this was row 2
this was row 3
You might want to encapsulate your specific logic in a method later on. Hope this helps ^^-d

Locating data in a complex table in Selenium Webdriver

I am currently trying to drill down on a user in a table full of users using Selenium webdriver, I have worked out how to iterate through the table but I'm having trouble actually selecting the person I want.
Here is the HTML (modified with X's due to it not being my data)
<table id="XXXXXXXXX_list" cellspacing="0" cellpadding="0" style=" border:0px black solid;WIDTH:100%;">
<tbody>
<tr cellspacing="0" style="height: 16px;">
<tr>
<tr onclick="widgetListView_onClick('XXXX_list',1,this,event)">
<tr onclick="widgetListView_onClick('XXXX_list',2,this,event)">
<tr onclick="widgetListView_onClick('XXXX_list',3,this,event)">
<tr onclick="widgetListView_onClick('XXXX_list',4,this,event)">
<tr onclick="widgetListView_onClick('XXXX_list',5,this,event)">
<tr onclick="widgetListView_onClick('XXXX_list',6,this,event)">
<tr onclick="widgetListView_onClick('XXXX_list',7,this,event)">
<td class="listView_default_dataStyle" nowrap="" style="font-size:12px ;
font-family: sans-serif ;color: black ;background: #FFFFFF "
ondblclick="XXXXListView_onDblClick('XXXXX_list',17, event)">NAME</td>
<td class="listView_default_dataStyle" nowrap="" style="font-size:12px ;font-family: sans-serif;
color: black ;background: #FFFFFF " ondblclick="XXXXX_onDblClick('XXXX_list',17, event)"> </td>
</tr>
Here is the code I am writing to try and find the user going by NAME in the table.
WebElement table = driver.findElement(By.id("table_list"));
// Now get all the TR elements from the table
List<WebElement> allRows = table.findElements(By.tagName("tr"));
// And iterate over them, getting the cells
for (WebElement row : allRows) {
List<WebElement> cells = row.findElements(By.tagName("td"));
for (WebElement cell : cells) {
List<WebElement> Names = cell.findElements(By.xpath("//td[text()='NAME']"));
System.out.println(Names);
This just prints thousands of [] (the table is huge in the real application).
Essentially what I need is to stop when I find the correct name and create a web element out of that table row. Which I can then click and drill down on.
Sorry if any of this is a bit vague,

Well if each name in the table is unique, you don't need to complicate things so much. Just search for element with text matching your 'Name' then select the row accordingly. Look at the code below:
WebElement name = driver.findElement(By.xpath("//table[#id='XXXXXXXXX_list']//td[contains(text(),'NAME')]"));//Select td with text NAME in table with id XXXXXXXXX_list
WebElement rowWithName = name.findElement(By.xpath("./.."));//Select the parent node, i.e., tr, of the td with text NAME
/*
* Look into that row for other element or perform any action on the row.
*/
If the names are not unique, i.e., same name exists twice at similar node, 1st instance will be picked each time. In that case we will have to try things differently, i.e., we will have to index the xpath for correct instance of matching name. Do ask if you have any further doubts :)

This will help you out.
try{
ArrayList<WebElement> cells = (ArrayList<WebElement>) driver.findElements(By.tagName("td"));
log4j.info("Value = "+input_type+" is stored in array from Webpage for "+keyword+" ");
for(WebElement type : cells)
{
if(type.getAttribute("name").equals("your correct name here")) {
type.sendKeys("ABC");
}
}
return true;
}catch(Throwable e){
return false;
}
You need to use Array list like this and you can compare your Name in which you wanna fill value Or wanna do any operation like getText(), click() etc.
Enjoy!

dynamic table data retrieve - selenium webdriver

<table id="tblListViewHeader" class="adminlist" cellspacing="1" cellpadding="0" style="table-layout: fixed; width: 1003px;">
<tbody>
</table>
</td>
</tr>
<tr>
<td>
<div id="divListView" style="width: 100%; height: 300px; overflow: auto; display: block;">
<table id="tblListView" class="adminlist" cellspacing="1" cellpadding="0" style="table-layout: fixed; width: 100%;">
<tbody data-bind="template: { name: 'ActiveGradeTemplate', foreach: ActiveGrade }">
<tr class="row0">
<td data-bind="text:$index()+1" style="width: 5%;">1</td>
<td data-bind="text: GradeName" style="width: 20%;">Vantage Point</td>
<td align="right" data-bind="text: DisplayCreatedDate" style="width: 10%;">27 Mar 2013</td>
<td align="right" data-bind="text: CreatedByUser" style="width: 10%;">Name</td>
<td align="right" data-bind="text: DisplayModifiedDate" style="width: 10%;">27 Mar 2013</td>
<td align="right" data-bind="text: ModifiedByUser" style="width: 10%;">Name</td>
<td align="center" data-bind="text: Status" style="width: 5%;">Active</td>
<td align="center" style="width: 10%;">
<a id="lnkEdit_7" data-bind="click: $root.lnkEdit, attr:{'id':'lnkEdit_' + GradeID}" href="#">Edit</a>
<span id="spanEdit_7" data-bind="attr:{'id':'spanEdit_' + GradeID}"></span>
</td>
</tr>
<tr class="row0">
<td data-bind="text:$index()+1" style="width: 5%;">2</td>
<td data-bind="text: GradeName" style="width: 20%;">test grade</td>
<td align="right" data-bind="text: DisplayCreatedDate" style="width: 10%;">Yesterday</td>
<td align="right" data-bind="text: CreatedByUser" style="width: 10%;">Name</td>
<td align="right" data-bind="text: DisplayModifiedDate" style="width: 10%;">Yesterday</td>
<td align="right" data-bind="text: ModifiedByUser" style="width: 10%;">Name</td>
<td align="center" data-bind="text: Status" style="width: 5%;">Active</td>
<td align="center" style="width: 10%;">
<a id="lnkEdit_11" data-bind="click: $root.lnkEdit, attr:{'id':'lnkEdit_' + GradeID}" href="#">Edit</a>
<span id="spanEdit_11" data-bind="attr:{'id':'spanEdit_' + GradeID}"></span>
</td>
</tr>
How can I retrieve the td values for each and every row, this is for Dynamic generation. All the tr class names are the same: <tr class="row0">. How do I retreive the table data for the above formatted table?

Try below code, this will print all cells data,
// Grab the table
WebElement table = driver.findElement(By.id("divListView"));
// Now get all the TR elements from the table
List<WebElement> allRows = table.findElements(By.tagName("tr"));
// And iterate over them, getting the cells
for (WebElement row : allRows) {
List<WebElement> cells = row.findElements(By.tagName("td"));
// Print the contents of each cell
for (WebElement cell : cells) {
System.out.println(cell.getText());
}
}

// Grab the table
WebElement table = driver.findElement(By.id("table-6"));
//Get number of rows in table
int numOfRow = table.findElements(By.tagName("tr")).size();
//Get number of columns In table.
int numOfCol = driver.findElements(By.xpath("//*[#id='table-6']/tbody/tr[1]/td")).size();
//divided Xpath In three parts to pass Row_count and Col_count values.
String first_part = "//*[#id='table-6']/tbody/tr[";
String second_part = "]/td[";
String third_part = "]";
//take the second column values
int j=2;
//List to store the second column
List<String> secondColumnList=new ArrayList<String>();
//Loop through the rows and get the second column and put it in a list
for (int i=1; i<=numOfRow; i++){
//Prepared final xpath of specific cell as per values of i and j.
String final_xpath = first_part+i+second_part+j+third_part;
//Will retrieve value from located cell and print It.
String test_name = driver.findElement(By.xpath(final_xpath)).getText();
secondColumnList.add(test_name);
System.out.println(test_name);
}

Dynamic table data capturing:
1.First of all Capture Table Head Count.
[int tHeadCount = driver.findElements(By.xpath("//table//tr//th")).size();]
2.Capture Table Row Count in which row your actual data exists.
[-in my point of view i need data from first row it self, so i am hard coding it to zero.]
If you want please add one for loop to existing code.
3.The actual solution starts from here.
Following is function call "Deposited By" is table heading text of corresponding table data.
String tableDataValue = managePackageTableData("Deposited By");
public String managePackageTableData(String columnName) {
//In Following line i am capturing table contains how many headers.
int tHeadCount = driver.findElements(By.xpath("//table//tr//th")).size();
int statusIndex = 0;
for(int i=0;i<tHeadCount-1;i++)
{
String theadValue = driver.findElements(By.className("table")).get(0).findElements(By.tagName("tr")).get(0).findElements(By.tagName("th")).get(i).getText();
if(theadValue.equalsIgnoreCase(columnName))
{
statusIndex = i;
break;
}
}
String tableData = driver.findElements(By.tagName("tbody")).get(0).findElements(By.tagName("tr")).get(0).findElements(By.tagName("td")).get(statusIndex).getText();
return tableData;
}

You have many options, but there's mine.
You catch all tds into a list.
List<WebElement> tdlist = driver.findElements(By.cssSelector("table[id='divListView'] tr td"));
and if you want to have the value, you can use a loop.
for(WebElement el: tdlist) {
Systeme.out.println(el.getText());
}

Check out this
Most common Challenge Automation tester face during iterating through Table and list. they often want to find some value from the table cell or list and want to perform action on the same value or find corresponding other element in the same block and perform action on it.
http://qeworks.com/iterate-table-lists-selenium-webdriver/

I have done this code using TestComplete and have replicated it now with Selenium C#, I have learnt this hard way but will work for any table control and you don't have to hardcode any xpath elements in it. Also if you have a nested table control within a td like for example where you have a nested table structure where your data is interpreted like below(happens in complex tables when you use developer express grids or angular grid controls).This case if you see the td tag again has a nested table structure which again has duplicate data. You can either capture such data or leave it depending on the case using the code which I am giving below.
Html
<table>
<tr>
<td>Account #
<table>
<tr>
<td>
Account #
</td>
</tr>
</table>
</td>
<td>Name
<table>
<tr>
<td>
Name
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td>1234
<table>
<tr>
<td>
1234
</td>
</tr>
</table>
</td>
<td>Bharat
<table>
<tr>
<td>
Bharat
</td>
</tr>
</table>
</td>
</tr>
</table>
Code
public List<TableDataCollection> StoreHtmlTableToList(IWebElement tblObj)
{
DataTable dataTbl = new DataTable();
int rowIndex = 1;
try
{
_tblDataCollection = new List<TableDataCollection>();
var tblRows = ((IJavaScriptExecutor)DriverContext.Driver).ExecuteScript("return arguments[0].rows; ", tblObj);
if (tblRows != null)
{
//Iterate through each row of the table
foreach (IWebElement tr in (IEnumerable)tblRows)
{
int colIndx = 1;
// Iterate through each cell of the table row
var tblCols = ((IJavaScriptExecutor)DriverContext.Driver).ExecuteScript("return arguments[0].cells; ", tr);
foreach (IWebElement td in (IEnumerable)tblCols)
{
//loop through any child or nested table structures if you want using the same approach
//Write Table to List : This part is not done yet
//Print the values
Console.WriteLine("Row[" + rowIndex.ToString() + "] Col[" + colIndx.ToString() + "] : " + td.Text);
colIndx++;
}
rowIndex++;
}
}
}
catch (Exception)
{
throw;
}
return _tblDataCollection;
}

# Ripon Al Wasim
The below code will helps you to find values column by column
WebElement customtable = t.driver.findElement(By.cssSelector("div.custom-table"));
List<WebElement> r = customtable.findElements(By.tagName("tr"));
for (WebElement row : r) {
List<WebElement> d = row.findElements(By.tagName("td"));
for(int i = 0; i<d.size(); i++) {
if(i==0) {
WebElement x =d.get(i);
JavascriptExecutor js = (JavascriptExecutor) t.driver;
js.executeScript("arguments[0].scrollIntoView();", x);
System.out.println(i+"."+d.get(i).getText()+"\n");
if(d.get(i).getText().contains(searchtext)) {
System.out.println(i+".yes\n");
}
else
{
System.out.println("No\n");
}
}
}
}
its working for me.

//tbody/tr
this will give you the total no of row-
//tbody/tr/td
this will give you all the cell for the above rows and you can iterate it based on your requiredment.

Below are different approach we can follow to handle dynamic data in application [Its not for dynamic elements];
Using excel approach;
a. Get the web-element of the field
b. Get its text .
c. Store the data in the excel and validate with actual result pattern.
Note : There are multiple data validation options in excel like compare columns, get duplicate etc..
Using collection;
a. Get the web-element of the field.
b. Get its data inside the collection [List, set, map etc]
c. Write the java code to compare the pattern of the application data.
i. Pattern can be data type, data length, data range ,Decimal places of amount field or other ,currency type, date and time pattern etc.
ii. You can write the java conditions to verify the values of charts/graphs/dashboard if you are using in your application.
d. Compare the actual data pattern[from the collection] and the expected data pattern[From the java code]
Use JDBC API to handle it through the database where you can check the actual data by using different commands.