I need to do a search with Selenium and display the results. Actually, I have a program in selenium to display a table and put a search word in the text field. And when the word is written automatically, tabulate the columns containing those words.
Here’s a code from my program.
private static String searchName = "//*[#id=\"searchName\"]";
private static String tableExist = "//table[#role='table']/tbody/tr";
//return if exist elements in the table
public static Boolean elementExistsInTheTableCustomer() {
return (driver.findElements(By.xpath(tableExist)).size() > 0);
}
// here i put the text search
public static void enterSearchName(String nameSearchTxt) {
driver.findElement(By.xpath(searchName)).sendKeys(nameSearchTxt);
}
// this function to put 2 char to search in the table
public static void searchStringName() {
String strSearch = RandomStringUtils.randomAlphabetic(2);
System.out.println("The String search : " + strSearch);
Customer.enterSearchName(strSearch);
}
public static WebElement displayElementTables() {
WebElement list = webDriverWait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath(tableExist)));
return list;
}
//here the test of element after and before the search name
#Test
public void GIVEN_Customers_On_Database() throws InterruptedException {
Customer.clickCustomers();
Thread.sleep(3000);
if (Customer.elementExistsInTheTableCustomer()) {
System.out.println("\t===============================");
for (WebElement detail : Customer.displayElementTables()) {
System.out.println("\t The elements in the table : " + detail.getText());
}
System.out.println("\t===============================");
}
Customer.searchStringName();
// here to display if i have elements or no
for (WebElement detail : Customer.displayElementTables()) {
System.out.println("\t The elements in the table : " + detail.getText());
}
}
In the last part, I will check if he has any lines to publish. the displayElementTables() function in which I can receive the search result. whether I have items or not and put the result in a variable to display.
public static List<WebElement> displayElementTables() {
List<WebElement> list = webDriverWait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath(tableExist)));
return list;
}
Related
I'm trying to click on element by text from list of elements, but sometimes elements could have the same text and if statement not executed.
public void clickByText() {
String myText = "Text1";
List<WebElement> elements = driver.findElements(myElements);
for (WebElement e : elements) {
if (e.getText().equals(myText)) {
e.click();
break;
} else {
System.out.println("not exists");
break;
}
}
}
Maybe don't look for the text by equals, use contains.
Remove the break from your code, this makes only the else or only the if block to run only once.
code example:
public void clickByText() {
String myText = "Text1";
List<WebElement> elements = driver.findElements(myElements);
for (WebElement e : elements) {
if (e.getText().contains(myText)) {
e.click();
} else {
System.out.println("not exists: " + e.getText());
}
}
}
I think it's an issue with duplicates. since list in Java can contains duplicates, whereas set do not. try the below code :
List<WebElement> elements = driver.findElements(By.cssSelector(""));
Set<WebElement> setElements = new HashSet<WebElement>(elements);
for (WebElement e : elements) {
// rest of your code
}
My object is to scrape data by using Java Selenium. I am able to load selenium driver, connect to the website and fetch the first column then go to the next pagination button until its become disable and write it to the console. Here is what I did so far:
public static WebDriver driver;
public static void main(String[] args) throws Exception {
System.setProperty("webdriver.chrome.driver", "E:\\eclipse-workspace\\package-name\\src\\working\\selenium\\driver\\chromedriver.exe");
System.setProperty("webdriver.chrome.silentOutput", "true");
driver = new ChromeDriver();
driver.get("https://datatables.net/examples/basic_init/zero_configuration.html");
driver.manage().window().maximize();
compareDispalyedRowCountToActualRowCount();
}
public static void compareDispalyedRowCountToActualRowCount() throws Exception {
try {
Thread.sleep(5000);
List<WebElement> namesElements = driver.findElements(By.cssSelector("#example>tbody>tr>td:nth-child(1)"));
System.out.println("size of names elements : " + namesElements.size());
List<String> names = new ArrayList<String>();
//Adding column1 elements to the list
for (WebElement nameEle : namesElements) {
names.add(nameEle.getText());
}
//Displaying the list elements on console
for (WebElement s : namesElements) {
System.out.println(s.getText());
}
//locating next button
String nextButtonClass = driver.findElement(By.id("example_next")).getAttribute("class");
//traversing through the table until the last button and adding names to the list defined about
while (!nextButtonClass.contains("disabled")) {
driver.findElement(By.id("example_next")).click();
Thread.sleep(1000);
namesElements = driver.findElements(By.cssSelector("#example>tbody>tr>td:nth-child(1)"));
for (WebElement nameEle : namesElements) {
names.add(nameEle.getText());
}
nextButtonClass = driver.findElement(By.id("example_next")).getAttribute("class");
}
//printing the whole list elements
for (String name : names) {
System.out.println(name);
}
//counting the size of the list
int actualCount = names.size();
System.out.println("Total number of names :" + actualCount);
//locating displayed count
String displayedCountString = driver.findElement(By.id("example_info")).getText().split(" ")[5];
int displayedCount = Integer.parseInt(displayedCountString);
System.out.println("Total Number of Displayed Names count:" + displayedCount);
Thread.sleep(1000);
// Actual count calculated Vs Dispalyed Count
if (actualCount == displayedCount) {
System.out.println("Actual row count = Displayed row Count");
} else {
System.out.println("Actual row count != Displayed row Count");
throw new Exception("Actual row count != Displayed row Count");
}
} catch (Exception e) {
e.printStackTrace();
}
}
I want to:
scrape more than one column or may be selected columns for example on this LINK name, office and age column
Then want to write these columns data in CSV file
Update
I tried like this but not running:
for(WebElement trElement : tr_collection){
int col_num=1;
List<WebElement> td_collection = trElement.findElements(
By.xpath("//*[#id=\"example\"]/tbody/tr[rown_num]/td[col_num]")
);
for(WebElement tdElement : td_collection){
rows += tdElement.getText()+"\t";
col_num++;
}
rows = rows + "\n";
row_num++;
}
Scraping:
Usually when I want to gather list elements I will select by Xpath instead of CssSelector. The structure of how to access elements through the Xpath is usually more clear, and depends on one or two integer values specifying the element.
So for your example where you want to find the names, you would find an element by the Xpath, the next element in the list's Xpath, and find the differing value:
The first name, 'Airi Satou' is found at the following Xpath:
//*[#id="example"]/tbody/tr[1]/td[1]
Airi's position has the following Xpath:
//*[#id="example"]/tbody/tr[1]/td[2]
You can see that across rows the Xpath for each piece of information differs on the 'td' markup.
The next name in the list, 'Angela Ramos' is found:
//*[#id="example"]/tbody/tr[2]/td[1]
And Angela's position is found:
//*[#id="example"]/tbody/tr[2]/td[2]
You can see that the difference in the column is controlled by the 'tr' markup.
By iterating over values of 'tr' and 'td' you can get the whole table.
As for writing to a CSV, there are a some solid Java libraries for writing to CSVs. I think a straightforward example to follow is here:
Java - Writing strings to a CSV file
UPDATE:
#User169 It looks like you're gathering a list of elements for each row in the table. You want to gather the Xpaths one by one, iterating over the list of webElements that you found originally. Try this, then add to it so it will get text and save it to an array.
for (int num_row = 1; num_row < total_rows; num_row++){
for (int num_col = 1; num_col < total_col; num_col++){
webElement info = driver.findElement(By.xpath("//*[#id=\"example\"]/tbody/tr[" + row_num + ']/td[' + col_num + "]");
}
}
I haven't tested it so it may need a few small changes.
I am trying to create a discord bot that searches up an item inputted by user "!price item" and then gives me a price that I can work with later on in the code. I figured out how to get the html code into a string or a doc file, but I am struggling on finding a way to extract only prices.
Here is the code:
#Override
public void onMessageReceived(MessageReceivedEvent event) {
String html;
System.out.println("I received a message from " +
event.getAuthor().getName() + ": " +
event.getMessage().getContentDisplay());
if (event.getMessage().getContentRaw().contains("!price")) {
String input = event.getMessage().getContentDisplay();
String item = input.substring(9).replaceAll(" ", "%20");
String URL = "https://www.google.lt/search?q=" + item + "%20price";
try {
html = Jsoup.connect(URL).userAgent("Mozilla/49.0").get().html();
html = html.replaceAll("[^\\ ,.£€eur0123456789]"," ");
} catch (Exception e) {
return;
}
System.out.println(html);
}
}
The biggest problem is that I am using google search so the prices are not in the same place in the html code. Is there a way I can extract only (numbers + EUR) or (a euro sign + price) from the html code?.
you can easily do that scrapping the website. Here's a simple working example to do what you are looking for using JSOUP:
public class Main {
public static void main(String[] args) {
try {
String query = "oneplus";
String url = "https://www.google.com/search?q=" + query + "%20price&client=firefox-b&source=lnms&tbm=shop&sa=X";
int pricesToRetrieve = 3;
ArrayList<String> prices = new ArrayList<String>();
Document document = Jsoup.connect(url).userAgent("Mozilla/5.0").get();
Elements elements = document.select("div.pslires");
for (Element element : elements) {
String price = element.select("div > div > b").text();
String[] finalPrice = price.split(" ");
prices.add(finalPrice[0] + finalPrice[1]);
pricesToRetrieve -= 1;
if (pricesToRetrieve == 0) {
break;
}
}
System.out.println(prices);
} catch (IOException e) {
e.printStackTrace();
}
}
}
That piece of code will output:
[347,10€, 529,90€, 449,99€]
And if you want to retrieve more information just connect JSOUP to the Google Shop url adding your desired query, and scrapping it using JSOUP. In this case I scrapped Google Shop for OnePlus to check its prices, but you can also get the url to buy it, the full product name, etc. In this piece of code I want to retrieve the first 3 prices indexed in Google Shop and add them to an ArrayList of String. Then before adding it to the ArrayList I split the retrieved text by "space" so I just get the information I want, the price.
This is a simple scrapping example, if you need anything else feel free to ask! And if you want to learn more about scrapping using JSOUP check this link.
Hope this helped you!
Using Selenium to gather text of all p elements within a specific div. I noticed while using List, Selenium scanned the whole DOM and stored empty text. So, I wanted to iterate through the DOM and only store values that are not equal to empty text via java.util.Iterator. Is this possible? Is there a more efficient way other than the List approach?
Iterator Approach:
public static boolean FeatureFunctionsCheck(String Feature){
try
{
Iterator<WebElement> all = (Iterator<WebElement>) Driver.Instance.findElement(By.xpath("//a[contains(text()," + Feature + ")]/ancestor::h3/following-sibling::div/div[#class='navMenu']/p"));
boolean check = false;
while(all.hasNext() && check){
WebElement temp = all.next();
if(!temp.getText().equals(""))
{
Log.Info("Functions: " + temp.getText());
all = (Iterator<WebElement>) Driver.Instance.findElement(By.xpath("//a[contains(text()," + Feature + ")]/ancestor::h3/following-sibling::div/div[#class='navMenu']/p"));
}
else
check = true;
}
return false;
}
catch(Exception e)
{
Log.Error("Failed()" + e);
return false;
}
}
Iterator Approach throws exception...
java.lang.ClassCastException: org.openqa.selenium.remote.RemoteWebElement cannot be cast to java.util.Iterator
List Approach Works, However Not Sure If This Is Efficient
public static boolean FeatureFunctionsCheck(String Feature){
try
{
List<WebElement> AllModelFunctions = new ArrayList<WebElement>();
Log.Info("[Test-235]: Selecting Feature");
for(WebElement element: AllModelFunctions){
if(!element.getText().equals(""))
{
Log.Info("Functions: " + element.getText());
}
}
return false;
}
catch(Exception e)
{
Log.Error("Failed()" + e);
return false;
}
}
findElement returns one WebElement. What you probably meant to do is to search for all elements with given xpath, using findElements:
Driver.Instance.findElements(...
Also the syntax is over-complicated. You can just get the list and iterate through it:
List<WebElement> elements = Driver.Instance.findElements(...);
for(WebElement element : elements) {
if(!element.getText().equals(""))
{
Log.Info("Functions: " + element.getText());
}
}
BTW I have to fully trust that Driver.Instance is an instance of the driver (typically in Java you don't have capitals for class instances, so I'm not sure if I understood it right). A more common syntax would be something like:
WebDriver driver = new FirefoxDriver(); // or another browser
driver.findElements(...);
// ...
I have problem regarding my Filtering of my Table. I'm using a ViewerFilter and override the select Method to fitler that Table. The Filtertext itself is entered via a Textfield.
So now to my problem. For example my table looks like the following:
column
123
124
In my textfield the user can enter the columnname=data1,data2 to show all rows which have either data1 or data2 as data. so in my above example if the user enters column=123,124 both rows should still be visible. The problem here is that I refresh my tableviewer after each entered character. So when the user enters column=123 the Table only shows one column. When adding ,124 to the filtertext I filter my already filtered table. So no data gets shown at the end. How can I still filter the original Tabledata?
#Override
public boolean select(final Viewer viewer, final Object parentElement, final Object element) {
final String filterString = filterText.getText().toLowerCase();
if (filterString.length() == 0) { return true; }
final mydata myData= (mydata) element;
if (filterString.matches("columnName" + ".+")) {
index = filterString.indexOf("columnName" + ".+");
evaluateText(myData, filterString, i, index + tableColumnsText[i].length())
}
public boolean evaluateText(final mydata data, final String filterText, final int beginningIndex) {
subString = filterText.substring(beginningIndex, filterText.length());
return evaluateString(data.getString(), subString);
}
public boolean evaluateString(final String cellString, final String commaString) {
int countSubstrings = 0;
final String[] items = commaString.split(",");
countSubstrings = items.length;
for (final String s : items) {
if (s.length() != 0) {
if (!cellString.contains(s)) { return false; }
}
}
return true;
}
So I tried to filter out the main components of the method. Can I somehow access the unfiltered table?