This is what my code does.
(1.) Open chrome browser and go to footlocker.ca
(2.) Click on the Mens button
(3.) Select 1 random product from the list of 60
(4.) Print the product name and price
(5.) Print all available sizes for the selected product (avaSizes)
(6.) Go back to the products page
(7.) Select a 2nd random product from the list of 60
(8.) Print the product name and price
(9.) Print all available sizes for the selected product (avaSizes)
(10.) Close chrome browser
My problem is that it fails to read the available sizes for the product. I think my problem is in the xpath but I am not to sure as I have tinkered with various xpath's so it may be my code that is the problem. The method is called (avaSizes). If someone can help it would be great. I am doing this for practice so if anyone has some real time job scenario test cases that I could add on to this code it would help me a lot. Thanks.
public class FootlockerExample {
WebElement next;
WebDriver driver = new ChromeDriver();
public void productOne (){
// Open Chrome Browser
System.setProperty("webdriver.chrome.driver", "C:\\Users\\Working\\Workspace\\SeleniumProject\\chromedriver.exe");
// Open Footlocker website and maximize window
driver.get("http://www.footlocker.ca/");
driver.manage().window().maximize();
// Find button element 'Mens' and click
next = driver.findElement(By.xpath("//*[#id='global-nav']/ul/li[1]/a"));
next.click();
// Select a random product
selectRandomProduct();
// Print out the product name and price
String productName = driver.findElement(By.xpath("//*[#id='product_form']/div/span[2]/div/div[1]")).getText();
String Price = driver.findElement(By.xpath("//*[#id='product_form']/div/span[2]/div/div[2]")).getText();
System.out.println("The 1st randomly selected product is " + productName + " and it's cost is " + Price + ".");
// Print all available product sizes
avaSizes();
// Execute new method
productTwo();
}
public void productTwo(){
// Go back a browser page
driver.navigate().back();
// Select a new random product
selectRandomProduct();
// Print out the product name and price
String productName = driver.findElement(By.xpath("//*[#id='product_form']/div/span[2]/div/div[1]")).getText();
String Price = driver.findElement(By.xpath("//*[#id='product_form']/div/span[2]/div/div[2]")).getText();
System.out.println("The 2nd randomly selected product is " + productName + " and it's cost is " + Price + ".");
// Print all available product sizes
avaSizes();
driver.close();
}
public void selectRandomProduct(){
// Find and click on a random product
List<WebElement> allProducts = driver.findElements(By.xpath("//*[#id='endecaResultsWrapper']/div[3]//img"));
Random rand = new Random();
int randomProduct = rand.nextInt(allProducts.size());
allProducts.get(randomProduct).click();
}
public void avaSizes(){
// Find all the available shoe sizes for each randomly selected product
List<WebElement> avaSizes = driver.findElements(By.xpath("//*[#id='product_sizes']"));
int totalSizes = 0;
for(int i=0; i<avaSizes.size(); i++){
if(avaSizes.get(i).isEnabled()==true){
avaSizes.get(i).getText();
System.out.println(avaSizes);
totalSizes++;
}else{
System.out.println("Out of stock in all sizes.");
}
}
System.out.println("This product is available in: " + totalSizes + " sizes.");
}
public static void main(String[] args) {
FootlockerExample obj1 = new FootlockerExample();
obj1.productOne();
}
}
I see that the dropdown element has an id and is of type Select.
You can work with it using the Select object:
Select select = new Select(driver.findElement(By.id("product_sizes")));
List<WebElement> availableSizes = select.getOptions();
for (WebElement size : availableSizes) {
System.out.println(size.getText());
}
With a little help from Marius D above, I have figured out the problem and fixed my code.
Here is my fixed answer for future in case someone is stuck on the same issue.
public void avaSizes(){
Select select = new Select(driver.findElement(By.id("product_sizes")));
// Find all the available shoe sizes for each randomly selected product
List<WebElement> avaSizes = select.getOptions();
int totalSizes = 0;
for(WebElement size:avaSizes){
if(size.isEnabled()==true){
System.out.println(size.getText());
totalSizes++;
}else{
System.out.println("Out of stock in " + size.getText());
}
}
System.out.println("This product is available in: " + totalSizes + " sizes.");
}
Related
My object is to scrape data by using Java Selenium. I am able to load selenium driver, connect to the website and fetch the first column then go to the next pagination button until its become disable and write it to the console. Here is what I did so far:
public static WebDriver driver;
public static void main(String[] args) throws Exception {
System.setProperty("webdriver.chrome.driver", "E:\\eclipse-workspace\\package-name\\src\\working\\selenium\\driver\\chromedriver.exe");
System.setProperty("webdriver.chrome.silentOutput", "true");
driver = new ChromeDriver();
driver.get("https://datatables.net/examples/basic_init/zero_configuration.html");
driver.manage().window().maximize();
compareDispalyedRowCountToActualRowCount();
}
public static void compareDispalyedRowCountToActualRowCount() throws Exception {
try {
Thread.sleep(5000);
List<WebElement> namesElements = driver.findElements(By.cssSelector("#example>tbody>tr>td:nth-child(1)"));
System.out.println("size of names elements : " + namesElements.size());
List<String> names = new ArrayList<String>();
//Adding column1 elements to the list
for (WebElement nameEle : namesElements) {
names.add(nameEle.getText());
}
//Displaying the list elements on console
for (WebElement s : namesElements) {
System.out.println(s.getText());
}
//locating next button
String nextButtonClass = driver.findElement(By.id("example_next")).getAttribute("class");
//traversing through the table until the last button and adding names to the list defined about
while (!nextButtonClass.contains("disabled")) {
driver.findElement(By.id("example_next")).click();
Thread.sleep(1000);
namesElements = driver.findElements(By.cssSelector("#example>tbody>tr>td:nth-child(1)"));
for (WebElement nameEle : namesElements) {
names.add(nameEle.getText());
}
nextButtonClass = driver.findElement(By.id("example_next")).getAttribute("class");
}
//printing the whole list elements
for (String name : names) {
System.out.println(name);
}
//counting the size of the list
int actualCount = names.size();
System.out.println("Total number of names :" + actualCount);
//locating displayed count
String displayedCountString = driver.findElement(By.id("example_info")).getText().split(" ")[5];
int displayedCount = Integer.parseInt(displayedCountString);
System.out.println("Total Number of Displayed Names count:" + displayedCount);
Thread.sleep(1000);
// Actual count calculated Vs Dispalyed Count
if (actualCount == displayedCount) {
System.out.println("Actual row count = Displayed row Count");
} else {
System.out.println("Actual row count != Displayed row Count");
throw new Exception("Actual row count != Displayed row Count");
}
} catch (Exception e) {
e.printStackTrace();
}
}
I want to:
scrape more than one column or may be selected columns for example on this LINK name, office and age column
Then want to write these columns data in CSV file
Update
I tried like this but not running:
for(WebElement trElement : tr_collection){
int col_num=1;
List<WebElement> td_collection = trElement.findElements(
By.xpath("//*[#id=\"example\"]/tbody/tr[rown_num]/td[col_num]")
);
for(WebElement tdElement : td_collection){
rows += tdElement.getText()+"\t";
col_num++;
}
rows = rows + "\n";
row_num++;
}
Scraping:
Usually when I want to gather list elements I will select by Xpath instead of CssSelector. The structure of how to access elements through the Xpath is usually more clear, and depends on one or two integer values specifying the element.
So for your example where you want to find the names, you would find an element by the Xpath, the next element in the list's Xpath, and find the differing value:
The first name, 'Airi Satou' is found at the following Xpath:
//*[#id="example"]/tbody/tr[1]/td[1]
Airi's position has the following Xpath:
//*[#id="example"]/tbody/tr[1]/td[2]
You can see that across rows the Xpath for each piece of information differs on the 'td' markup.
The next name in the list, 'Angela Ramos' is found:
//*[#id="example"]/tbody/tr[2]/td[1]
And Angela's position is found:
//*[#id="example"]/tbody/tr[2]/td[2]
You can see that the difference in the column is controlled by the 'tr' markup.
By iterating over values of 'tr' and 'td' you can get the whole table.
As for writing to a CSV, there are a some solid Java libraries for writing to CSVs. I think a straightforward example to follow is here:
Java - Writing strings to a CSV file
UPDATE:
#User169 It looks like you're gathering a list of elements for each row in the table. You want to gather the Xpaths one by one, iterating over the list of webElements that you found originally. Try this, then add to it so it will get text and save it to an array.
for (int num_row = 1; num_row < total_rows; num_row++){
for (int num_col = 1; num_col < total_col; num_col++){
webElement info = driver.findElement(By.xpath("//*[#id=\"example\"]/tbody/tr[" + row_num + ']/td[' + col_num + "]");
}
}
I haven't tested it so it may need a few small changes.
I am trying to grab the URL of each Laptop that is on sale on the first 3 pages of this Amazon page
URL: https://www.amazon.com/s?i=computers&rh=n%3A565108%2Cp_72%3A1248879011&pf_rd_i=565108&pf_rd_p=b2e34a42-7eb2-50c2-8561-292e13c797df&pf_rd_r=CP4KYB71SY8E0WPHYJYA&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&qid=1590091272&ref=sr_pg_1
Every time I run the script, the driver.findElements(By.xpath) returns an inconsistent amount of URLs. The first page is pretty consistent and it return 4 URLs but page 2 and 3 can return anywhere between 1 and 4 URLs even though page 2 has 8 URLs I am looking for and page 3 has 4 URLs I am looking for.
I doubt the problem is in the grabData method since it grabs the data based on the inconsistent URLs list given. I am pretty new to this so I hope that all made sense. Any help would be appreciated. Let me know if you need more clarification
public static String dealURLsXpath = "//span[#data-a-strike=\"true\" or contains(#class,\"text-strike\")][.//text()]/parent::a[#class]";
public static List<String> URLs = new ArrayList<String>();
public static void main(String[] args)
{
//Initialize Browser
System.setProperty("webdriver.chrome.driver", "C:\\Users\\email\\eclipse-workspace\\ChromeDriver 81\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);
//Search through laptops and starts at page 1
Search.searchLaptop(driver);
//Grabs data for each deal and updates Products List directly
listingsURL = driver.getCurrentUrl();
//updates the global URLs List with the URLs found by driver.findElements(By.xpath)
updateURLsList(driver);
//Iterates through each URL and grabs laptop information to add to products list
grabData(driver, URLs, "Laptop");
// Clears URLs list so that it can be populated by the URLs in the next page
URLs.clear();
// returns driver to Amazon page to click on "page 2" button to go to next page and repeat process
driver.get(listingsURL);
driver.findElement(By.xpath("//a [contains(#href,'pg_2')]")).click();
listingsURL = driver.getCurrentUrl();
updateURLsList(driver);
grabData(driver, URLs, "Laptop");
URLs.clear();
driver.get(listingsURL);
driver.findElement(By.xpath("//a [contains(#href,'pg_3')]")).click();
listingsURL = driver.getCurrentUrl();
updateURLsList(driver);
grabData(driver, URLs, "Laptop");
URLs.clear();
driver.get(listingsURL);
}
public static void updateURLsList(WebDriver driver)
{
//list of deals on amazon page
/////////////////////////////////////////////INCONSISTENT/////////////////////////////////////////////
List<WebElement> deals = driver.findElements(By.xpath(dealURLsXpath));
//////////////////////////////////////////////////////////////////////////////////////////////////////
System.out.println("Deals Size: " + deals.size());
for(WebElement element : deals)
{
URLs.add(element.getAttribute("href"));
}
System.out.println("URL List size: " + URLs.size());
deals.clear();
}
public static void grabData(WebDriver driver, List<String> URLs, String category)
{
for(String url : URLs)
{
driver.get(url);
String name = driver.findElement(By.xpath("//span [#id = \"productTitle\"]")).getText();
System.out.println("Name: " + name);
String price = driver.findElement(By.xpath("//span [#id = \"priceblock_ourprice\"]")).getText();
System.out.println("price: " + price);
String Xprice = driver.findElement(By.xpath("//span [#class = \"priceBlockStrikePriceString a-text-strike\"]")).getText();
System.out.println("Xprice: " + Xprice);
String picURL = driver.findElement(By.xpath("//img [#data-old-hires]")).getAttribute("src");
System.out.println("picURL: " + picURL);
BufferedImage img;
System.out.println("URL: " + url);
try
{
img = ImageIO.read(new URL(picURL));
products.add(new Product(
name,
Integer.parseInt(price.replaceAll("[^\\d.]", "").replace(".", "").replace(",", "")),
Integer.parseInt(Xprice.replaceAll("[^\\d.]", "").replace(".", "").replace(",", "")),
img,
category,
url));
}
catch(IOException e)
{
System.out.println("Error: " + e.getMessage());
}
}
To grab the href attribute of each Laptop that is on sale on the first 3 pages of this Amazon page you need to induce WebDriverWait for the visibilityOfAllElementsLocatedBy() and you can use the following Locator Strategy:
Code Block:
driver.get("https://www.amazon.com/s?i=computers&rh=n%3A565108%2Cp_72%3A1248879011&pf_rd_i=565108&pf_rd_p=b2e34a42-7eb2-50c2-8561-292e13c797df&pf_rd_r=CP4KYB71SY8E0WPHYJYA&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&qid=1590091272&ref=sr_pg_1");
List<WebElement> deals = new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//span[#class='a-price a-text-price']//parent::a[1]")));
for(WebElement deal:deals)
System.out.println(deal.getAttribute("href"));
Console Output:
https://www.amazon.com/Apple-MacBook-13-inch-256GB-Storage/dp/B08636NKF8/ref=sr_1_2?dchild=1&pf_rd_i=565108&pf_rd_p=b2e34a42-7eb2-50c2-8561-292e13c797df&pf_rd_r=CP4KYB71SY8E0WPHYJYA&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&qid=1590134317&refinements=p_72%3A1248879011&s=pc&sr=1-2
https://www.amazon.com/Apple-MacBook-16-Inch-512GB-Storage/dp/B081FZV45H/ref=sr_1_5?dchild=1&pf_rd_i=565108&pf_rd_p=b2e34a42-7eb2-50c2-8561-292e13c797df&pf_rd_r=CP4KYB71SY8E0WPHYJYA&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&qid=1590134317&refinements=p_72%3A1248879011&s=pc&sr=1-5
https://www.amazon.com/Apple-MacBook-13-inch-128GB-Storage/dp/B07V49KGVQ/ref=sr_1_9?dchild=1&pf_rd_i=565108&pf_rd_p=b2e34a42-7eb2-50c2-8561-292e13c797df&pf_rd_r=CP4KYB71SY8E0WPHYJYA&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&qid=1590134317&refinements=p_72%3A1248879011&s=pc&sr=1-9
https://www.amazon.com/New-Microsoft-Surface-Pro-Touch-Screen/dp/B07YNHXX8D/ref=sr_1_23?dchild=1&pf_rd_i=565108&pf_rd_p=b2e34a42-7eb2-50c2-8561-292e13c797df&pf_rd_r=CP4KYB71SY8E0WPHYJYA&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&qid=1590134317&refinements=p_72%3A1248879011&s=pc&sr=1-23
Similarly, Page 2 gives 4 and Page 3 gives 4 urls respectively.
You should try using wait in a selenium way:
WebDriverWait wait = new WebDriverWait(driver, 20);
List<WebElement> deals = wait.until(ExpectedConditions.presenceOfAllElementsLocatedBy(By.xpath(dealURLsXpath)));
System.setProperty("webdriver.chrome.driver", "C:\\Users\\Testing\\Downloads\\chromedriver_win32\\chromedriver.exe");
driver = new ChromeDriver();
driver.navigate().to("https://jpetstore.cfapps.io/catalog");
driver.findElement(By.xpath("//a[contains(text(),'Sign In')]")).click();
driver.findElement(By.name("username")).sendKeys("Testing6738788");
driver.findElement(By.name("password")).sendKeys("test#123");
driver.findElement(By.id("login")).click();
driver.findElement(By.xpath("//div[#id='SidebarContent']/a[contains(#href,'FISH')]/img")).click();
driver.findElement(By.xpath("//td[contains(text(),'Angelfish')]//preceding-sibling::td//a")).click();
List<WebElement> tablelist = driver.findElements(By.xpath("//div[#id='Catalog']//tr"));
for(int i = 0; i < tablelist.size(); i++)
{
String gotvalues = tablelist.get(i).getText();
System.out.println("Values got from the table " +gotvalues);
// Here im using split function but no luck
String[] splitword = gotvalues.split(" ");
for(String words : splitword)
{
System.out.println("Got single words from the split " + words);
// I want to compare the Large Angelfish value from the output
if(words.equalsIgnoreCase("Large Angelfish"))
{
System.out.println("Element present " + words);
}
}
}
Words should be split as "Item ID" -EST-1. I'm facing an issue with the description. The complete word is not getting displayed. How to write code to get item ID, product ID, and description?
Without String Array also you can verify.Try this code see if this help.Take input from keyboard.You have to type both values in console Like EST-1 then Enter and Then Large Angelfish and its compared later.Try Now.
Scanner scan= new Scanner(System.in);
String textID= scan.nextLine(); //Enter ID Here
String textDesc= scan.nextLine();//Enter Desc Here
System.setProperty("webdriver.chrome.driver", "C:\\Users\\Testing\\Downloads\\chromedriver_win32\\chromedriver.exe");
driver = new ChromeDriver();
driver.navigate().to("https://jpetstore.cfapps.io/catalog");
driver.findElement(By.xpath("//a[contains(text(),'Sign In')]")).click();
driver.findElement(By.name("username")).sendKeys("Testing6738788");
driver.findElement(By.name("password")).sendKeys("test#123");
driver.findElement(By.id("login")).click();
driver.findElement(By.xpath("//div[#id='SidebarContent']/a[contains(#href,'FISH')]/img")).click();
driver.findElement(By.xpath("//td[contains(text(),'Angelfish')]//preceding-sibling::td//a")).click();
List<WebElement> tablelist = driver.findElements(By.xpath("//div[#id='Catalog']//tr/td"));
System.out.println(tablelist.size());
for(int i=0;i<tablelist.size();i++)
{
String gotvalues = tablelist.get(0).getText();
String gotvaluesdesc = tablelist.get(2).getText();
// System.out.println("Values got from the table " +gotvalues );
if(gotvalues.trim().equalsIgnoreCase(textID) && gotvaluesdesc.trim().equalsIgnoreCase(textDesc))
{
System.out.println("Element present ID: " + gotvalues + " Desc :" + gotvaluesdesc);
break;
}
As you want to compare the table data, you need to modify your xPath little to fetch the data effectively so that you can avoid the splitting part and you can compare the data easily.
Try the below code :
String xPath = "//div[#id='Catalog']//tr";
List<WebElement> tableList = driver.findElements(By.xpath(xPath));
System.out.println("Item ID\t\tProduct ID\t\tDescription\t\tList Price\t\tOther");
System.out.println("--------------------------------------------------------------------------------------------------");
for(int i=1;i<tableList.size();i++) {
// Below line fetches/stores each table data as column wise
List<WebElement> listData = driver.findElements(By.xpath(xPath+"["+(i+1)+"]/td"));
for(int j=0;j<listData.size();j++) {
// Printing the column data
System.out.print(listData.get(j).getText()+"\t\t");
}
System.out.println();
}
// As per the above output, description is in the 3rd(2nd index) column so you can fetch that with the index number 2.
for(int i=1;i<tableList.size();i++) {
List<WebElement> listData = driver.findElements(By.xpath(xPath+"["+(i+1)+"]/td"));
if(listData.get(2).getText().trim().equals("Large Angelfish")) {
System.out.println("=> 'Large Angelfish' is Matching...");
}
if(listData.get(2).getText().trim().equals("Large Angelfish")) {
System.out.println("=> 'Small Angelfish' is Matching...");
}
}
If you execute the above code, it will print output as below :
Item ID Product ID Description List Price Other
----------------------------------------------------------------------------
EST-1 FI-SW-01 Large Angelfish $16.50 Add to Cart
EST-2 FI-SW-01 Small Angelfish $16.50 Add to Cart
In the above output, Description column number is 3 so you can substitute an index number in the below line for the corresponding column :
listData.get(2).getText().trim().equals("Large Angelfish")
I hope it helps...
I have a home work that requires user to input the name and process of products using for loop and print the price and name of the most expensive product. I have already figure out using the loop and transferring the variables to the constructor.
int numberOfProducts; //user input
for (i=0; i<=numberOfProducts; i++)
{
System.out.print("Name of product" + i);
System.out.println("price of product" +i);
Product myProduct= new Product (name, price);
//enter code here
}
I know I can write something like:
If max<price
price=max;
to find the max, but, have no idea how to incorporate the name when I print the maximum price.
Could you please give me a hint???
Thanks!
You just need to keep both the maximum price and the name of the product with that maximum price. For example,
Product[] products = // your products.
Product mostExpensiveProduct = product[0];
for (Product product : products) {
if (product.getPrice() > mostExpensiveProduct.getPrice()) {
mostExpensiveProduct = product;
}
}
System.out.println("Most expensive product is " + mostExpensiveProduct.getName() + " with price " + mostExpensiveProduct.getPrice());
Here is the link to print name and meaning columns of all pages using drop down
Try to build the script for following:
1. Go to http://babynames.merschat.com/index.cgi?function=Search&origin=Sanskrit&gender=f
2. print the name and meaning columns to syso.
I was able to print page 1 as it is a default page.
Here is the code:
public class BabyNamesAndMeanings {
WebDriver driver = new FirefoxDriver();
#BeforeClass
public void setUp() {
driver.get("http://babynames.merschat.com/index.cgi?function=Search&origin=Sanskrit&gender=f");
driver.manage().window().maximize();
}
#Test
public void printBabyNamesAndMeaningsOfFirstPage() {
WebElement baby_names = driver
.findElement(By
.xpath("//tbody/tr[7]/td[3]/table[2]/tbody/tr[2]/td[2]/font/table[1]/tbody"));
List<WebElement> names = baby_names.findElements(By
.xpath("//tr/td[1]/font/a"));
List<WebElement> meanings = baby_names.findElements(By
.xpath("//tr/td[4]/font/a"));
for (int i = 0; i < names.size(); i++) {
System.out.println("Name: " + names.get(i).getText()
+ " Meaning: " + meanings.get(i).getText());
}
}
I don't know how to loop through rest of the options in the drop down list at the bottom of the page and hit submit button to print name and meaning of all the pages.
There are 100+ pages.
Thanks in advance.
The code below will do your job.
driver.get("http://babynames.merschat.com/index.cgi?function=Search&origin=Sanskrit&gender=f");
List<WebElement> pageOptions = new Select(driver.findElement(By.xpath("//select[#name='page']"))).getOptions();//Get all options in dropdown
ArrayList<String> pageDd = new ArrayList<String>();
for(WebElement eachPage:pageOptions){
pageDd.add(eachPage.getText());//Save text of each option
}
int i=1;
for(String eachVal:pageDd){
new Select(driver.findElement(By.xpath("//select[#name='page']"))).selectByVisibleText(eachVal);//Select page
driver.findElement(By.xpath("//input[#value='Go']")).click();//Click on go
List<WebElement> names = driver.findElements(By.xpath("//a[contains(#title,' meanings and popularity')]"));//Get all names on page
for(WebElement eachName:names){
String name = eachName.getText(); //Get each name's text
WebElement mean = eachName.findElement(By.xpath("./../../..//a[contains(#title,'Names for baby with meanings like ')]"));//Get meaning for that name
String meaning = mean.getText();//Get text of meaning
System.out.println(i+") Name: " +name+ " Meaning: " + meaning);//Print the data
i++;
}
}
Try and understand the way requirement is achieved. If you have any doubt ask.
Another method to iterate and select all the Dropdown values
Select dropdown= new Select(WebUIDriver.webDr.findElement(By.xpath("enter xpath")));
int noOfDropDownValues= dropdown.getOptions().size()-1;
for(int i=0;i<noOfDropDownValues;i++){
new Select(WebUIDriver.webDr.findElement(By.xpath("Enter Xpath']"))).selectByValue(String.valueOf(i));
}