Develop Reference

How do i loop through divs using jsoup

Hi guys I'm using jsoup in a java webapplication on IntelliJ. I'm trying to scrape data of port call events from a shiptracking website and store the data in a mySQL database. The data for the events is organised in divs with the class name table-group and the values are in another div with the class name table-row. My problem is the divs rows for all the vessel are all the same class name and im trying to loop through each row and push the data to a database. So far i have managed to create a java class to scrape the first row. How can i loop through each row and store those values to my database. Should i create an array list to store the values?
this is my scraper class
public class Scarper {
private static Document doc;
public static void main(String[] args) {
final String url =
"https://www.myshiptracking.com/ports-arrivals-departures/?mmsi=&pid=277&type=0&time=&pp=20";
try {
doc = Jsoup.connect(url).get();
} catch (IOException e) {
e.printStackTrace();
}
Events();
}
public static void Events() {
Elements elm = doc.select("div.table-group:nth-of-type(2) > .table-row");
List<String> arrayList = new ArrayList();
for (Element ele : elm) {
String event = ele.select("div.col:nth-of-type(2)").text();
String time = ele.select("div.col:nth-of-type(3)").text();
String port = ele.select("div.col:nth-of-type(4)").text();
String vessel = ele.select(".td_vesseltype.col").text();
Event ev = new Event();
System.out.println(event);
System.out.println(time);
System.out.println(port);
System.out.println(vessel);
}
}
}
sample of the div classes i want to scrape
<div style="box-sizing: border-box;padding: 0px 10px 10px 10px;">
<div class="cs-table">
<div class="heading">
<div class="col" style="width: 10px"></div>
<div class="col" style="width: 110px">Event</div>
<div class="col" style="width: 120px">Time (<span class="tooltip" title="My Time: In your current TimeZone">MT</span>)</div>
<div class="col" style="width: 150px">Port</div>
<div class="col">Vessel</div>
</div>
<div class="table-group">
<div class="table-row">
<div class="col"><i class="fa fa-sign-out red"></i></div>
<div class="col">Departure</div>
<div class="col" style="text-align: center;">2022-02-14 <b>16:51</b></div>
<div class="col"><img class="flag_line tooltip" src="/icons/flags2/16/GB.png" title=" United Kingdom"/>BELFAST</div>
<div class="col td_vesseltype"><img src="/icons/icon7_511.png"><span class="padding_18">WILSON BLYTH [GB]</span></div>
</div>
</div>
<div class="table-group">
<div class="table-row">
<div class="col"><i class="fa fa-flag-checkered green"></i></div>
<div class="col">Arrival</div>
<div class="col" style="text-align: center;">2022-02-14 <b>16:51</b></div>
<div class="col"><img class="flag_line tooltip" src="/icons/flags2/16/GB.png" title=" United Kingdom"/>HUNTERS QUAY</div>
<div class="col td_vesseltype"><img src="/icons/icon6_511.png"><span class="padding_18">SOUND OF SOAY [GB]</span></div>
</div>
</div>
<div class="table-group">
<div class="table-row">
<div class="col"><i class="fa fa-sign-out red"></i></div>
<div class="col">Departure</div>
<div class="col" style="text-align: center;">2022-02-14 <b>16:51</b></div>
<div class="col"><img class="flag_line tooltip" src="/icons/flags2/16/GB.png" title=" United Kingdom"/>LARGS</div>
<div class="col td_vesseltype"><img src="/icons/icon6_511.png"><span class="padding_18">LOCH SHIRA [GB]</span></div>
</div>
</div>
<div class="table-group">
<div class="table-row">
<div class="col"><i class="fa fa-sign-out red"></i></div>
<div class="col">Departure</div>
<div class="col" style="text-align: center;">2022-02-14 <b>16:51</b></div>
<div class="col"><img class="flag_line tooltip" src="/icons/flags2/16/GB.png" title=" United Kingdom"/>RYDE</div>
<div class="col td_vesseltype"><img src="/icons/icon4_511.png"><span class="padding_18">ISLAND FLYER [GB]</span></div>
</div>
</div>

You can start with looping over the table's rows: the selector for the table is .cs-table so you can get the table with Element table = doc.select(".cs-table").first();. Next you can get the table's rows with the selector div.table-row - Elements rows = doc.select("div.table-row"); now you can loop over all the rows and extract the data from each row. The code should look like:
Element table = doc.select(".cs-table").first();
Elements rows = doc.select("div.table-row");
for (Element row : rows) {
String event = row.select("div.col:nth-of-type(2)").text();
String time = row.select("div.col:nth-of-type(3)").text();
String port = row.select("div.col:nth-of-type(4)").text();
String vessel = row.select(".td_vesseltype.col").text();
System.out.println(event + "-" + time + " " + port + " " + vessel);
System.out.println("---------------------------");
// Do stuff with data here
}
Now it's up to you to decide if you want to keep the data in some array/list inside the loop and use it later, or to insert it directly to your database.

Can't select Multiple Dropdown with Selenium webdriver (not class Select)

I'm having a problem with my tests with Selenium webdriver. I'm using Java. Can't select from a multiple drop-down that is not class Select. This is how the drop-down looks like:
Drop-Down picture
And that's the code:
<div class="form-group ">
<label for="CurrentCategoriesNomIds-selectized">Categories</label>
<select placeholder="" multiple="multiple" id="CurrentCategoriesNomIds" name="CurrentCategoriesNomIds" tabindex="-1" class="selectized" style="display: none;">
<option value="325" selected="selected">Education</option>
</select>
<div class="selectize-control multi plugin-remove_button">
<div class="selectize-input items not-full has-options has-items">
<div class="item" data-value="325">
Education
×
</div>
<input type="text" autocomplete="off" tabindex="" id="CurrentCategoriesNomIds-selectized" style="width: 4px; opacity: 1; position: relative; left: 0px;"></div>
<div class="selectize-dropdown multi plugin-remove_button" style="display: none; visibility: visible; width: 800px; top: 36px; left: 0px;">
<div class="selectize-dropdown-content">
<div class="option" data-selectable="" data-value="324">Agriculture</div>
<div class="option" data-selectable="" data-value="298">Culture</div>
<div class="option" data-selectable="" data-value="326">Employment</div>
<div class="option" data-selectable="" data-value="323">Environment</div>
<div class="option" data-selectable="" data-value="327">Other</div>
<div class="option" data-selectable="" data-value="297">Political</div>
<div class="option" data-selectable="" data-value="322">Transport</div>
</div>
</div>
</div>
</div>
This is how it looks like when 2 options are selected. I was wondering if I can try with KEYS but the page doesn't work like that. Haven't seen that kind of field before, and not sure how to proceed?

You can click on dropdown using this code :
public static void selectOption(WebDriver driver, String optionName) {
List<WebElement> options = driver.findElements(By.xpath("//div[#class='selectize-dropdoun-content']//div[#class='option']"));
options.forEach(option -> {
if (option.getAttribute("innerText").equals(optionName)) {
Actions actions = new Actions(driver);
actions.moveToElement(option).click().build().perform();
}
});
}
and then use like this:
String option = "Education";
selectOption(driver,option);
Hope that helps you:)
Adding screenshot for what I have tried on website : https://semantic-ui.com/modules/dropdown.html

I don't much use Java, so I'll be writing some pseudo code for this that should give you an outline on how to achieve it (but may not run as written).
public static void selectOptionFromSelectizeDropdown(String optionText, String dropdown){
boolean completed = false;
int numberOfOptions = driver.findElements(By.css(dropdown + " .option")).length
for(int i = 0; i < numberOfOptions && completed === false; i++){
// Check if it's displayed, if it is, HUZZAH! Click the option
if(driver.findElement(By.xpath('//*/*[contains(#class, "option") and contains(text(), "'+optionText+'")])')).isDisplayed()){
driver.findElement(By.xpath('//*/*[contains(#class, "option") and contains(text(), "'+optionText+'")])')).click();
completed === true;
break;
} else {
// In case there are many options, and you have to scroll through them.
int x = 0;
while(x <= 6){
driver.findElement(By.css(dropdown)).sendKeys(Keys.DOWN);
i++;
}
}
if(i===numberOfOptions - 1){
throw new Error("Option Not Found");
}
}
}
selectOptionFromSelectizeDropdown("Education", ".selectize-dropdown-content");
If this doesn't work, I'd recommend changing the click() to a sendKeys(Key.ENTER) to see if that would work.
Explanation
Will loop through, seeing if the option is displayed on the page. If not, will scroll down x times, and check again, until the option is found.
If it reaches the number of options inside the box, it will throw an error.

Selenium WebDriver Java - I'm not able to get dynamic text from span

static By yearFoundErrorMessage = By.className("error");
public static WebElement ca1_YearFoundError(WebDriver driver){
element = driver.findElement(By.id("yearFoundErrorMessage"));
return element;
}
//Main class from where I'm calling above function
String errorMessage = Ca1GeneralInformation.ca1_YearFoundError(driver).getText();
System.out.println(errorMessage);
<div class="controls">
<input id="ca1_GEN_foundedDate" class="error" type="text" value="2008" maxlength="4" name="data[CaInformation][ca1_GEN_foundedDate]" tabindex="5" unsaved="true">
<span id="ca1_GEN_foundedDate_error" class="inline-error ajax-failed-msg">
<span class="error" for="ca1_GEN_foundedDate">Please enter a valid year (i.e. 2013)</span>
</span>
</div>
I'm working on webapplication where there is error message in span I want to get text of this span using webdriver. Above is my html and code.Please help me out.
thanks in advance.

yearFoundErrorMessage is a By variable but you are sending it as string to another By. Try this
public static WebElement ca1_YearFoundError(WebDriver driver) {
element = driver.findElement(yearFoundErrorMessage);
return element;
}

Try below xPath :
WebElement element = driver.findElement(By.xpath("//span[contains(text(),'Please enter a valid year')]");
String message = element.getText();

Getting list of items inside div using Selenium Webdriver

Say I have the following
<div class="facetContainerDiv">
<div>
<label class="facetLabel">
<input class="facetCheck" type="checkbox" />
</label>
<label class="facetLabel">
<input class="facetCheck" type="checkbox" />
</label>
<label class="facetLabel">
<input class="facetCheck" type="checkbox" />
</label>
<label class="facetLabel">
<input class="facetCheck" type="checkbox" />
</label>
<label class="facetLabel">
<input class="facetCheck" type="checkbox" />
</label>
</div>
</div>
Now I want to put a check mark on the checkbox based on the index I provide. So I write a method like below
How do I access all elements inside the div class="facetContainerDiv" ?
I tried
List<WebElements> elementsList = driver.findElements(By.cssSelector(".facetContainerDiv"));
for(WebElement checkBox:elementsList) {
int i=0;
checkBox = elementsList.get(i);
bla bla bla..
}
in the above code elementsList has only one element with "type" as null.

Follow the code below exactly matched with your case.
Create an interface of the web element for the div under div with class as facetContainerDiv
ie for
<div class="facetContainerDiv">
<div>
</div>
</div>
2. Create an IList with all the elements inside the second div i.e for,
<label class="facetLabel">
<input class="facetCheck" type="checkbox" />
</label>
<label class="facetLabel">
<input class="facetCheck" type="checkbox" />
</label>
<label class="facetLabel">
<input class="facetCheck" type="checkbox" />
</label>
<label class="facetLabel">
<input class="facetCheck" type="checkbox" />
</label>
<label class="facetLabel">
<input class="facetCheck" type="checkbox" />
</label>
3. Access each check boxes using the index
Please find the code below
using System;
using System.Collections.Generic;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.Support.UI;
namespace SeleniumTests
{
class ChechBoxClickWthIndex
{
static void Main(string[] args)
{
IWebDriver driver = new FirefoxDriver();
driver.Navigate().GoToUrl("file:///C:/Users/chery/Desktop/CheckBox.html");
// Create an interface WebElement of the div under div with **class as facetContainerDiv**
IWebElement WebElement = driver.FindElement(By.XPath("//div[#class='facetContainerDiv']/div"));
// Create an IList and intialize it with all the elements of div under div with **class as facetContainerDiv**
IList<IWebElement> AllCheckBoxes = WebElement.FindElements(By.XPath("//label/input"));
int RowCount = AllCheckBoxes.Count;
for (int i = 0; i < RowCount; i++)
{
// Check the check boxes based on index
AllCheckBoxes[i].Click();
}
Console.WriteLine(RowCount);
Console.ReadLine();
}
}
}

I'm not sure if your findElements statement gets you all the divs. I would try the following:
List<WebElement> elementsRoot = driver.findElements(By.xpath("//div[#class=\"facetContainerDiv\"]/div));
for(int i = 0; i < elementsRoot.size(); ++i) {
WebElement checkbox = elementsRoot.get(i).findElement(By.xpath("./label/input"));
checkbox.click();
blah blah blah
}
The idea here is that you get the root element then use another a 'sub' xpath or any selector you like to find the node element. Of course the xpath or selector may need to be adjusted to properly find the element you want.

You're asking for all the elements of class facetContainerDiv, of which there is only one (your outer-most div). Why not do
List<WebElement> checks = driver.findElements(By.class("facetCheck"));
// click the 3rd checkbox
checks.get(2).click();

alternatively, you can try writing a specific element:
//label[1] is the first element.
el = await driver.findElement(By.xpath("//div[#class=\"facetContainerDiv\"]/div/label[1]/input")));
await el.click();
More information can be found here: https://www.browserstack.com/guide/locators-in-selenium

How to select an option from a drop-down using Selenium WebDriver with Java?

I need to select a category from the below html.
I tried different options, and different ways showed in the blogs, but not able to select the options.
Any help will be appreciated.
One way I am using as (Not a better one)
private boolean select_dropdown_xpath(String value, String seleniumObjectValue) {
try {
boolean isListItemFound = false;
int i = 0;
do {
i++;
String category = driver.findElement(By.xpath(seleniumObjectValue+"/div["+ i +"]")).getText();
if(category.equals(value)) {
driver.findElement(By.xpath(seleniumObjectValue+"/div["+ i +"]")).click();
isListItemFound = true;
}
} while (isListItemFound == false);
if(!(isListItemFound)) {
return false;
}
} catch(Exception e) {
return false;
}
return true;
}
Tool: Selenium WebDriver 2.28
with Java
Thanks
Purna
HTML:
<div class="drop-down">
<div class="label_field">
<label>Category:</label>
<fieldset>
<div id="Ccategory" class="jSym_select_element jSym_pie jSym_noSelectText false hover" tabindex="0" textval="Default" style="width: 350px;">
<div class="jSym_drop_arrow false"/>
<div class="jSym_select_inner false">Default</div>
</div>
<div id="selectDrop" class="jSym_select_drop jSym_noSelectText " style="height: 50px; width: 350px; margin-top: 10px;">
<div class="jSym_select_item jSym_noSelectText" optionval="Default">Default</div>
<div class="jSym_select_item jSym_noSelectText" optionval="Reset">Reset</div>
</div>
<select id="select_category" class="jSym_dropdown" name="category" style="visibility: hidden;">
<option value="Default">Default</option>
<option value="Reset">Reset</option>
</select>
</fieldset>
</div>
</div>

Try this code:
Select sele = new Select(driver.findElement(By.id("select_category")));
//Select the dropdown by using the displayed value.
sele.selectByVisibleText(`displayed value`);
//or you can Select the dropdown by using the index value.
sele.selectByIndex(`index value`);
//or you can Select the dropdown by using the value attribute.
sele.selectByIndex(`value in the value attribute`);
In your case the dropdown visibility is hidden. So, first make it as visible by using JavaScript Executor class. Then use the above code.