Develop Reference

UnexpectedTagNameException: Element should have been "select" but was "a" while trying to get the texts of dropdown menu using Selenium and Java

While trying to get the menu list, I'm getting this error message:
Exception in thread "main" org.openqa.selenium.support.ui.UnexpectedTagNameException: Element should have been "select" but was "a".
Here below is the code:
public static void main(String[] args) {
// TODO Auto-generated method stub
System.setProperty("webdriver.chrome.driver", "D:\\selenium files\\chromedriver_win32_new\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.manage().window().maximize();
driver.get("https://www.tutorialspoint.com/tutor_connect/index.php");
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
WebElement ele = driver.findElement(By.xpath("//*[#id=\"logo-menu\"]/div/div[1]/div/a"));
Select s = new Select(ele);
//getting list of menu
List <WebElement> op = s.getOptions();
int size = op.size();
for(int i =0; i<size ; i++){
String options = op.get(i).getText();
System.out.println(options);
}
}
}

That is because the element you are trying to cast is a link tag and not a select tag.
You need to give the Xpath or CSS of the correct Select element and then cast it from WebElement into a Select ojbect.
In the example you are using there is not real selector, you first need to click on the buttons that says "Categories" and later take the options that appear:
WebElement button = driver.findElementByCSS("div[class='mui-dropdown']");
button.click();
WebElement SelectObj = driver.findElementByCSS("ul[class*='mui--is-open']");
Select s = new Select(SelectObj);

The desired element is not a <select> element but a <ul> element. Once you click on the <a> element then only the classname mui--is-open is appended to the desired <ul> element.
Solution
So to get the contents of the dropdown menu you need to induce WebDriverWait for the visibilityOfAllElementsLocatedBy() and you can use Java8 stream() and map() and you can use either of the following Locator Strategies:
Using cssSelector:
new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.cssSelector("a.mui-btn.mui-btn--primary.categories"))).click();
System.out.println(new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.cssSelector("ul.mui-dropdown__menu.cat-menu.mui--is-open a"))).stream().map(element->element.getText()).collect(Collectors.toList()));
Using xpath:
new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.xpath("//a[#class='mui-btn mui-btn--primary categories']"))).click();
System.out.println(new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//ul[#class='mui-dropdown__menu cat-menu mui--is-open']//a"))).stream().map(element->element.getText()).collect(Collectors.toList()));
References
You can find a couple of relevant detailed discussions in:
How to extract the text iterating specific rows within a table using XPath with Selenium and Java
How to extract the dynamic values of the id attributes of the table elements using Selenium and Java
How to print runs scored by a batsmen in a scoreboard format webelements through CSS selector using Selenium and Java

how to get the id from the Xpath as a text

I had taken the xPath as below but I am not able to get the value 145666 when I try to print.
WebDriverWait wait = new WebDriverWait(driver, 20);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//span[contains(#class,'mat-content')]//div[#id='demandTrackingID']")));
Now iam getting the below Expception :
org.openqa.selenium.TimeoutException: Expected condition failed: waiting for visibility of element located by By.xpath: //span[contains(#class,'mat-content')]//div[#id='TrackID']
Note: This Html Element is a Non Visible Element I need to inspect the panel and get this element as there is no field for this Element
HTML:
<div _ngcontent-wbh-c179="" id="TrackID" class="disp-none">14566 </div>

Please try this:
WebElement trackid= driver.findElement(By.xpath("//span[contains(#class,'mat-content')]//div[#id='TrackID']"));
System.out.println("getid:"+ trackid.getText());
You will possible need to add an explicit wait before that to make element fully loaded before retrieving it's content
WebDriverWait wait = new WebDriverWait(driver, 30);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//span[contains(#class,'mat-content')]//div[#id='TrackID']")));
WebElement trackid= driver.findElement(By.xpath("//span[contains(#class,'mat-content')]//div[#id='TrackID']"));
System.out.println("getid:"+ trackid.getText());

id is ID not id
so use this :
WebElement trackid= driver.findElement(By.id("TrackID"));
System.out.println("getid:"+ trackid.getText());

The element that contains the text you want has an ID, TrackID, not to be confused with the "ID" text on the page. You can use that to locate the element more easily than the XPath you are currently using. Once you have the element, you can use WebDriverWait to wait for the element to contain text. Code is below.
WebDriverWait wait = new WebDriverWait(driver, 10);
wait.until((ExpectedCondition<Boolean>) d -> d.findElement(By.id("TrackID")).getText().length() != 0);
Given the class class="disp-none" on the element, I'm guessing that the element might not be visible. If this is the case, you'll have to use JavaScript to get the invisible text. Selenium was designed to only interact with visible elements and will throw an exception if you try to interact with elements that are not visible.
WebDriverWait wait = new WebDriverWait(driver, 10);
wait.until((ExpectedCondition<Boolean>) d -> d.findElement(By.id("TrackID")).getText().length() != 0);
WebElement element = driver.findElement(By.id("TrackID"));
String trackID = (JavascriptExecutor)driver.executeScript("return arguments[0].text", element);

selenium issue trying to retrieve text from a div using driver.findElement(By.xpath()

I'm trying to verify a few things on a page and have a grid with header columns. I want to grab the text from those are part of the verification. For some reason it's returning empty and I don't understand why considering I'm using contains text to find the element.
String expectedName = "Employee Name";
String actualName = driver.findElement(By.xpath("//div[contains(text(), 'Employee Name')]")).getText();
Assert.assertEquals(expectedName, actualName);
The error I get is this: java.lang.AssertionError: expected [Employee Name] but found []
The HTML:
<div class="dx-datagrid-text-content dx-text-content-alignment-left dx-header-filter-indicator" role="presentation">Employee Name</div>

You seem to be close. To extract the textContent of the <div> element ideally you need to induce WebDriverWait for the visibilityOfElementLocated() and you can use either of the following Locator Strategies:
cssSelector:
String expectedName = "Employee Name";
String actualName = new WebDriverWait(driver, 10).until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("div.dx-datagrid-text-content.dx-text-content-alignment-left.dx-header-filter-indicator"))).getText();
Assert.assertEquals(expectedName, actualName);
xpath:
String expectedName = "Employee Name";
String actualName = new WebDriverWait(driver, 10).until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//div[#class='dx-datagrid-text-content dx-text-content-alignment-left dx-header-filter-indicator']"))).getText();
Assert.assertEquals(expectedName, actualName);

How to interact with the elements within #shadow-root (open) while Clearing Browsing Data of Chrome Browser using cssSelector

I had been following the discussion How to automate shadow DOM elements using selenium? to work with #shadow-root (open) elements.
While in the process of locating the Clear data button within the Clear browsing data popup, which appears while accessing the url chrome://settings/clearBrowserData through Selenium I am unable to locate the following element:
#shadow-root (open)
<settings-privacy-page>
Snapshot:
Using Selenium following are my code trials and the associated errors encountered:
Attempt 1:
WebElement root5 = shadow_root4.findElement(By.tagName("settings-privacy-page"));
Error:
Exception in thread "main" org.openqa.selenium.JavascriptException: javascript error: b.getElementsByTagName is not a function
Attempt 2:
WebElement root5 = shadow_root4.findElement(By.cssSelector("settings-privacy-page"));
Error:
Exception in thread "main" org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"settings-privacy-page"}
Attempt 3:
WebElement root5 = (WebElement)((JavascriptExecutor)shadow_root4).executeScript("return document.getElementsByTagName('settings-privacy-page')[0]");
Error:
Exception in thread "main" java.lang.ClassCastException: org.openqa.selenium.remote.RemoteWebElement cannot be cast to org.openqa.selenium.JavascriptExecutor
Incase if it is helpful the initial code block (till the above line) works perfect:
driver.get("chrome://settings/clearBrowserData");
WebElement root1 = driver.findElement(By.tagName("settings-ui"));
WebElement shadow_root1 = expand_shadow_element(root1);
WebElement root2 = shadow_root1.findElement(By.cssSelector("settings-main#main"));
WebElement shadow_root2 = expand_shadow_element(root2);
WebElement root3 = shadow_root2.findElement(By.cssSelector("settings-basic-page[role='main']"));
WebElement shadow_root3 = expand_shadow_element(root3);
WebElement root4 = shadow_root3.findElement(By.cssSelector("settings-section[page-title='Privacy and security']"));
WebElement shadow_root4 = expand_shadow_element(root4);
PS: expand_shadow_element() works flawless.

If you are trying to get 'Clear Data' element then you can use the below js to get the element and then perform.
return document.querySelector('settings-ui').shadowRoot.querySelector('settings-main').shadowRoot.querySelector('settings-basic-page').shadowRoot.querySelector('settings-section > settings-privacy-page').shadowRoot.querySelector('settings-clear-browsing-data-dialog').shadowRoot.querySelector('#clearBrowsingDataDialog').querySelector('#clearBrowsingDataConfirm')
Here is the sample script.
driver.get("chrome://settings/clearBrowserData");
driver.manage().window().maximize();
JavascriptExecutor js = (JavascriptExecutor) driver;
WebElement clearData = (WebElement) js.executeScript("return document.querySelector('settings-ui').shadowRoot.querySelector('settings-main').shadowRoot.querySelector('settings-basic-page').shadowRoot.querySelector('settings-section > settings-privacy-page').shadowRoot.querySelector('settings-clear-browsing-data-dialog').shadowRoot.querySelector('#clearBrowsingDataDialog').querySelector('#clearBrowsingDataConfirm')");
// now you can click on clear data button
clearData.click();
Edit 2: Explanation
Problem: Selenium does not provide explicit support to work with Shadow DOM elements, as they are not in the current dom. That's the reason why we will get NoSuchElementException exception when try to access the elements in the shadow dom.
Shadow DOM:
Note: We will be referring to the terms shown in the picture. So please go through the picture for better understanding.
Solution:
In order to work with shadow element first we have to find the shadow host to which the shadow dom is attached. Here is the simple method to get the shadow root based on the shadowHost.
private static WebElement getShadowRoot(WebDriver driver,WebElement shadowHost) {
JavascriptExecutor js = (JavascriptExecutor) driver;
return (WebElement) js.executeScript("return arguments[0].shadowRoot", shadowHost);
}
And then you can access the shadow tree element using the shadowRoot Element.
// get the shadowHost in the original dom using findElement
WebElement shadowHost = driver.findElement(By.cssSelector("shadowHost_CSS"));
// get the shadow root
WebElement shadowRoot = getShadowRoot(driver,shadowHost);
// access shadow tree element
WebElement shadowTreeElement = shadowRoot.findElement(By.cssSelector("shadow_tree_element_css"));
In order to simplify all the above steps created the below method.
public static WebElement getShadowElement(WebDriver driver,WebElement shadowHost, String cssOfShadowElement) {
WebElement shardowRoot = getShadowRoot(driver, shadowHost);
return shardowRoot.findElement(By.cssSelector(cssOfShadowElement));
}
Now you can get the shadowTree Element with single method call
WebElement shadowHost = driver.findElement(By.cssSelector("shadowHost_CSS_Goes_here));
WebElement shadowTreeElement = getShadowElement(driver,shadowHost,"shadow_tree_element_css");
And perform the operations as usual like .click(), .getText().
shadowTreeElement.click()
This Looks simple when you have only one level of shadow DOM. But here, in this case we have multiple levels of shadow doms. So we have to access the element by reaching each shadow host and root.
Below is the snippet using the methods that mentioned above (getShadowElement and getShadowRoot)
// Locate shadowHost on the current dom
WebElement shadowHostL1 = driver.findElement(By.cssSelector("settings-ui"));
// now locate the shadowElement by traversing all shadow levels
WebElement shadowElementL1 = getShadowElement(driver, shadowHostL1, "settings-main");
WebElement shadowElementL2 = getShadowElement(driver, shadowElementL1,"settings-basic-page");
WebElement shadowElementL3 = getShadowElement(driver, shadowElementL2,"settings-section > settings-privacy-page");
WebElement shadowElementL4 = getShadowElement(driver, shadowElementL3,"settings-clear-browsing-data-dialog");
WebElement shadowElementL5 = getShadowElement(driver, shadowElementL4,"#clearBrowsingDataDialog");
WebElement clearData = shadowElementL5.findElement(By.cssSelector("#clearBrowsingDataConfirm"));
System.out.println(clearData.getText());
clearData.click();
You can achieve all the above steps in single js call as at mentioned at the beginning of the answer (added below just to reduce the confusion).
WebElement clearData = (WebElement) js.executeScript("return document.querySelector('settings-ui').shadowRoot.querySelector('settings-main').shadowRoot.querySelector('settings-basic-page').shadowRoot.querySelector('settings-section > settings-privacy-page').shadowRoot.querySelector('settings-clear-browsing-data-dialog').shadowRoot.querySelector('#clearBrowsingDataDialog').querySelector('#clearBrowsingDataConfirm')");
Screenshot:

I had to do a similar test which required clearing browsing the chrome history. A minor difference was that I was clearing the data after going to the advanced section of the pop-up. As you are struggling to click only the "Clear data" button, I'm quite sure that you've missed one or two hierarchy elements mistakenly. Or got confused between sibling and parent elements probably. As per seeing your code, I assume that you already know that to access a particular shadow DOM element you need proper sequencing and it has been explained also quite nicely above.
Coming right at your problem now, here is my code snippet which is working correctly. The code waits until the data is cleaned and then will proceed to your next action-
public WebElement expandRootElement(WebElement element) {
WebElement ele = (WebElement) ((JavascriptExecutor) driver).executeScript("return arguments[0].shadowRoot",
element);
return ele;
}
public void clearBrowsingHistory() throws Exception {
WebDriverWait wait = new WebDriverWait(driver, 15);
driver.get("chrome://settings/clearBrowserData");
// Get shadow root elements
WebElement shadowRoot1 = expandRootElement(driver.findElement(By.xpath("/html/body/settings-ui")));
WebElement root2 = shadowRoot1.findElement(By.cssSelector("settings-main"));
WebElement shadowRoot2 = expandRootElement(root2);
WebElement root3 = shadowRoot2.findElement(By.cssSelector("settings-basic-page"));
WebElement shadowRoot3 = expandRootElement(root3);
WebElement root4 = shadowRoot3
.findElement(By.cssSelector("#advancedPage > settings-section > settings-privacy-page"));
WebElement shadowRoot4 = expandRootElement(root4);
WebElement root5 = shadowRoot4.findElement(By.cssSelector("settings-clear-browsing-data-dialog"));
WebElement shadowRoot5 = expandRootElement(root5);
WebElement root6 = shadowRoot5
.findElement(By.cssSelector("cr-dialog div[slot ='button-container'] #clearBrowsingDataConfirm"));
root6.click();
wait.until(ExpectedConditions.invisibilityOf(root6));
}
It should work properly in your case too if you don't intend to change any of the options selected by default in the pop-up (In that case, you will have to add a few more codes regarding selecting those checkboxes). Please tell me if this solves your issue. Hope this is helpful
I've added a snapshot of the the screen here too-
image

The Locator Strategy in #supputuri's answer using document.querySelector() works perfect through google-chrome-devtools
However, as the desired element opens from the shadow-dom you need to induce WebDriverWait for the elementToBeClickable() and you can you the following solution:
Code Block:
driver.get("chrome://settings/clearBrowserData");
new WebDriverWait(driver, 5).until(ExpectedConditions.elementToBeClickable((WebElement) ((JavascriptExecutor)driver).executeScript("return document.querySelector('settings-ui').shadowRoot.querySelector('settings-main').shadowRoot.querySelector('settings-basic-page').shadowRoot.querySelector('settings-section > settings-privacy-page').shadowRoot.querySelector('settings-clear-browsing-data-dialog').shadowRoot.querySelector('#clearBrowsingDataDialog').querySelector('#clearBrowsingDataConfirm')"))).click();
System.out.println("Clear data Button Clicked");
Console Output:
Clear data Button Clicked

I was getting InvalidArgumentEXception when trying to identify shadowRoot element in DOM using Selenium 4.3.0 and Chrome Version 103.0.5060.134
The solution to this is
SearchContext se= driver.findElment(By.locator("...").getShadowRoot(); return type is SearchContext
in the above line try using locator as xpath
and secondly trying to locate element using SearchContext reference e.g.
WebElement we= se.findElement(By.locator("....."));
use locater as cssSelector
And boom it works like charm
Didn't find this solution available and took me half a day to figure out
Hope this helps!!!

How to extract the 'src' attribute as per the html through Selenium and Java

I have a html page:
<video crossorigin="anonymous" class="" id="video" playsinline="true"
src="https://df.dfs.bnt.com/
DEEAB832j06E9j413FjAA8Dj2zxc535DA2072E3jW01j15579/mp4-
hi/jFNf2IbBoGF28IzyU_WqkA,1535144598/zxxx/
contents/1/8/1a57ae021173751b468cca136e0192.mp4?
rnd=0.38664488150364296">
</video>
Through Selenium WebDriver I tried get video url:
By videoLocator = By.id("video");
WebElement videoElement = driver.findElement(videoLocator);
String videoUrl = videoElement.getAttribute("src");
But videoUrl - always returns ""(is empty).
Нowever for example:
videoElement.getAttribute("crossorigin")
return correct answer: "anonymous".
I have tried get this url from src attribute using javascript:
String videoUrl = (String) js.executeScript("return document.getElementById( 'video' ).getAttribute( 'src' );");
But the result is still the same: "".
I guess that the problem is in crossorigin="anonymous" but what to do with it? How can I get src value?
Sorry, for my poor English.

As per the HTML you have provided you need to induce WebDriverWait and you can use the following solution:
WebElement videoElement = new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//video[#id='video' and #crossorigin='anonymous'][starts-with(#src,'http')]")));
System.out.println(videoElement.getAttribute("src"));

Try fetching innerHTML attribute. Code :
By videoLocator = By.id("video");
WebElement videoElement = driver.findElement(videoLocator);
String videoUrl = videoElement.getAttribute("innerHTML");