I'm trying to fill in multiple forms that come after each other, all the forms get filled swiftly with no errors because I make sure to add
WebDriverWait wait = new WebDriverWait(driver, 20);
wait.until(ExpectedConditions.elementToBeClickable(By.xpath("")));
before doing anything on a new page, and I know I'm on the correct page.
On the last form, I encounter this error :
Exception in thread "main" org.openqa.selenium.NoSuchElementException: Unable to locate element: //*[#id="formtovalidate"]/fieldset[1]/div/label/input For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html
So I went to check on the browser by taking a screenshot and the browser is on the correct page with the correct form, I also checked the xpath values and even tried other attributes.. nothing seemed to work.
So I went ahead and printed out the PageSource which showed a totally different page (not the previous page), I also noticed the this page flashed for a second before the final form appeared.
I also tried driver.navigate().refresh() but that didn't work. I kept searching and looking but nothing appeared. I also changed browsers, that did nothing..
This is the method I'm trying to execute:
private void method() {
WebDriverWait wait = new WebDriverWait(driver, 20);
wait.until(ExpectedConditions.elementToBeClickable(By.xpath("//*[#id=\"formtovalidate\"]/fieldset[1]/div/label/input")));
driver.findElement(By.xpath("//*[#id=\"formtovalidate\"]/fieldset[1]/div/label/input")).sendKeys(email); }
Update
Here's the form screenshot:
Here's the execution results:
Code:
String body_text = driver.findElement(By.tagName("body")).getText();
System.out.println(body_text);
Result: The form but in text
Code:
String body_innerHTML = driver.findElement(By.tagName("body")).getAttribute("innerHTML");
System.out.println(body_innerHTML);
Result: A different page :(
<zendesk-ticketing-form base-url="https://www.runescape.com/a=870/c=K0aO9WO69EI" css-cachebust="129" sitekey="6Lcsv3oUAAAAAGFhlKrkRb029OHio098bbeyi_Hv" grecaptcha="" has-valid-session="true" weblogin-url="https://secure.runescape.com/m=weblogin/a=870/c=K0aO9WO69EI/loginform?mod=www&ssl=1&dest=zendesk/support-form?form=360000065898">
<div class="x-display-none ie-error-display" data-js-ie-error="">
<section class="c-article">
<div class="c-article__content">
<h1>Error: Unsupported Browser</h1>
<p>
We do not support your web browser. Please use a supported web browser by choosing one below.
<br>
FireFox
<br>
Chrome
</p>
</div>
</section>
</div>
Code:
String pagesource = driver.getPageSource();
System.out.println(pagesource);
Result: Same as the previous one.. different page..
Firefox Page Source: https://pastebin.com/Kv15V2SK
Firefox Inspect Element of the page screenshot: http://prntscr.com/qvi6hc
This is weird, as the page source is different to the form!
I couldn't find time to solve your problem. If you want to do it on your own, please Search this on Google, "Shadow Root, Selenium", I had this kind of error before. What I know is, you cannot directly reach an element that stays inside of a shadow root, This is why you are not getting the source code inside of it.
What you need to do is go through the element step by step:
You have to expand the shadow root,
Here is shadow root expand function:
public static WebElement expand_shadow_element(WebElement element)
{
WebElement shadow_root = (WebElement)((JavascriptExecutor)driver).executeScript("return arguments[0].shadowRoot", element);
return shadow_root;
}
You can imagine this function like
.switchTo.frame()
for now..
After some researches you will understand the shadow root.
I hope I got the problem right..
Try this function, If you cannot, I will help you later on. Good Luck.
The PageSource from the <body> tag, containing...
<zendesk-ticketing-form base-url="https://www.runescape.com/a=870/c=K0aO9WO69EI" css-cachebust="129" sitekey="6Lcsv3oUAAAAAGFhlKrkRb029OHio098bbeyi_Hv" grecaptcha="" has-valid-session="true" weblogin-url="https://secure.runescape.com/m=weblogin/a=870/c=K0aO9WO69EI/loginform?mod=www&ssl=1&dest=zendesk/support-form?form=360000065898">
<div class="x-display-none ie-error-display" data-js-ie-error="">
<section class="c-article">
<div class="c-article__content">
<h1>Error: Unsupported Browser</h1>
<p>
We do not support your web browser. Please use a supported web browser by choosing one below.
<br>
FireFox
<br>
Chrome
</p>
</div>
</section>
</div>
...implies that the WebDriver driven Browsing Context was detected as a BOT and the navigation was blocked due to presence of reCAPTCHA.
There are different approaches to solve captcha / recaptcha. You can find a couple of relevant discussion in:
How to bypass Google captcha with Selenium
Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection
Update
From your comments now it is clear that you want to fill up the fields within the form:
At this point it is worth to mention that you had been redirected to this page for either of the following reasons:
You EmailID / UserID is banned / blocked from accessing the site.
You EmailID / UserID is black-listed from accessing the site.
As you have used a BOT to access/scrape the site which may have violated the T&C.
Solution
It would be tough to propose a solution to automatically fillup the fields as presumably the elements in the BAN APPEAL REQUEST page may be protected by Invisible reCAPTCHA and you may have to Programmatically invoke the challenge
As others have suggested, it appears RuneScape's website has detected that you're using a bot to interact with their site. It doesn't matter that you solved the captcha manually, as they can still detect automated behavior quite easily without one (and no, the navigator.webdriver flag is not their only way to detect this).
The captcha is meant to prevent automated interaction with their site, which means they don't want you using Selenium/WebDriver to interact with it. You should respect this, especially as it seems you want your account unbanned (going by the pasted snippets and screenshots), so trying to do exactly what they don't want won't win you any favors.
Related
I am trying to capture a drop-menu field using test automation in SELENIUM with chrome driver.
Why does the element "id" changes in some web pages when inspecting elements time to time, with Chrome browser? How to keep the "id"s static, without changing?
Steps I followed:
When I inspect elements in the web page, the particular drop-menu shows its "id" as: id="combo-1782-inputEl"
HTML:
<input id="combo-1782-inputEl**" type="text" class="x-form-field x-form-required-field x-form-text x-trigger-noedit x-form-focus x-field-form-focus x-field-default-form-focus" autocomplete="off" name="type" readonly="readonly" aria-invalid="false" data-errorqtip="" style="width: 135px;">
id observed by inspecting the web page in normal chrome browser:
Then I used the above id in my java code(automation script) as below:
driver.findElement(By.id("combo-1782-inputEl")).click();
When I run the test > The google chrome browser opens automatically > The test gets successful till it meets the above line of code.
But, when it meets the above code line, the test failed Throwing the following exception:
class org.openqa.selenium.NoSuchElementException *
Then I inspected the same drop menu item in chrome web page opened (controlled) by automated test software, and found out that the "id" is different than the previous id mentioned in step 1.
The "id" in this case is: "combo-1781-inputEl"
id observed by inspecting the web page in chrome browser controlled by automated software:
As you can see, the number in the middle of the id has reduced from 1.
(1782-1 = 1781)
Same issue was found in the other drop menu items on the same web page.
What is the issue cause for this? How can I overcome this situation? Please help. :)
P.S. When I used "combo-1781-inputEl" ("id" from step 5) in my code, the test passed successfully.
driver.findElement(By.id("combo-1782-inputEl")).click(); //Test: failed
driver.findElement(By.id("combo-1781-inputEl")).click(); //Test: passed
I expected the test to be passed when I used the "id" I got in step 1 by inspecting the web page in normal chrome browser which is not controlled by automated software.
In order to provide you with the best answer, I'd need to see a section of HTML code for the web page, enough that I can see other attributes. When you have dynamic ID's you have two options:
Option 1: use an xpath that uses part of the ID that is constant, i.e.:
//*[starts-with(#id, 'combo-')]
might do the trick, but only if there are no other similar ID's.
Or perhaps:
//*[starts-with(#id, 'combo-') and ends-with(#id, '-inputEl')]
but that might still not be specific-enough. This is why seeing a section of your HTML would help.
Option 2: use other attributes instead of ID, i.e. class, text, or some other attribute.
//*[#name='FirstName']
for example. You can craft rather elaborate xpaths using combinations of attributes, and it will be fairly stable if you do it right. Sharpening your xpath creation skills will come in handy for things like this.
Use Xpath and WebDriverWait to handle dynamic element.Hope this will work.
WebDriverWait wait = new WebDriverWait(driver, 30);
WebElement element = wait.until(ExpectedConditions.elementToBeClickable(By.xpath("//input[starts-with(#id, 'combo-')][#type='text']")));
As it is evident that the id attribute of the <input> tag changes dynamically e.g. "combo-1782-inputEl", "combo-1781-inputEl" and so on so you need to create a Dynamic Locator inducing WebDriverWait and you can use either of the following Locator Strategies:
cssSelector:
new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.cssSelector("input.x-form-field.x-form-required-field.x-form-text.x-trigger-noedit.x-form-focus.x-field-form-focus.x-field-default-form-focus[id$='-inputEl'][name='type']"))).click();
xpath:
new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.xpath("//input[starts-with(#id, 'combo-') and #class='x-form-field x-form-required-field x-form-text x-trigger-noedit x-form-focus x-field-form-focus x-field-default-form-focus'][#name='type']"))).click();
I am trying to locate an iframe by partial id. For this method, I used:
driver.switchTo().frame(driver.findElement(By.cssSelector("iframe[id*='card-fields-number']"))); I have also tried xpath.
driver.switchTo().frame(driver.findElement(By.xpath("//iframe[contains(#id,'card-fields-number')]")));
However, I still receive this exception:
org.openqa.selenium.NoSuchElementException: Returned node (null) was not a DOM element
I found out that when I enable javascript for HtmlUnit, it is able to locate the frame, and switch to it. I would rather not enable javascript as it runs very slow for me, and adds unneeded delay.
iFrame HTML code:
<iframe class="card-fields-iframe" frameborder="0" id="card-fields-number-7pbvqg7azsf00000" name="card-fields-number-7pbvqg7azsf00000" scrolling="no" src="https://checkout.shopifycs.com/number?identifier=438599641d0ed8fe61c161d72e62b5f8&location=https%3A%2F%2Fshopnicekicks.com%2F2192362%2Fcheckouts%2F438599641d0ed8fe61c161d72e62b5f8&dir=ltr&fonts[]=Lato" title="Field container for: Card number" style="height: 43px;"></iframe>
iFrame ID is dynamic, so that is why I resort to using partial ID.
Website link: https://shopnicekicks.com/checkout
You must fill everything out until you reach the last page, which is the credit card information page.
Update
The iFrame is inside of the parent frame.
Parent Frame:
<iframe srcdoc="<script>!function(){var e=function(e){var t={exports:{}};return e.call(t.exports,t,t.exports),t.exports},t=function(){function e(e,t){for(var n=0;n<t.length;n++){var i=t[n];i.enumerable=i.enumerable||!1,i.configurable=!0,"value"in i&&(i.writable=!0),Object.defineProperty(e,i.key,i)}}return function(t,n,i){return n&&e(t.prototype,n),i&&e(t,i),t}}(),n=function(e,t){if(!(e instanceof t))throw new TypeError("Cannot call a class as a function")},i=function(e){return e&&e.__esModule?e:{"default":e}},o=e(function(e,i){"use strict";Object.defineProperty(i,"__esModule",{value:!0});var o=function(){function e(){var t=this;n(this,e),this.calls=[],window.ga=function(){for(var e=arguments.length,n=Array(e),i=0;i<e;i++)n[i]=arguments[i];return t.gaCall(n)}}return t(e,[{key:"gaCall",value:function(e){var t=this;this.calls.push(e),clearTimeout(this.timeout),this.timeout=setTimeout(function(){t.calls.length>0&&t.sendMessage()},0)}},{key:"listen",value:function(){var e=this;window.addEventListener("message",function(t){return e.receiveMessage(t)},!1)}},{key:"sendMessage",value:function(){window.parent.postMessage({type:"analytics",calls:this.calls},this.origin),this.calls=[]}},{key:"receiveMessage",value:function(e){if(e.source===window.parent&&"checkout_context"===e.data.type){this.origin=e.origin,window.Shopify=e.data.Shopify,window.__st=e.data.__st;try{window.additionalScripts()}catch(e){console.error("User script error: ",e)}}}}]),e}();i["default"]=o});e(function(){"use strict";var e=i(o);!function(){(new e["default"]).listen()}()})}("undefined"!=typeof global?global:"undefined"!=typeof window&&window); window.additionalScripts = function () {};</script>" src="https://checkout.shopify.com/2192362/sandbox/google_analytics_iframe" onload="this.setAttribute('data-loaded', true)" sandbox="allow-scripts" id="google-analytics-sandbox" tabindex="-1" class="visually-hidden" style="display:none" aria-hidden="true" data-loaded="true"></iframe>
This error message...
org.openqa.selenium.NoSuchElementException: Returned node (null) was not a DOM element
...implies that there was no such element found as the returned node was null or was not a DOM element.
This is still a open issue with htmlunit-driver team.
However there are certain things which you need to take care as follows:
First and foremost, all the modern browsers come with built-in support for JavaScript.
HtmlUnitDriver is a WebDriver compatible driver for HtmlUnit headless browser. It has fairly good JavaScript support (which is constantly improving) and is able to work even with quite complex AJAX libraries, simulating Chrome, Firefox or Internet Explorer depending on the configuration used. So ideally while working with HtmlUnitDriver, JavaScript must be enabled, else HtmlUnitDriver may not ne able to detect the JavaScript based elements.
You can find a detailed discussion in HtmlUnitDriver does not load javascript when navigating a page from an url
The element seems to be a credit card field and historically Credit Card Number, etc resides within <iframes>.
You can find a detailed discussion in Unable to locate element of credit card number using selenium python
Whenever an <iframe> is in play src attribute of the <iframe> tag plays a vital role.
You can find a detailed discussion in Ways to deal with #document under iframe
As per best practices while switching <iframe> you need to induce WebDriverWait for the desired frame to be available and switch to it.
We have discussed this aspect in your previous question Selenium can't locate iframe
So you can use either of the following solutions:
cssSelector:
new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.cssSelector("iframe.card-fields-iframe[id^='card-fields-number-'][src*='shopifycs']")));
xpath:
new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.xpath("//iframe[#class='card-fields-iframe' and starts-with(#id,'card-fields-number-')][contains(#src, 'shopifycs')]")));
Update
See the snapshot of the CssSelector which identifies the element perfecto as per the HTML you have provided:
Just try with frame indexing "//iframe[index]", where index integer.
e.g. xpath : //iframe[1]
Frame id may change dynamically but in a few application structure remains same so indexing solves the problem.
Please let me know if it solves the issue.
As the iframe has tag iframe, you can switch to the iframe using the tagname like:
driver.switchTo().frame(driver.findElement(By.tagName("iframe")));
And if you want to again switch to the default content, then you can use driver.switchTo().defaultContent();
Well not to mention that I'm new on using the selenium web driver, I'm trying to automatize a mailbox so I can log myself in and to send a message, the thing is that I download the firebug plugin for Mozilla, that is the browser I'm currently working with, when I'm trying to find the element for the log in and the password everything goes well, because I do it by their name.
input class="_nb-input-controller" type="text" **name="login"** autocorrect="off" autocapitalize="off" value=""
input class="_nb-input-controller" type="password" **name="passwd"** value=""
but as soon as im at the mailbox entry the identificator changes it, i go to compose link and this is the code that shows up:
a class="b-toolbar__item b-toolbar__item_compose js-toolbar-item-compose daria-action" title="Compose (w, c)" href="#compose" data-action="compose.go" data-params="toolbar=1&toolbar.button=compose"
Since I don't find any method that could help on this, a friend came and told me about the xpath which was:
/html/body/div[2]/div/div[5]/div/div[3]/div/div[2]/div/div/div/div[2]/a[2]
by doing this selenium did find the element and I could click on the icon,
but again.. when I'm trying to get into the "TO" prompt so I can write inside
it is not finding the element, not by xpath, or anything is there a way or formula to get elements or translate them into a way that selenium can find them???
oh i see, i re try with the element and i noticed that the one i wanted was showing as hidden using firebug so i just try the next one up and the web driver could locate the item, it was a simple app to send an email the prompt that i wanted to locate was the 'TO' field. and thanks guys for giving me the correct way of using the Xpath property
Having locators such as :
/html/body/div[2]/div/div[5]/div/div[3]/div/div[2]/div/div/div/div[2]/a[2]
is generally a bad idea. If you want to build a stable test I would suggest you stay away from such.
Now I see you have attributes like #name and #class. XPath is provides you with a way to reach these elements directly by pointing the identifiers instead of the whole path to the element.
You can try something like:
//input[#name='login']
//input[#name='password']
More info at: http://www.w3schools.com/xsl/xpath_intro.asp
What I need to do is browse to a webpage, login, then browse to another webpage on that site that requires you to be logged in, so it needs to save cookies. After that, I need to click an element on that page, in which I would fill out the form and get the message that the webpage returns to me. The reason I need to actually go to the page and click the button as suppose to just navigating directly to the link is because the you are assigned a session ID every time you log in and click the link, and its always different. The button looks like this, its not a normal href link:
<span id=":tv" idlink="" class="sA" tabindex="0" role="link">Next</span>
Anyway, what would be the easiest way to do this? Thanks.
Update:
After trying HTMLunit, and other headless browser libraries, it doesnt seem that its happening using anything "headless." Another thing that I recently found out about this page is that that all the HTML is in some weird format... Its all inside a script tag. Here is a sample.
"?ui\x3d2\x26view\x3dss\x26mset\x3dmain\x26ver\x3d-68igm85d1771\x26am\x3d!Zsl-0RZ-XLv0BO3aNKsL0sgMg3nH10t5WrPgJSU8CYS-KNWlyrLmiW3HvC5ykER_n_5dDw\x26fri"],"http://example.com/?ctx\x3d%67mail\x26hl\x3den",,0,"Gmail","Gmail",[["us","c130f0854ca2c2bb",[["n"],["m","New features!"],["u"],["k","0"],["p","1000:500000,10,200000,5,100000,3,75000,2,0,1"],["h","https://survey.googleratings.com/wix/p1679258.aspx?l\x3d1033"],["at","query,5,contacts,5,adv,5,cf,5,default,20"],["v","https://www.youtube.com/embed/Ra8HG6MkOXY?showinfo\x3d0"],
When I do inspect element on the button, the HTML code that I posted above for the button comes up, but not when doing view source. Basically, what I am going to need to do is use some sort of GUI and have the user navigate to the link and then have the program fill out the info. Does anyone know how I can do this? Thanks.
Have a look at the 5 Minute Getting Started Guide for Selenium: http://code.google.com/p/selenium/wiki/GettingStarted
On the login page, look at the form's HTML to see the url it posts to and the url parameters. Then request that url with the same parameters filled in with correct info, and make sure to save all the cookie headers to send to the second page. Then use an html parser to find your link. There are several html parsers available on sourceforge, and you could even try java's built in xml parsers, though if the site has even a tiny html mistake they will glitch.
EDIT didn't notice the fact that it is not a normal link. In that case you will need to look at the site's javascript to see where the link leads. If the link requires javascript to run, it gets more complicated. Java is not able to execute browser javascript, but I found a library called DJ native swing which includes a web browser class that you can add to jframes. It uses your native browser to render, and to run javascript.
This should be possible in Selenium as others have noted.
I have used Selenium to login then crawl a site and discover every permuation of values for every form on the site (30+ forms). These values are later used to fill and submit the form with a specific perumation of values. This site was very JS/jQuery heavy and I used Selenium's built-in support of javascript executor, css selectors, and XPath to accomplish this.
I implemented HtmlUnit and HttpUnit as faster alternatives, but found they were not as reliable as Selenium given the JS semantics of the site I was crawling.
It's hard to give you code on how to accomplish it because your Selenium implementation will be quite page-specific and I can't look at the page you're coding against to figure out what's going on with that button script junk. However, I have include some possibly relevant selenium code (Java) snippets:
Element element = driver.findElements(By.id(value)); //find element on page
List<Element> buttons = parent.findElements(By.xpath("./tr/td/button")); //find child element
button.click();
element.submit() //submit enclosing form
element.sendKeys(text); //enter text in an input
String elementText = (String) ((JavascriptExecutor) driver).executeScript("return arguments[0].innerText || arguments[0].textContent", element); //interact with a selenium element via JS
If you are coding similar functions on different pages, then PageObjects behind interfaces can help.
The link Anew posted is a good starting point and good ol' StackOverflow has answers to just about any Selenium problem ever.
Instead of trying to browse around programmatically, try executing the login request and save the cookies then set those in the next request to the form post.
HTMLUnit is pretty bad at processing JavaScript, the Rhino JS library produces often errors (actually no errors is much the exception). I would advise to use Selenium, which is basically a framework to control headless browsers (chrome, firefox based).
For your question, the following code would do the work
selenium.open(myurl);
selenium.click("id=:tv");
You then have to wait for the page to load
selenium.waitForPageToLoad(someTime);
I would recommend htmlunit any day. It's a great library.
First, check out their web page(http://htmlunit.sourceforge.net/) to get htmlunit up and running. Make sure you use the latest snapshot(2.12 when writing this)
Try these settings to ignore pretty much any obstacle:
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_17);
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setUseInsecureSSL(true);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getCookieManager().setCookiesEnabled(true);
Then when fetching your page, make sure you wait for background Javascript before doing anything with the page, like posting a login form:
//Get Page
HtmlPage page1 = webClient.getPage("https://login-url/");
//Wait for background Javascript
webClient.waitForBackgroundJavaScript(10000);
//Get first form on page
HtmlForm form = page1.getForms().get(0);
//Get login input fields using input field name
HtmlTextInput userName = form.getInputByName("UserName");
HtmlPasswordInput password = form.getInputByName("Password");
//Set input values
userName.setValueAttribute("MyUserName");
password.setValueAttribute("MyPassword");
//Find the first button in form using name, id or xpath
HtmlElement button = (HtmlElement) form.getFirstByXPath("//button");
//Post by clicking the button and cast the result, login arrival url, to a new page and repeat what you did with page1 or something else :)
HtmlPage page2 = (HtmlPage) button.click();
//Profit
System.out.println(page2.asXml());
I hope this basic example will help you!
I am trying to get a WebElement with Selenium:
driver.findElement(By.xpath("//input[#name='j_username']"))
But Selenium says: "Unable to find element with XPath ...".
The XPath is valid, I proofed it with FirePath.
But the input element has the following invalid code:
<input size="10" type="text" name="j_username" maxlength="8">
I can't change the html-file, despite the fact is there any solution to get the webElement?
Thanks in advance!
try select element with css selector. and also verify in firepath(firebug addon that element is located properly).
so your css selector be something like
input[name='j_username']
2nd approach is to use internal firebug mechanism for finding xPaths of elements.
See screen attached below
After these manipulations driver shoulda handle element properly.
Well I will suggest adding an id to your html code -
<input id="j_username"size="10" type="text" name="j_username" maxlength="8">
and findElement by id -
driver.findElement(By.id("j_username"));
I have faced similar issues with xpath(borwser issues??) but id never fails for me. ;)
By the way I feel your code should be -
driver.findElement(By.xpath(".//*[#name='j_username']"));
The best solution is to find out what selenium is doing wrong, but without a URL or sample page to test on it's a little hard. Is there anyway you could dump the HTML into a jsfiddle? If there is do that and paste the url into the question and I'm sure someone can find a solution.
If not however, another way to get the results is to do it with jQuery. If firebug is picking it up but not selenium, then there's no reason why jQuery wouldn't get it. Here's how to go about doing that if needed:
Step 1: Is jQuery already present on the page? If so then you don't need to do this bit, otherwise you will need to add it yourself by using driver.executeScript(addjQueryScript) where the script does something like this.
Step 2: call WebElement input = driver.executeScript(elementSelector); where the elementSelector script would be something like \"return $('input[name=\"j_username\"]')\");
My jQuery's not so good, but I believe that should work...
Best of luck!