Develop Reference

unable to capture 'g-recaptcha-response' for Recaptchav2 with Selenium

So I've been trying to build a webscraper but some of the data I need to scrape is locked behind a reCaptcha. From what I've gathered scouring around on the internet is every captcha has a TextArea element with the 'g-recaptcha-response' that gets filled in as the captcha is completed. The current solution for testing is to simply get around the captcha with me manually doing it and trying to capture the response and feed it back into the headless browser however I'm unable to get the response since as soon as the answer is submitted it can no longer find the response element.
org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"*[name='g-recaptcha-response']"}
public static String captchaSolver(String captchaUrl) {
setUp();
driver.get(captchaUrl);
new WebDriverWait(driver,2);
try {
while (true) {
String response = driver.findElement(By.name("g-recaptcha-response")).getText();
if (response.length()!=0) {
System.out.println(response);
break;
}
}
}catch (Exception e){
e.printStackTrace();
}
return "";
}

Try to find the element by CSS like this:
*[name*='g-recaptcha-response']

selenium webdriver invalid login

i want to close the browser for an invalid username/password but i am not able to do that. It runs well for a valid input though but for an invalid one it just freezes in the page where it shows "invalid login". this is my code. setup1 is just opening the browser.
while(recordset.next()){
setUp1();
System.out.println(driver1);
driver1.manage().timeouts().implicitlyWait(3000, TimeUnit.SECONDS);
System.out.println(recordset.getField("NAME"));
System.out.println(recordset.getField("PASSWORD"));
String store1 = recordset.getField("NAME");
String store = recordset.getField("PASSWORD");
driver1.findElement(By.id("signin_button")).click();
driver1.findElement(By.id("user_login")).clear();
driver1.findElement(By.id("user_password")).clear();
driver1.findElement(By.id("user_login")).sendKeys(store1);
driver1.findElement(By.id("user_password")).sendKeys(store);
driver1.findElement(By.name("submit")).click();
driver1.findElement(By.xpath("//div/ul[#class = 'nav float-right']/li[3]")).click();
driver1.findElement(By.id("logout_link")).click();

You can call driver1.close() if driver1.findElement(By.xpath("//div/ul[#class = 'nav float-right']/li[3]")) throws a NoSuchElementException exception.
The browser will be closed if the current page is the only open page in the browser.
Here is an example:
try{
WebElement elm = driver1.findElement(By.xpath("//div/ul[#class = 'nav float-right']/li[3]"));
elm.click();
}catch(NoSuchElementException e){
driver1.close();
break; // exit the loop.
}

Download a webpage's source which uses a loading spinner

I need to download a source code of this webpage: https://app.zonky.cz/#/marketplace/ so I could have the code checking if there is a new loan available. Unfortunate for me, the web page uses a loading spinner for the time the page is being loaded in the background. When I try to download the page's source using:
String url = "https://app.zonky.cz/#/marketplace/";
StringBuilder text = new StringBuilder();
try
{
URL pageURL = new URL(url);
Scanner scanner = new Scanner(pageURL.openStream(), "utf-8");
try {
while (scanner.hasNextLine()){
text.append(scanner.nextLine() + "\n");
}
}
finally{
scanner.close();
}
}
catch(Exception ex)
{
//
}
System.out.println(text.toString());
I get the page's source from the moment the spinner is being shown. Do you know of a better approach?
Solution:
public static String getSource() {
WebDriver driver = new FirefoxDriver();
driver.get("https://app.zonky.cz/#/marketplace/");
String output = driver.getPageSource();
driver.close();
return output;
}

You could always wait until the page has finished loading by checking if an element exists(one that is only loaded after the spinner disappears)
Also have you looked into using selenium ? it can be really useful for interacting with websites and handling tricky procedures such as waiting for elements :P
Edit: a pretty simple tutorial for Selenium waiting can be found here - http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp#explicit-and-implicit-waits

How to select element in pop up message using webdriver?

i have a web page where i can create a document.
When i save the document i have a message like "Document successfully saved"
The message appears only for 3 seconds.
How can i find the element with webdriver to check the message has been displayed during my test.
In the html code, i tried to catch the code for this message, it appears for 3 seconds and the disappaeared.
The code is here:
<div class="alert alert-success top center alert-notify alert-icon>
Document successfully saved.
<span class="Close"></span>
I tried this without success:
try {
Alert alert = driver.switchTo().alert();
message = alert.getText();
alert.accept();
} catch (Exception e) {
message = null;
}
System.out.println("message"+message);
return message;
}
then i tried this without success too:
driver.findElement(By.className("alert alert-success top center alert-notify")).isDisplayed();
Thread.sleep(3000);
System.out.println(" Alert message is displayed");
Can anyone help me please, i don't know how to handle with

your second try is the right way to detect the message, but your have to try for 3 seconds to find the element.
It should look like this
while (!elementFound || stillWithin3Seconds){
elementFound = findAlertByClassName()
}
assetTrue(elementFound)
You have to look all the time. In your code you are sleeping. When the dialog appears in this time, you won't notice it.

java: Sleep until website is fully loaded

I want to open a website with selenium and extract some text via xpath. It's working but I can get the text only if I wait a few seconds until the whole website is loaded, but it would be nicer to check if the website is fully loaded. I guess I should check for any background connections (ajax calls, GETs, POSTs or whatever), what is the best way to do it? At the moment I'm trying to extract the text in a while loop (ugly solution):
WebDriver driver = new FirefoxDriver();
driver.get("http://www.website.com");
// Try to get text
while (true) {
try {
WebElement findElement = driver.findElement(By.xpath("expression-here"));
System.out.println(findElement.getText());
break;
// If there is no text sleep one second and try again
} catch (org.openqa.selenium.NoSuchElementException e) {
System.out.println("Waiting...");
Thread.sleep(1000);
}
}
How would you solve this?

Selenium has already timeouts for this.
Look here: http://seleniumhq.org/docs/04_webdriver_advanced.jsp
I have solve it with:
webDriver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);

Prefer explicit waits:
1) more readable
2) been using those for long time
3) what if you know the element is not there (your test is to verify it) and don't need to poll for 10 seconds? With implicit you are now wasting time.
WebElement myDynamicElement = (new WebDriverWait(driver, 10))
.until(new ExpectedCondition<WebElement>(){
#Override
public WebElement apply(WebDriver d) {
return d.findElement(By.id("myDynamicElement"));
}});