Develop Reference

Java Selenium - Bypass driver.get() waiting time

I am trying to access a website using Selenium WebDriver, but the website will keep loading although I can still interact with it. (The website is nitrotype.com if you are wondering.) I think it is because driver.get() waits until the page is fully loaded. Can I bypass this until just a certain element loads ?
TL;DR
How do I bypass the driver.get() waiting until a site is completely loaded before proceeding?

Look into page load strategy: https://www.selenium.dev/documentation/en/webdriver/page_loading_strategy/
Normal, Eager & None are your options. I suggest you combine your strategy with the proper explicit wait.
NORMAL
normal This will make Selenium WebDriver to wait for the entire page
is loaded. When set to normal, Selenium WebDriver waits until the load
event fire is returned.
By default normal is set to browser if none is provided.
EAGER
eager This will make Selenium WebDriver to wait until the initial HTML
document has been completely loaded and parsed, and discards loading
of stylesheets, images and subframes.
When set to eager, Selenium WebDriver waits until DOMContentLoaded
event fire is returned.
NONE
none When set to none Selenium WebDriver only waits until the initial
page is downloaded.
To implement:
public class pageLoadStrategy {
public static void main(String[] args) {
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.setPageLoadStrategy(PageLoadStrategy.NONE);
WebDriver driver = new ChromeDriver(chromeOptions);
try {
// Navigate to Url
driver.get("https://google.com");
} finally {
driver.quit();
}
}
}

Selenium Java Client is not getting back the control once driver.get(URL) method is invoked

driver.get("MyURL");
System.out.println("URL is opened");
executeAutoItScript(scriptFileLocation);
when i open the URL i get an Authentication Required pop up.
To handle that I am using AutoIt script. But the problem is As soon as the first command
(driver.get("MyURL");)
is executed, Chrome will get open and the
Authentication pop up appears. And i have observed that the second line
System.out.println("URL is opened");
is not being executed. I debugged it and observed that the
control is not given to next line from
driver.get("MyURL");
and it hangs
there. I changed driver.get("MyURL"); to driver.navigate().to("MyURL"); but
there is no luck. Could anyone please help me to resolve this. Attached is
the pop up screenshot.

As per your code trials and the browser snapshot, it seems that the Browser Client (i.e. the Google Chrome Browser) is not returning back the control to the WebDriver instance and subsequently Selenium Java Client can't achieve the state of 'document.readyState' equal to "complete". Hence neither your next line of code:
System.out.println("URL is opened");
is getting executed, nor the AutoIt Script in next line:
executeAutoItScript(scriptFileLocation);
Solution
It is not clear from your question about the source of this Authentication Popup. Perhaps following the discussion Selenium - Basic Authentication via url you can pass the username and password being embedded within the URL as follows:
driver.get("http://admin:admin123#MyURL");

From: http://selenium-python.readthedocs.io/navigating.html
WebDriver will wait until the page has fully loaded (that is, the onload event has fired) before returning control to your test or script. It’s worth noting that if your page uses a lot of AJAX on load then WebDriver may not know when it has completely loaded. If you need to ensure such pages are fully loaded then you can use waits.
So, in this case your webpage is not fully loaded since it requires authentication. Here is what you can do
driver.get("MyURL");
executeAutoItScript(scriptFileLocation);
Thread.sleep(2000);// to wait for autoit script, you can also use other wait explicit wait
//Assert statement
System.out.println("URL is opened");

->First define the page load time for the driver.
->By using try-catch time out exception invoke the url.
->After that use robot class key events or key events class to enter the authentication details
Try the below one if any queries do lemme know:
driver.manage().timeouts().pageLoadTimeout(10, TimeUnit.SECONDS);
try{
driver.navigate().to("yourURL");
}
catch(TimeoutException te){
System.out.println(te);
System.out.println("Line went to Upto that Link");
after this you could proceed with the authentication pop-up code.
Do lemme know if you have any queries.

This helped me:
InternetExplorerOptions options = new InternetExplorerOptions();
options.setCapability("initialBrowserUrl", "about:blank");
options.setPageLoadStrategy(PageLoadStrategy.NONE);
WebDriver driver = new InternetExplorerDriver(options);
driver.manage().deleteAllCookies();
driver.manage().window().maximize();
driver.manage().timeouts().pageLoadTimeout(10, TimeUnit.SECONDS);
driver.get(url);
//execute AutoItScript goes here

Web-scraping: how to choose proxy

I want to write as reusable as possible web scraper. I gonna write it on Selenium + PhantomJS. PhantomJS will use a pool of IPs (proxies). There is a huge list of free proxies, for example.
How could I choose at runtime the best proxy for specific URL? By the best I mean the fastest an one which won't be blocked with target resource.
Workarround
I deployed my simplest app on Heroku. The app serves some html content. I used different proxies (with response time < 300ms) instead of 151.252.120.177:8080 (see code below), and noticed that most of them aren't able to parse simplest html in 15 seconds timeout. And some of them (that are even slower) retrieved content in a second. Why some proxies are unable to reach my content? Are they blacklisted with Heroku?
DesiredCapabilities caps = new DesiredCapabilities();
caps.setJavascriptEnabled(true);
caps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY, "drivers/phantomjs");
ArrayList<String> cliArgsCap = new ArrayList<String>();
cliArgsCap.add("--proxy=151.252.120.177:8080");
cliArgsCap.add("--proxy-type=socks");
caps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, cliArgsCap);
WebDriver driver = new PhantomJSDriver(caps);
driver.get(REMOTE_URL);
WebDriverWait wait = new WebDriverWait(driver, 15);
WebElement element = wait.until(ExpectedConditions.elementToBeClickable(By.className("btn-success")));
element.click();
driver.quit();

If you need just a scraper result you can use free scrapers with proxy support. Try this one http://datathief.verych.ru/

Selenium Webdriver not waiting for element

Here is my selenium web driver initialization for firefox browser.
driver = new FirefoxDriver();
driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
driver.manage().timeouts().pageLoadTimeout(30, TimeUnit.SECONDS);
Even though I gave implicit wait selenium not waiting for the element. It is throwing the not found exception immediately. If I put Thread.sleep then it is working fine without any issues. But putting Thread.sleep everywhere the test case contains now more Thread.sleep than the actual test case code. Can anyone suggest me the right way to do this?

in that case you should use ExplicitWait to wait for an specific element to be visible or present, because it's not a good practice tosleep the thread. I'll recommend to use:
WebDriver driver wait = new WebDriverWait(driver, "time here");
wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath(xPath)));

Selenium Implicit Timeout stopped working

I'm creating an API based on Selenium API for our system that is controlled through the web-browser (similar to the way you control routers and switches).
One of the things that happens in this API is checking whether certain elements exist on the page (depending on the setting, they may or may not exit, for example, text box or some status).
I have this line:
driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
I had it working just fine until when at one point in time, I ran it, and the driver.findElement(By....); has never timed out (the element was missing from the web-page)... The longest I have waited was somewhere about half-an hour.
I'm running Ubuntu 12.04 x64 as my host, Windows 7 as my guest, and on it I have my Selenium API and the code written in Java that controls out product. When I port the same code to another Windows VM or even to a different work station, it works fine. Not on my particular VM though.
Tried "reinstalling" eclipse, reimported Selenium JARs, pulled the latest working code from the server... Nothing works...
What are your thoughts, fellas?
Will appreciate any response. Thanks.

first of all try using fluentWait to wait for all the AJAX on the page:
public WebElement fluentWait(final By locator){
Wait<WebDriver> wait = new FluentWait<WebDriver>(driver)
.withTimeout(30, TimeUnit.SECONDS)
.pollingEvery(5, TimeUnit.SECONDS)
.ignoring(org.openqa.selenium.NoSuchElementException.class);
WebElement foo = wait.until(
new Function<WebDriver, WebElement>() {
public WebElement apply(WebDriver driver) {
return driver.findElement(locator);
}
}
);
return foo; } ;
fluentWait(By.xpath(....blablabla..)).click();
//fluentWait(By.xpath(....blablabla..)).getText();
Secondly, try using Thead.sleep(1000); as alternative to
driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);