I use Selenium webdriver with Firefox for scraping web pages. Sometimes web browser waits endless time for some excessive requests complete (e.g. to facebook.net).
I've tried to use BrowserMob-Proxy to filter these requests. But it didn't help. These requests, even after receiving 200 or 404 code, doesn't stop.
I thought about some possibility to stop web browser loads page after some amount of time.
For example:
try {
Thread.sleep(5000);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt(); }
((JavascriptExecutor) driver).executeScript("window.stop();");
But it doesn't work until web page loads completely.
What can you suggest me to do in my case?
P.S. This is a code with using a pageLoadTimeout parameter.
WebDriver driver;
FirefoxBinary firefox;
FirefoxProfile customProfile;
public static void main(String[] args) {
openFirefox();
for (String url : listOfUrls) {
Boolean pageLoaded = false;
while (pageLoaded == false) {
try {
driver.get(url);
pageLoaded = true;
} catch (org.openqa.selenium.TimeoutException ex) {
System.out.println("Got TimeoutException on page load. Restarting browser...");
restartFirefox();
}
}
//here I do something with a content of a webpage
}
}
public static void openFirefox(){
firefox = new FirefoxBinary(new File(Constants.PATH_TO_FIREFOX_EXE));
customProfile = new FirefoxProfile();
customProfile.setAcceptUntrustedCertificates(true);
customProfile.setPreference("webdriver.load.strategy", "unstable");
driver = new FirefoxDriver(firefox, customProfile);
driver.manage().deleteAllCookies();
driver.manage().timeouts().pageLoadTimeout(60, TimeUnit.SECONDS);
}
private static void restartFirefox() {
driver.close();
firefox.quit();
openFirefox();
}
How about using timeouts? So for each WebDriver instance that you are using you need to set:
WebDriver.Timeouts pageLoadTimeout(long time, java.util.concurrent.TimeUnit unit)
Which by the Documentation:
Sets the amount of time to wait for a page load to complete before
throwing an error. If the timeout is negative, page loads can be
indefinite.
Parameters:
time - The timeout value.
unit - The unit of time. Returns:
A Timeouts interface.
I've tried to use BrowserMob-Proxy to filter these requests. But it
didn't help. These requests, even after receiving 200 or 404 code,
doesn't stop.
What do you mean "didn't help". I don't believe you. Please share your code for blacklisting URLs. For example, following code code returned HTTP.200 for any google-analytics related site for me
server.blacklistRequests("https?://.*\\.google-analytics\\.com/.*", 200); // server is bmp proxy server
I have heard, that WebDriver should now have webdriver.load.strategy. I have never used it though. So the default behavior of WebDrivers blocking calls (a'la get()) is to wait for document.readyState to be complete, but I have read that with this property you could tell the driver to return at once. So might be worth googling it for a while.
Related
I'm trying to check login page control by using dataprovider but i don't want to initialize webdriver again and again for each username password control. Once i come into login page, checking all concerned scenarios on login page in single time without starting another driver seems more convenient to me but i couldn't figure it out. When running following code, data[0][0] and data[0][1] is being correctly checked but it gives no such element on Login method having second priority test annotation when being tried to be typed data[1][0] and data[1][1]. Probably, it causes because driver is not looking at that page on that time. How can I handle this issue ?
error:
org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#class='q-input-wrapper email-input']//input[#class='q-input']"}
code:
public class TestCaseFirst {
public WebDriver driver;
#BeforeTest
public void Start() throws InterruptedException {
WebDriverManager.chromedriver().setup();
driver= new ChromeDriver();
driver.get("https://www.faxzas.com/");
driver.manage().window().maximize();
Thread.sleep(2000);}
#Test(priority=1)
public void RoadtoLogin() throws InterruptedException {
driver.findElement(By.xpath("//a[#title='Close']")).click();
Thread.sleep(1000);
driver.findElement(By.xpath("//div[#class='login-container']//span[#id='not-logged-in-container']")).click();;
Thread.sleep(1000);
}
#Test(dataProvider="loginInfos", priority=2)
public void Login(String mail, String password) throws InterruptedException {
driver.findElement(By.xpath("//div[#class='q-input-wrapper email-input']//input[#class='q-input']")).sendKeys(mail);
Thread.sleep(1000);
driver.findElement(By.xpath("//div[#class='q-input-wrapper']//input[#class='q-input']")).sendKeys(password);
Thread.sleep(1000);
driver.findElement(By.xpath("//button[#type='submit']")).click();
Thread.sleep(1000);
String description = driver.findElement(By.xpath("//div[#id='error-box-wrapper']//span[#class='message']")).getText();
System.out.println(description);
}
#DataProvider(name="loginInfos")
public Object[][] getData(){
Object[][] data = new Object[6][2];
data[0][0]="blackkfredo#gmail.com";
data[0][1]="";
data[1][0]="blackkfredo#gmail.com";
data[1][1]="443242";
data[2][0]="";
data[2][1]="1a2b3c4d";
data[3][0]="";
data[3][1]="";
data[4][0]="blackkfredogmail.com";
data[4][1]="1a2b3c4d";
data[5][0]="blackkfredo#gmail.com";
data[5][1]="1a2b3c4d";
return data;
}
}
You need to reset your page to the login page where you are expecting the element to be. Either put an #AfterMethod and go back to the page you are trying to test or put an #BeforeMethod for the same. You may even want to wrap up your find element calls and handle the exceptions by going back to the main page.
So I'm trying to learn how to use Selenium, and I found a tutorial on the internet on how to do it (http://toolsqa.com/selenium-webdriver/first-test-case/). I know I am using internet explorer rather than firefox, but I've followed the instructions on how to setup the IEDriver. The problem is that when I use their code to simply open and close a window, it opens it, and ignores the driver.quit() that is in there.
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.ie.InternetExplorerDriver;
public class FirstTestCase {
public static void main(String[] args) {
String service = "C:\\Users\\abc\\Documents\\IE Explorer Server\\IEDriverServer.exe";
System.setProperty("webdriver.ie.driver", service);
InternetExplorerDriver driver = new InternetExplorerDriver();
//Launch the Online Store Website
driver.get("http://www.store.demoqa.com");
// Print a Log In message to the screen
System.out.println("Successfully opened the website www.Store.Demoqa.com");
//Wait for 5 Sec
Thread.sleep(5000);
// Close the driver
driver.quit();
System.out.println(",");
}
}
It prints the comma I have set to print after it closes, but it doesn't actually close the browser. Thread.sleep() also gives an error. I've looked for over 3 hours for some kind of fix for this but couldn't find anything.
Try this:
public static void main(String[] args) throws InterruptedException {
String service = "C:\\Users\\abc\\Documents\\IE Explorer Server\\IEDriverServer.exe";
System.setProperty("webdriver.ie.driver", service);
InternetExplorerDriver driver = new InternetExplorerDriver();
//Launch the Online Store Website
driver.get("http://www.store.demoqa.com");
// Print a Log In message to the screen
System.out.println("Successfully opened the website www.Store.Demoqa.com");
//Wait for 5 Sec
Thread.sleep(5000);
// Close the driver
driver.quit();
System.out.println(",");
}
Here is the Answer to your Question:
I don't see any significant error in your code block. A few suggestions about the solution:
I retested your code block with Selenium 3.4.0, IEDriverServer 3.4.0 & MSIE 10.0 with only one additional parameter as
public static void main(String[] args) throws InterruptedException
and your code executes just fine. Whenever you add Thread.sleep(5000); in your code you may consider to add throws InterruptedException. But again, as per best practices we must avoid using Thread.sleep() and replace it by ImplicitlyWait or ExplicitWait.
You have used the InternetExplorerDriver implementation to initiate the driver. As per W3C standards you must consider using the WebDriver interface instead.
As per your comment driver.quit() not only closes the driver but also kills the driver instance and releases the memory.
Here is your own code with some small tweaks which works well at my side:
public static void main(String[] args) throws InterruptedException
{
String service = "C:\\your_directory\\IEDriverServer.exe";
System.setProperty("webdriver.ie.driver", service);
WebDriver driver = new InternetExplorerDriver();
//Launch the Online Store Website
driver.get("http://www.store.demoqa.com");
// Print a Log In message to the screen
System.out.println("Successfully opened the website www.Store.Demoqa.com");
// Quit the driver
driver.quit();
System.out.println(",");
}
Let me know if this Answers your Question
Does Webdriver 2.28 automatically take a screenshot on exception/fail/error?
If it does, where can I find the screenshot? Which directory is the default?
No, it doesn't do it automatically. Two options you can try:
Use WebDriverEventListener that is attachable to a EventFiringWebDriver which you can simply wrap around your usual driver. This will take a screenshot for every Exception thrown by the underlying WebDriver, but not if you fail an assertTrue() check.
EventFiringWebDriver driver = new EventFiringWebDriver(new InternetExplorerDriver());
WebDriverEventListener errorListener = new AbstractWebDriverEventListener() {
#Override
public void onException(Throwable throwable, WebDriver driver) {
takeScreenshot("some name");
}
};
driver.register(errorListener);
If you're using JUnit, use the #Rule and TestRule. This will take a screenshot if the test fails for whatever reason.
#Rule
public TestRule testWatcher = new TestWatcher() {
#Override
public void failed(Throwable t, Description test) {
takeScreenshot("some name");
}
};
The takeScreenshot() method being this in both cases:
public void takeScreenshot(String screenshotName) {
if (driver instanceof TakesScreenshot) {
File tempFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
try {
FileUtils.copyFile(tempFile, new File("screenshots/" + screenshotName + ".png"));
} catch (IOException e) {
// TODO handle exception
}
}
}
...where the FileUtils.copyFile() method being the one in Apache Commons IO (which is also shipped with Selenium).
WebDriver does not take screenshot itself. But you cat take by this way : ((TakesScreenshot)webDriver).getScreenshotAs(OutputType.FILE);
Then save the screenshot anywhere you want.
Also you cat use something like Thucydides, wich may take screenshot on each action or on error and put it in a pretty report.
The short answer is No. WebDriver is an API to interact with the browser. You can make screenshots with it but you should know when to do it. So it's not done automatically as WebDriver doesn't know anything about testing.
If you are using TestNG as testing library you can implement Listener whose methods will be executed on different events (failure, success or other). In these methods you can implement the required logic (e.g. making screenshots).
i like to access some pages that are not under my control. It could be that this pages execute some slow get requests but the main html is fully loaded and displayed. I tried many options but i could make it. The firefoxWebDriver.get(...) doesn't terminate on some sites in a realistic time.
To reproduice the problem, I wrote this small UnitTest showing the problem:
public class Timeout {
private FirefoxDriver driver;
#Before
public void setup() {
final FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("dom.max_script_run_time", 0);
profile.setPreference("webdriver.load.strategy", "fast");
this.driver = new FirefoxDriver(profile);
// this.driver.manage().timeouts().pageLoadTimeout(10, TimeUnit.SECONDS);
// this.driver.manage().timeouts().setScriptTimeout(10, TimeUnit.SECONDS);
}
#Test(timeout = 15000)
public void shouldRetriveREDCAFEPageQuiteFast() {
this.driver.get("http://redcafe.vn/Home/su-kien-binh-luan/kagawa-tu-choi-mac-ao-so-7");
}
#Test(timeout = 15000)
public void shouldRetriveMUFCPageQuiteFast() {
this.driver.get("http://news.mufc.vn/detail/172-hoan-tat-giay-phep-lao-dong-m-u-chinh-thuc-so-huu-kagawa.html");
}
#After
public void tearDown() {
this.driver.close();
}
}
Thanks for you help.
<driver>.manage().timeouts().pageLoadTimeout(60, TimeUnit.SECONDS);
will set the page load timeout to 60 seconds, after which it will throw an error. You need to set this up before your first get() call.
The API is supported from Webdriver release 2.20.0 onwards.
Refer API Reference for new Timeout API's
I have been trying to access an HTTPS URL with the HTMLUnitDriver API of Selenium 2.0, but somehow the execution gets stuck at the "This Connection is Untrusted" window and and the control doesn't return back. Following is the code I have tried working on after I got some hint from this thread:
WebDriver driver = new HtmlUnitDriver() {
protected WebClient modifyWebClient(final WebClient client) {
try {
client.setUseInsecureSSL(true);
} catch (GeneralSecurityException e) {
e.printStackTrace();
}
return client;
}
};
driver.get("https://172.25.194.91:8443/meta/homeScreen.do");
I'd highly appreciate any help to get it work .
The issue was something else and is resolved now: HtmlUnitDriver uses WaitingRefreshHandler without parameters and unfortunately that is inappropriate for some sites - for example, HtmlUnitDriver hangs on http://news.google.com.
Cause & Scenario:
You load a page, which refreshes to itself after a period of time, because of the presence of a <meta http-equiv="refresh"...> directive in your HTML header.
WaitingRefreshHandler waits for the time specified but after that time elapses, it again redirects HtmlUnitDriver to get that page!
Consequently, you loop forever in this redirection process.
Solution:
One needs to extend the HtmlUnitDriver and override the modifyWebClient method to set a new (read: to clear the) refresh handler.
#Override
protected WebClient modifyWebClient(WebClient client) {
RefreshHandler rh = new RefreshHandler() {
public void handleRefresh(final Page page, final URL url, final int seconds) { }
};
client.setRefreshHandler(rh);
return client;
}