I have been trying to access an HTTPS URL with the HTMLUnitDriver API of Selenium 2.0, but somehow the execution gets stuck at the "This Connection is Untrusted" window and and the control doesn't return back. Following is the code I have tried working on after I got some hint from this thread:
WebDriver driver = new HtmlUnitDriver() {
protected WebClient modifyWebClient(final WebClient client) {
try {
client.setUseInsecureSSL(true);
} catch (GeneralSecurityException e) {
e.printStackTrace();
}
return client;
}
};
driver.get("https://172.25.194.91:8443/meta/homeScreen.do");
I'd highly appreciate any help to get it work .
The issue was something else and is resolved now: HtmlUnitDriver uses WaitingRefreshHandler without parameters and unfortunately that is inappropriate for some sites - for example, HtmlUnitDriver hangs on http://news.google.com.
Cause & Scenario:
You load a page, which refreshes to itself after a period of time, because of the presence of a <meta http-equiv="refresh"...> directive in your HTML header.
WaitingRefreshHandler waits for the time specified but after that time elapses, it again redirects HtmlUnitDriver to get that page!
Consequently, you loop forever in this redirection process.
Solution:
One needs to extend the HtmlUnitDriver and override the modifyWebClient method to set a new (read: to clear the) refresh handler.
#Override
protected WebClient modifyWebClient(WebClient client) {
RefreshHandler rh = new RefreshHandler() {
public void handleRefresh(final Page page, final URL url, final int seconds) { }
};
client.setRefreshHandler(rh);
return client;
}
Related
I'm trying to check login page control by using dataprovider but i don't want to initialize webdriver again and again for each username password control. Once i come into login page, checking all concerned scenarios on login page in single time without starting another driver seems more convenient to me but i couldn't figure it out. When running following code, data[0][0] and data[0][1] is being correctly checked but it gives no such element on Login method having second priority test annotation when being tried to be typed data[1][0] and data[1][1]. Probably, it causes because driver is not looking at that page on that time. How can I handle this issue ?
error:
org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#class='q-input-wrapper email-input']//input[#class='q-input']"}
code:
public class TestCaseFirst {
public WebDriver driver;
#BeforeTest
public void Start() throws InterruptedException {
WebDriverManager.chromedriver().setup();
driver= new ChromeDriver();
driver.get("https://www.faxzas.com/");
driver.manage().window().maximize();
Thread.sleep(2000);}
#Test(priority=1)
public void RoadtoLogin() throws InterruptedException {
driver.findElement(By.xpath("//a[#title='Close']")).click();
Thread.sleep(1000);
driver.findElement(By.xpath("//div[#class='login-container']//span[#id='not-logged-in-container']")).click();;
Thread.sleep(1000);
}
#Test(dataProvider="loginInfos", priority=2)
public void Login(String mail, String password) throws InterruptedException {
driver.findElement(By.xpath("//div[#class='q-input-wrapper email-input']//input[#class='q-input']")).sendKeys(mail);
Thread.sleep(1000);
driver.findElement(By.xpath("//div[#class='q-input-wrapper']//input[#class='q-input']")).sendKeys(password);
Thread.sleep(1000);
driver.findElement(By.xpath("//button[#type='submit']")).click();
Thread.sleep(1000);
String description = driver.findElement(By.xpath("//div[#id='error-box-wrapper']//span[#class='message']")).getText();
System.out.println(description);
}
#DataProvider(name="loginInfos")
public Object[][] getData(){
Object[][] data = new Object[6][2];
data[0][0]="blackkfredo#gmail.com";
data[0][1]="";
data[1][0]="blackkfredo#gmail.com";
data[1][1]="443242";
data[2][0]="";
data[2][1]="1a2b3c4d";
data[3][0]="";
data[3][1]="";
data[4][0]="blackkfredogmail.com";
data[4][1]="1a2b3c4d";
data[5][0]="blackkfredo#gmail.com";
data[5][1]="1a2b3c4d";
return data;
}
}
You need to reset your page to the login page where you are expecting the element to be. Either put an #AfterMethod and go back to the page you are trying to test or put an #BeforeMethod for the same. You may even want to wrap up your find element calls and handle the exceptions by going back to the main page.
I have a problem that concerns the functioning of the site from which I want to take the data.
In practice when I download the HTML of the site it downloads everything but not the data I want, I noticed that when you open the site before the data I want comes out there is a buffering gif so I added a delay of many seconds (I also tried several minutes but the result does not change) to allow htmlUnit to load the site but apparently, the problem is not this. I have been looking for answers all afternoon but have found nothing.
I leave you the site: https://www.finderbet.it/surebet/
and the code:
import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import java.io.IOException;
public class Scraper {
private static final String url="https://www.finderbet.it/surebet/";
public static void main(String[] args) {
WebClient client= new WebClient(BrowserVersion.BEST_SUPPORTED);
client.getOptions().setCssEnabled(false);
client.getOptions().setJavaScriptEnabled(false);
try {
HtmlPage page= client.getPage(url);
Thread.sleep(10000);
System.out.println(page.asXml());
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
That "buffer" you are seeing is an AJAX call -- If you were to call it directly .. It gives the data you are looking for. IE
finderbet.it/wp-json/bet/v1/getItems
You can use get parameters to filter the JSON including pagination IE
https://www.finderbet.it/wp-json/bet/v1/getItems?surebet_do_set_filter=NOPE&action-set-filtri_nonce=d20ddf9c95&bookmakers=&sports=&data_evento_da=&data_evento_a=&profitto_min=&puntate=tutti&orderBy=profitto&order=desc&page=1
Which breaks down to:
https://www.finderbet.it/wp-json/bet/v1/getItems
surebet_do_set_filter=NOPE
action-set-filtri_nonce=d20ddf9c95
bookmakers=
sports=
data_evento_da=
data_evento_a=
profitto_min=
puntate=tutti
orderBy=profitto
order=desc
page=1
I use Selenium webdriver with Firefox for scraping web pages. Sometimes web browser waits endless time for some excessive requests complete (e.g. to facebook.net).
I've tried to use BrowserMob-Proxy to filter these requests. But it didn't help. These requests, even after receiving 200 or 404 code, doesn't stop.
I thought about some possibility to stop web browser loads page after some amount of time.
For example:
try {
Thread.sleep(5000);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt(); }
((JavascriptExecutor) driver).executeScript("window.stop();");
But it doesn't work until web page loads completely.
What can you suggest me to do in my case?
P.S. This is a code with using a pageLoadTimeout parameter.
WebDriver driver;
FirefoxBinary firefox;
FirefoxProfile customProfile;
public static void main(String[] args) {
openFirefox();
for (String url : listOfUrls) {
Boolean pageLoaded = false;
while (pageLoaded == false) {
try {
driver.get(url);
pageLoaded = true;
} catch (org.openqa.selenium.TimeoutException ex) {
System.out.println("Got TimeoutException on page load. Restarting browser...");
restartFirefox();
}
}
//here I do something with a content of a webpage
}
}
public static void openFirefox(){
firefox = new FirefoxBinary(new File(Constants.PATH_TO_FIREFOX_EXE));
customProfile = new FirefoxProfile();
customProfile.setAcceptUntrustedCertificates(true);
customProfile.setPreference("webdriver.load.strategy", "unstable");
driver = new FirefoxDriver(firefox, customProfile);
driver.manage().deleteAllCookies();
driver.manage().timeouts().pageLoadTimeout(60, TimeUnit.SECONDS);
}
private static void restartFirefox() {
driver.close();
firefox.quit();
openFirefox();
}
How about using timeouts? So for each WebDriver instance that you are using you need to set:
WebDriver.Timeouts pageLoadTimeout(long time, java.util.concurrent.TimeUnit unit)
Which by the Documentation:
Sets the amount of time to wait for a page load to complete before
throwing an error. If the timeout is negative, page loads can be
indefinite.
Parameters:
time - The timeout value.
unit - The unit of time. Returns:
A Timeouts interface.
I've tried to use BrowserMob-Proxy to filter these requests. But it
didn't help. These requests, even after receiving 200 or 404 code,
doesn't stop.
What do you mean "didn't help". I don't believe you. Please share your code for blacklisting URLs. For example, following code code returned HTTP.200 for any google-analytics related site for me
server.blacklistRequests("https?://.*\\.google-analytics\\.com/.*", 200); // server is bmp proxy server
I have heard, that WebDriver should now have webdriver.load.strategy. I have never used it though. So the default behavior of WebDrivers blocking calls (a'la get()) is to wait for document.readyState to be complete, but I have read that with this property you could tell the driver to return at once. So might be worth googling it for a while.
I am trying to parse a site, but I encountered a Too much redirect exception.
Here is my code:
WebClient client = new WebClient(BrowserVersion.FIREFOX_24);
HtmlPage homePage = null;
String url = "http://www.freelake.org/pages/Freetown-Lakeville_RSD/Departments/Director_of_Financial_Operatio";
try {
client.getOptions().setUseInsecureSSL(true);
client.setAjaxController(new NicelyResynchronizingAjaxController());
client.getOptions().setThrowExceptionOnFailingStatusCode(false);
client.getOptions().setThrowExceptionOnScriptError(false);
client.waitForBackgroundJavaScript(30000);
client.waitForBackgroundJavaScriptStartingBefore(30000);
client.getOptions().setCssEnabled(false);
client.getOptions().setJavaScriptEnabled(true);
client.getOptions().setRedirectEnabled(true);
homePage = client.getPage(url);
synchronized (homePage) {
homePage.wait(25000);
}
System.out.println(homePage.asXml());
} catch (Exception e) {
e.printStackTrace();
}
Exception are mention below
com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException: Too much redirect for http://www.freelake.org/resolver/2345183424.20480.0000/route.00/pages/Freetown-Lakeville_RSD/Departments/Director_of_Financial_Operatio
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1353)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1371)
Is there any way to solve this issue?
This is because HtmlUnit caches the response, and there is redirection to another page then returning back.
I tested with the below, and it works:
client.getCache().setMaxSize(0);
I was facing the same problem, but I am doing this through Selenium. In Selenium you cannot get access to the WebClient directly, because it is protected.
I worked around it like this:
WebDriver driver = new HtmlUnitDriver(true) {
{
this.getWebClient().getCache().setMaxSize(0);
}
};
The page http://www.freelake.org/pages/Freetown-Lakeville_RSD/Departments/Director_of_Financial_Operatio sends 2 redirects:
http://www.freelake.org/GroupHome.page, then to
http://www.freelake.org/pages/Freetown-Lakeville_RSD/Departments/Director_of_Financial_Operatio
Use the 2nd url and it should work. Or look for a way to tell the library to allow a certain amount of redirects; 2 in this case.
Edit: This might help. Don't use this library myself:
client.getOptions().setRedirectEnabled(true);
I'm using MockWebServer library in my Android JUnit tests. I'm testing an SDK that makes calls to a server. So I'm using MockWebServer to override these server URLs and capture what the SDK is sending to make assertions on it.
The problem that I'm running into is that if I try to do server.takeRequest() and assign it to a new RecordedRequest variable, the test hangs up on the second server.takeRequest() and sometimes, even on the first one -- if I run it on an emulator it hangs on the first server.takeRequest() method but if I run it on my physical Android device, it freezes on the second server.takeRequest() method.
public void testSomething() {
final MockWebServer server = new MockWebServer();
try {
server.play();
server.enqueue(new MockResponse().setBody("")
.setResponseCode(HttpURLConnection.HTTP_INTERNAL_ERROR));
server.enqueue(new MockResponse().setBody("")
.setResponseCode(HttpURLConnection.HTTP_OK));
server.enqueue(new MockResponse().setBody("")
.setResponseCode(HttpURLConnection.HTTP_OK));
URL url = server.getUrl("/");
// This internal method overrides some of the hardcoded URLs
// within the SDK that I'm testing.
Util.overrideUrls(url.toString())
// Do some server calls via the SDK utilizing the mock server url.
RecordedRequest requestFor500 = server.takeRequest();
// Do some assertions with 'requestFor500'
// Do some more server calls via the SDK utilizing the mock server url.
/*
* This is the part where the JUnit test hangs or seems to go into
* an infinite loop and never recovers
*/
RecordedRequest requestAfter500Before200 = server.takeRequest();
} catch {
...
}
}
Am I doing something wrong or is this some type of bug with MockWebServer?
Add timeout to MockWebServer so that it does not hang
server.takeRequest(1, TimeUnit.SECONDS);
There seems to be a problem with MockWebServer's dispatch queue, which freezes for some reason when serving responses which are not 200 or 302. I have solved this by providing a custom dispatcher:
MockWebServer server = ...;
final MockResponse response = new MockResponse().setResponseCode(401);
server.setDispatcher(new Dispatcher() {
#Override
public MockResponse dispatch(RecordedRequest request)
throws InterruptedException {
return response; // this could have been more sophisticated
}
});
Tested with MockWebServer 2.0.0