Develop Reference

Selenium can't open Gitlab login page

I'm doing an automated test, using Java (8), Selenium (4) and Chromedriver (98.0).
The test is for a site that requires to login with different third party accounts, one of them being GitLab.Unfortunately the test always gets stuck while trying to access Gitlab's login page, on the "Checking your browser before accessing gitlab.com" part. If I pause the test on this step and duplicate the tab manually, the newly opened tab will be able to enter and then the first one will be also able to do it (probably because at that point finally has a valid cookie).I've tried out different solutions but with no luck. Currently this is my code:
#Test
public void test() throws MalformedURLException {
ChromeOptions options = new ChromeOptions();
options.addArguments("--disable-dev-shm-usage");
options.addArguments("--disable-blink-features=AutomationControlled");
options.addArguments("--start-maximized");
options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
options.setExperimentalOption("useAutomationExtension", false);
RemoteWebDriver driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), options);
driver.navigate().to("https://gitlab.com/users/sign_in");
new FluentWait<>(driver)
.withTimeout(Duration.ofSeconds(60))
.pollingEvery(Duration.ofSeconds(2))
.until(x -> driver.findElements(By.xpath("//*[#data-translate='checking_browser']")).size() == 0);
}
When trying to use undetected_chromedriver, using Python, it worked, but it's a requirement for me to use Java. Is there something similar for Java or is there an extra ChromeOption that I'm missing?

need to download chrome driver https://chromedriver.chromium.org/downloadshttps://chromedriver.chromium.org/downloads
and add this code while setting up driver
System.setProperty("webdriver.chrome.driver", "/path/to/chromedriver");
WebDriver driver = new ChromeDriver();

Add ssl certificate to selenium-webdriver

I use selenium for end-to-end test with chromeDriver. The websites to test require an ssl certificate. When I manually open the browser, there is a popup that lets me select an installed certificate. Different tests access different URLs and also need different certificates. However, if I run the tests in headless mode, there is no popup. So I need a way to programatically set a certificate (eg. set a .pem file) to be used for the current test.
How can I achieve this?
I tried setting up a browserMob proxy which I then configured as a proxy in selenium - however, this does not seem to do anything... Are there better approaches? What am I doing wrong? Here's what I tried:
PemFileCertificateSource pemFileCertificateSource = new PemFileCertificateSource(
new File("myCertificate.pem"),
new File("myPrivateKey.pem"),
"myPrivateKeyPassword");
ImpersonatingMitmManager mitmManager = ImpersonatingMitmManager.builder()
.rootCertificateSource(pemFileCertificateSource)
.build();
BrowserMobProxy browserMobProxy = new BrowserMobProxyServer();
browserMobProxy.setTrustAllServers(true);
browserMobProxy.setMitmManager(mitmManager);
browserMobProxy.start(8080);
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.setProxy(ClientUtil.createSeleniumProxy(browserMobProxy));
WebDriver webDriver = new ChromeDriver(chromeOptions);
// use the webdriver for tests, e.g. assertEquals("foo", webDriver.findElement(...))

So apparantly this is not possible with BrowserMob out of the box. I therefore wrote a proxy extension SeleniumSslProxy that can be plugged into Selenium and adds certificate based authentication to create a HTTPS connection.
This is how it works:
intercept Selenium HTTP requests with BrowserMob
setup an SSLContext given a certificate (.pfx file) and password
use okhttp to forward the request to the target URL
convert the okhttp Response to a netty FullHttpResponse so it can be handled by Selenium
You can find the code on github. Here's an example how it can be used in Selenium end-to-end tests (also works in headless mode):
#Before
public void setup() {
ClassLoader classLoader = ClassLoader.getSystemClassLoader();
File clientSslCertificate = new File(
classLoader.getResource("certificates/some-certificate.pfx").getFile());
String certificatePassword = "superSecret";
this.proxy = new SeleniumSslProxy(clientSslCertificate, certificatePassword);
this.proxy.start();
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.setProxy(proxy);
this.webDriver = new ChromeDriver(chromeOptions);
}
#Test
public void pageTitleIsFoo() {
// given
String url = "http://myurl.lol";
// NOTE: do not use https in the URL here. It will be converted to https by the proxy.
// when
this.webDriver.get(url);
this.webDriver.manage().timeouts().implicitlyWait(5, SECONDS);
// then
WebElement title = this.webDriver.findElement(By.className("title"));
assertEquals("Foo", title.getText());
}
#After
public void teardown() {
this.webDriver.quit();
this.proxy.stop();
}
Note that I only used chromeDriver and never tested it with other drivers. Minor adjustments to the SeleniumSslProxy might be necessary to be used with other drivers.

Web-scraping: how to choose proxy

I want to write as reusable as possible web scraper. I gonna write it on Selenium + PhantomJS. PhantomJS will use a pool of IPs (proxies). There is a huge list of free proxies, for example.
How could I choose at runtime the best proxy for specific URL? By the best I mean the fastest an one which won't be blocked with target resource.
Workarround
I deployed my simplest app on Heroku. The app serves some html content. I used different proxies (with response time < 300ms) instead of 151.252.120.177:8080 (see code below), and noticed that most of them aren't able to parse simplest html in 15 seconds timeout. And some of them (that are even slower) retrieved content in a second. Why some proxies are unable to reach my content? Are they blacklisted with Heroku?
DesiredCapabilities caps = new DesiredCapabilities();
caps.setJavascriptEnabled(true);
caps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY, "drivers/phantomjs");
ArrayList<String> cliArgsCap = new ArrayList<String>();
cliArgsCap.add("--proxy=151.252.120.177:8080");
cliArgsCap.add("--proxy-type=socks");
caps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, cliArgsCap);
WebDriver driver = new PhantomJSDriver(caps);
driver.get(REMOTE_URL);
WebDriverWait wait = new WebDriverWait(driver, 15);
WebElement element = wait.until(ExpectedConditions.elementToBeClickable(By.className("btn-success")));
element.click();
driver.quit();

If you need just a scraper result you can use free scrapers with proxy support. Try this one http://datathief.verych.ru/

Selenium (Chrome) and BrowserMob doesn't work for https

I have been trying to integrate BrowserMob to my selenium tests. It works fine with website that work on http, but with https websites the browsers stop working and the HAR file doesn't contain any requests.
When navigating to a https site I get this error on the browser.
"There is something wrong with the proxy server or the address is incorrect."
Here is my code.
public class Browsermob {
BrowserMobProxy proxy = new BrowserMobProxyServer();
#Test
public void browsermobtest() {
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
// enable more detailed HAR capture, if desired (see CaptureType for the complete list)
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
// create a new HAR with the label "google.com"
proxy.newHar("http://www.google.com/");
// open google.com
driver.get("https://www.google.ee/#gfe_rd=cr");
driver.findElement(By.cssSelector("#gb_70")).click();
}
#AfterMethod
public void Afterthetest() {
// get the HAR data
Har har = proxy.getHar();
File harFile = new File("C:/Users/Madis/Documents/har.har");
try {
har.writeTo(harFile);
} catch (IOException e) {
e.printStackTrace();
}
}
}

You don't need to specify the sslProxy on the Selenium Proxy object. ClientUtil.createSeleniumProxy does this for you, and in most simple cases it chooses a suitable default value (using InetAddress.getLocalHost(); if that's working for HTTP, it will work for HTTPS as well).
A few things to keep in mind:
You'll receive SSL warnings in the browser unless you either tell the browser to ignore cert errors (on Chrome, use the --ignore-certificate-errors command-line flag), or install the BMP CA in the browser's trust store (for Chrome on Windows, you must install it in the Windows trust store).
Depending on your version of Chrome and OS, you may need to specify an alternate user-data-dir using a command line option. For example, --user-data-dir=/tmp/insecurechrome.
BMP has its own source of trusted certificates (Java trust store + a recent list from Mozilla), so if you're trying to connect to internal websites with certificates issued by a private CA, you need to tell BMP to either trust the private CA or skip certificate validation using .setTrustAllServers(true).
The proxy must be started using .start(...) before calling createSeleniumProxy().
Combining all these things, your code would look something like this:
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// NOTE: there is no call to .setSslProxy() here
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArgument("--ignore-certificate-errors");
// replace 'somedirectory' with a suitable temp dir on your filesystem
options.addArgument("--user-data-dir=somedirectory");
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
WebDriver driver = new ChromeDriver(capabilities);
// [...]

I had this problem. After numerous trials, I got to know we have to add setmitmManager and upstream proxy if you are connected to corporate proxy. It worked for me.
Here is the example code.
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
//Add below line if you are under corporate proxy.
proxy.setChainedProxy(new InetSocketAddress("XXX.XXX.com", 8080));
proxy.setMitmManager(ImpersonatingMitmManager.builder().trustAllServers(true).build());
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver","C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
// your code to start, get har

You're confusing the browser mob proxy object and the selenium proxy object.
Your proxy variable proxy is the actual proxy which your browser will connect to.
Your seleniumProxy variable is an object which represents your browser's proxy settings.
You are telling your browser to use "trustAllSSLCertificates" as the address for your proxy server, which is why you are getting an error. Instead, you should tell browsermob (proxy) to trustAllSSLCertificates, and your sslProxy needs to reference your browsermob proxy.
Start the proxy like so:
public void startProxy() {
proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
proxy.start(9091);
}
Start the driver like so:
public void startBrowserWithProxy() {
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
seleniumProxy.setSslProxy("localhost:" + proxy.getPort());
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
}

I managed to get it to work. After adding log4j and debugging the browsermob logs the issue was caused by
Caught an exception on ClientToProxyConnection
java.lang.NoSuchMethodError: com.google.common.net.HostAndPort.fromHost(Ljava/lang/String;)Lcom/google/common/net/HostAndPort;
In order to make it to work I had to add a dependency to my maven project. This fixed this issue and I was able to see the capture the traffic on https sites aswell http sites.
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>20.0</version>
</dependency>

I hade a lot for capabilities, options and etc but it did not work
In my case, I changed exist dependency in pom
<artifactId>browsermob-core-littleproxy</artifactId>
to
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.5</version>
</dependency>
and up "guava" version.
After that everything became good

Set Selenium proxy programmatically

In my automation project I need to set proxy server. I tried with system variable settings and profile setting for firefox browser. But those technique does not work for me. Please any one help me in this regard.
Note: I also tried with executing shell command using java but I got stuck when password is asked.

You definitely don't need to set any System-level properties. This is one way to do it in Firefox:
FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("network.proxy.type", 1); // Manual proxy config
profile.setPreference("network.proxy.http", "proxy3.proxy.net");
profile.setPreference("network.proxy.http_port", 3128);
profile.setPreference("network.proxy.ssl", "proxy3.proxy.net");
profile.setPreference("network.proxy.ssl_port", 3128);
WebDriver driver = new FirefoxDriver(profile);
Or a more flexible, less browser-specific alternative:
org.openqa.selenium.Proxy proxy = new org.openqa.selenium.Proxy();
proxy.setHttpProxy("proxy3.proxy.net:3128");
proxy.setSslProxy("proxy3.proxy.net:3128");
DesiredCapabilities caps = DesiredCapabilities.firefox(); // or chrome() etc.
caps.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new FirefoxDriver(caps);