Develop Reference

Selenium + Proxy = Fail

Proxy not working in Selenium
First of all i use free proxys from proxyfish (maybe thats the problem alrdy) https://hidemyna.me/en/proxy-checker/ tells me theyre working.
I found lots of code/solutions, most is outdated (capabilities=options..) or not working for me.
Tried several code/proxys chrome/ff however https://www.iplocation.net/ and https://whatismyipaddress.com showing actual IP address (IPv4 and IPv6).
What I tried..
System.setProperty("webdriver.chrome.driver", "C:\\selenium\\chromedriver.exe");
System.getProperties().put("74.208.112.***", "8080"); //1st try
Proxy proxy = new Proxy();
proxy.setHttpProxy("74.208.112.***:8080"); //2nd try
ChromeOptions options = new ChromeOptions();
options.setCapability("proxy", proxy);
options.setProxy(proxy);
options.addArguments("--proxy-server=74.208.112.***:8080"); //3rd try
ChromeDriver driver = new ChromeDriver(options);
driver.get("https://www.iplocation.net/");
driver.get("https://whatismyipaddress.com/");
Solution:
proxy.setSslProxy("74.208.112.***:8080");
Is there a way to test proxy connection before using proxy?

How to accept personal certificate using selenium java

We are trying to load a client URL and it is must to accept the personal certificate installed in my local machine. I am using robot keys to click on 'OK' button on the certificate popup. By the time it is clicking on 'Ok' button i am experiencing session timeout and the script is failing. I have also tried to reduce the implicit time.
Is there a way to solve the issue by setting chrome capabilities to select the personal certificate based on the name of the certificate(I have Multiple certificates based on the user) during the driver initialization.

For accepting alert you could use below code :
DesiredCapabilities caps = DesiredCapabilities.chrome ()
caps.setCapability (CapabilityType.ACCEPT_SSL_CERTS, true)
WebDriver driver = new ChromeDriver (caps);

You can try something like this at driver initialization level.
Code :
ChromeOptions options = new ChromeOptions();
options.setCapability(capabilityName, value);
//options.setAcceptInsecureCerts(acceptInsecureCerts)
WebDriver driver = new ChromeDriver(options);

Selenium (Chrome) and BrowserMob doesn't work for https

I have been trying to integrate BrowserMob to my selenium tests. It works fine with website that work on http, but with https websites the browsers stop working and the HAR file doesn't contain any requests.
When navigating to a https site I get this error on the browser.
"There is something wrong with the proxy server or the address is incorrect."
Here is my code.
public class Browsermob {
BrowserMobProxy proxy = new BrowserMobProxyServer();
#Test
public void browsermobtest() {
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
// enable more detailed HAR capture, if desired (see CaptureType for the complete list)
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
// create a new HAR with the label "google.com"
proxy.newHar("http://www.google.com/");
// open google.com
driver.get("https://www.google.ee/#gfe_rd=cr");
driver.findElement(By.cssSelector("#gb_70")).click();
}
#AfterMethod
public void Afterthetest() {
// get the HAR data
Har har = proxy.getHar();
File harFile = new File("C:/Users/Madis/Documents/har.har");
try {
har.writeTo(harFile);
} catch (IOException e) {
e.printStackTrace();
}
}
}

You don't need to specify the sslProxy on the Selenium Proxy object. ClientUtil.createSeleniumProxy does this for you, and in most simple cases it chooses a suitable default value (using InetAddress.getLocalHost(); if that's working for HTTP, it will work for HTTPS as well).
A few things to keep in mind:
You'll receive SSL warnings in the browser unless you either tell the browser to ignore cert errors (on Chrome, use the --ignore-certificate-errors command-line flag), or install the BMP CA in the browser's trust store (for Chrome on Windows, you must install it in the Windows trust store).
Depending on your version of Chrome and OS, you may need to specify an alternate user-data-dir using a command line option. For example, --user-data-dir=/tmp/insecurechrome.
BMP has its own source of trusted certificates (Java trust store + a recent list from Mozilla), so if you're trying to connect to internal websites with certificates issued by a private CA, you need to tell BMP to either trust the private CA or skip certificate validation using .setTrustAllServers(true).
The proxy must be started using .start(...) before calling createSeleniumProxy().
Combining all these things, your code would look something like this:
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// NOTE: there is no call to .setSslProxy() here
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArgument("--ignore-certificate-errors");
// replace 'somedirectory' with a suitable temp dir on your filesystem
options.addArgument("--user-data-dir=somedirectory");
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
WebDriver driver = new ChromeDriver(capabilities);
// [...]

I had this problem. After numerous trials, I got to know we have to add setmitmManager and upstream proxy if you are connected to corporate proxy. It worked for me.
Here is the example code.
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
//Add below line if you are under corporate proxy.
proxy.setChainedProxy(new InetSocketAddress("XXX.XXX.com", 8080));
proxy.setMitmManager(ImpersonatingMitmManager.builder().trustAllServers(true).build());
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver","C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
// your code to start, get har

You're confusing the browser mob proxy object and the selenium proxy object.
Your proxy variable proxy is the actual proxy which your browser will connect to.
Your seleniumProxy variable is an object which represents your browser's proxy settings.
You are telling your browser to use "trustAllSSLCertificates" as the address for your proxy server, which is why you are getting an error. Instead, you should tell browsermob (proxy) to trustAllSSLCertificates, and your sslProxy needs to reference your browsermob proxy.
Start the proxy like so:
public void startProxy() {
proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
proxy.start(9091);
}
Start the driver like so:
public void startBrowserWithProxy() {
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
seleniumProxy.setSslProxy("localhost:" + proxy.getPort());
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
}

I managed to get it to work. After adding log4j and debugging the browsermob logs the issue was caused by
Caught an exception on ClientToProxyConnection
java.lang.NoSuchMethodError: com.google.common.net.HostAndPort.fromHost(Ljava/lang/String;)Lcom/google/common/net/HostAndPort;
In order to make it to work I had to add a dependency to my maven project. This fixed this issue and I was able to see the capture the traffic on https sites aswell http sites.
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>20.0</version>
</dependency>

I hade a lot for capabilities, options and etc but it did not work
In my case, I changed exist dependency in pom
<artifactId>browsermob-core-littleproxy</artifactId>
to
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.5</version>
</dependency>
and up "guava" version.
After that everything became good

BrowserMob Proxy’s - WebDriver / Error: The proxy server is refusing connections

I try to use BrowserMob Proxy’s with WebDriver. I use the next code:
public static void main(String[] args) throws Exception {
String strFilePath = "";
// start the proxy
ProxyServer server = new ProxyServer(4455);
server.start();
//captures the moouse movements and navigations
server.setCaptureHeaders(true);
server.setCaptureContent(true);
// get the Selenium proxy object
Proxy proxy = server.seleniumProxy();
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
// start the browser up
WebDriver driver = new FirefoxDriver(capabilities);
// create a new HAR with the label "apple.com"
server.newHar("assertselenium.com");
// open yahoo.com
driver.get("http://assertselenium.com");
driver.get("http://assertselenium.com/2012/10/30/transformation-from-manual-tester-to-a-selenium-webdriver-automation-specialist/");
// get the HAR data
Har har = server.getHar();
FileOutputStream fos = new FileOutputStream(strFilePath);
har.writeTo(fos);
server.stop();
driver.quit();
}
And I got the next error: The proxy server is refusing connections: Firefox is configured to use a proxy server that is refusing connections.
I try also to run the browsermob-proxy.bat with port 4455, and then I get the next error when I run the main:
java.net.BindException: Address already in use: JVM_Bind
How I can use BrowserMob Proxy’s?

The code for stating the proxy seems to be correct. For the BindException, it should be obvious that something is already using the port 4455. You can check it (on Windows machine, written from memory):
netstat -ano | find "4455"
in Linux use lsof -i:4455 to get the PID and kill it.
Anyway, for your proxy refusing connections, try setting the proxy explicitly, see if you have any luck, something like
proxy.setHttpProxy("localhost:4455");
proxy.setSslProxy("localhost:4455");
Also, make sure you are using up-to-date versions of FF and BMP.

java.net.BindException: Address already in use: JVM_Bind
You get this error because on the mentioned port there is already one server running. May be you run your code again without stopping the server you started it at first instance.

Try disabling internet explorer proxy on your pc.

Selenium Webdrivers: Load Page without any resources

I am trying to prevent Javascript from changing the site's source code I'm testing with Selenium. The problem is, I can't just simply turn Javascript off in the Webdriver, because I need it for a test. Here's what I'm doing for the Firefox Webdriver:
firefoxProfile.setPreference("permissions.default.image", 2);
firefoxProfile.setPreference("permissions.default.script", 2);
firefoxProfile.setPreference("permissions.default.stylesheet", 2);
firefoxProfile.setPreference("permissions.default.subdocument", 2);
I don't allow Firefox to load any Images, Scripts and Stylesheets.
How can I do this with the Internet Explorer Webdriver and the Chrome Webdriver? I have not found any similar preferences. Or is there even a more elegant way to stop the webdrivers from loading the site's JS Files after all?
Thank you!

Solution is to use proxy. Webdriver integrates very well with browsermob proxy: http://bmp.lightbody.net/
private WebDriver initializeDriver() throws Exception {
// Start the server and get the selenium proxy object
ProxyServer server = new ProxyServer(proxy_port); // package net.lightbody.bmp.proxy
server.start();
server.setCaptureHeaders(true);
// Blacklist google analytics
server.blacklistRequests("https?://.*\\.google-analytics\\.com/.*", 410);
// Or whitelist what you need
server.whitelistRequests("https?://*.*.yoursite.com/.*. https://*.*.someOtherYourSite.*".split(","), 200);
Proxy proxy = server.seleniumProxy(); // Proxy is package org.openqa.selenium.Proxy
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
// start the driver ;
Webdriver driver = new FirefoxDriver(capabilities);
//WebDriver driver = new InternetExplorerDriver();
return driver;
}

Probably the easiest way to accomplish what you want in a cross-browser way is to use a proxy. This would allow you to intercept requests for resources, and block them. This would also have the advantage of using the same code for all browsers, rather than having to special-case each browser with settings unique to that browser.