Develop Reference

Selenium opening browser but not loading websites in Java

I am using Selenium in a Burp plugin but I can't load pages with the get method. Browsers open correctly, both Firefox and Chrome, but they don't load the page. Chrome address bar shows "data;.", while Firefox has no text in it. I am using the last driver available, Chrome 81.0.4044.183 and the driver for this exact version, while Firefox is 76.0.1 and I am using GeckoDriver 0.24 (since 0.25+ have a known bug) and it works with the last version of Firefox.
The code is the following
void runBrowserAutomatization(File fileDriver, String seleniumTrack, boolean isHeadless) {
WebDriver driver;
if (gui.usedBrowser().toLowerCase().contains("chrome")) {
ChromeOptions options = new ChromeOptions();
Proxy proxy = new Proxy();
proxy.setHttpProxy("localhost:8080");
proxy.setSslProxy("localhost:8080");
options.setCapability(CapabilityType.PROXY, proxy);
options.setHeadless(isHeadless);
System.setProperty("webdriver.chrome.driver", fileDriver.getPath());
driver = new ChromeDriver(options);
} else if (gui.usedBrowser().toLowerCase().contains("firefox")) {
FirefoxOptions options = new FirefoxOptions();
Proxy proxy = new Proxy();
proxy.setHttpProxy("localhost:8080");
proxy.setSslProxy("localhost:8080");
options.setCapability(CapabilityType.PROXY, proxy);
options.setHeadless(isHeadless);
System.setProperty("webdriver.gecko.driver", fileDriver.getPath());
driver = new FirefoxDriver(options);
} else {
PrintMsg("No browser selected...");
return;
}
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.get("https://www.nytimes.com/");
driver.quit();
}
I may also think it is a Proxy misconfiguration, Burp certificate is installed in Firefox and Windows (where Chrome gets Certificate Authorities) but is not shown in the settings of the instance started by Selenium. Any help or suggestion is highly appreciated, thnaks.

Getting Request And Response Using BrowserMobProxy, Selenium, Firefox, marionette/gecko

I'm trying to get response and request using BMP's RequestFilter and ResponseFilter. However, when the webpage loads, nothing gets printed in the console.
Everything else seems to work though. Maybe BMP is not watching GeckoDriver?
I'm using Firefox 50.0, BrowserMobProxy 2.1.2, Selenium 3.0.1, and GeckoDriver 0.11.1
The testing code is below. Could someone please help me?
Thank you very much!
BrowserMobProxy server = new BrowserMobProxyServer();
server.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
server.start();
int port = server.getPort();
server.addRequestFilter((request, content, info) -> {
String q = URLDecoder.decode(info.getOriginalUrl(), "UTF-8");
System.out.println("Request: "+q);
return null;
});
server.addResponseFilter((response, content, info) -> {
String type = response.headers().get("Content-Type");
System.out.println("Response: "+info.getOriginalRequest());
System.out.println(type);
});
Proxy proxy = ClientUtil.createSeleniumProxy(server);
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(CapabilityType.PROXY, proxy);
capabilities.setCapability("marionette", true);
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
capabilities.setCapability(CapabilityType.SUPPORTS_JAVASCRIPT, true);
FirefoxProfile fp = new FirefoxProfile();
capabilities.setCapability(FirefoxDriver.PROFILE, fp);
String gecko = "d:/Programming/java/geckodriver.exe";
System.setProperty("webdriver.gecko.driver", gecko);
driver = new FirefoxDriver(capabilities);
driver.get("https://google.com");;

In Firefox 51 and lower, there is a bug/missing feature in Selenium 3's GeckoDriver that prevents Firefox from picking up the proxy settings when setting CapabilityType.PROXY on the DesiredCapabilities object.
However, you can still set the proxy settings directly on the FirefoxProfile. There's an example of this in one of BMP's tests. Since you're already using a FirefoxProfile object, this would probably be a sensible solution for you. It would look something like this (replace localhost with a hostname/ip address as appropriate):
FirefoxProfile fp = new FirefoxProfile();
fp.setPreference("network.proxy.http", "localhost");
fp.setPreference("network.proxy.http_port", server.getPort());
fp.setPreference("network.proxy.ssl", "localhost");
fp.setPreference("network.proxy.ssl_port", server.getPort());
fp.setPreference("network.proxy.type", 1);
fp.setPreference("network.proxy.no_proxies_on", "");
This geckodriver issue also discusses a few other alternatives to using CapabilityType.PROXY on the DesiredCapabilities object.
UPDATE
According to the mozilla bug report, this issue is fixed in Firefox 52, which is scheduled to be released on March 7, 2017. In the meantime, the solution with the FirefoxProfile should work with 51 (and lower), and should also continue to work with 52+.

Selenium (Chrome) and BrowserMob doesn't work for https

I have been trying to integrate BrowserMob to my selenium tests. It works fine with website that work on http, but with https websites the browsers stop working and the HAR file doesn't contain any requests.
When navigating to a https site I get this error on the browser.
"There is something wrong with the proxy server or the address is incorrect."
Here is my code.
public class Browsermob {
BrowserMobProxy proxy = new BrowserMobProxyServer();
#Test
public void browsermobtest() {
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
// enable more detailed HAR capture, if desired (see CaptureType for the complete list)
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
// create a new HAR with the label "google.com"
proxy.newHar("http://www.google.com/");
// open google.com
driver.get("https://www.google.ee/#gfe_rd=cr");
driver.findElement(By.cssSelector("#gb_70")).click();
}
#AfterMethod
public void Afterthetest() {
// get the HAR data
Har har = proxy.getHar();
File harFile = new File("C:/Users/Madis/Documents/har.har");
try {
har.writeTo(harFile);
} catch (IOException e) {
e.printStackTrace();
}
}
}

You don't need to specify the sslProxy on the Selenium Proxy object. ClientUtil.createSeleniumProxy does this for you, and in most simple cases it chooses a suitable default value (using InetAddress.getLocalHost(); if that's working for HTTP, it will work for HTTPS as well).
A few things to keep in mind:
You'll receive SSL warnings in the browser unless you either tell the browser to ignore cert errors (on Chrome, use the --ignore-certificate-errors command-line flag), or install the BMP CA in the browser's trust store (for Chrome on Windows, you must install it in the Windows trust store).
Depending on your version of Chrome and OS, you may need to specify an alternate user-data-dir using a command line option. For example, --user-data-dir=/tmp/insecurechrome.
BMP has its own source of trusted certificates (Java trust store + a recent list from Mozilla), so if you're trying to connect to internal websites with certificates issued by a private CA, you need to tell BMP to either trust the private CA or skip certificate validation using .setTrustAllServers(true).
The proxy must be started using .start(...) before calling createSeleniumProxy().
Combining all these things, your code would look something like this:
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// NOTE: there is no call to .setSslProxy() here
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArgument("--ignore-certificate-errors");
// replace 'somedirectory' with a suitable temp dir on your filesystem
options.addArgument("--user-data-dir=somedirectory");
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
WebDriver driver = new ChromeDriver(capabilities);
// [...]

I had this problem. After numerous trials, I got to know we have to add setmitmManager and upstream proxy if you are connected to corporate proxy. It worked for me.
Here is the example code.
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
//Add below line if you are under corporate proxy.
proxy.setChainedProxy(new InetSocketAddress("XXX.XXX.com", 8080));
proxy.setMitmManager(ImpersonatingMitmManager.builder().trustAllServers(true).build());
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver","C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
// your code to start, get har

You're confusing the browser mob proxy object and the selenium proxy object.
Your proxy variable proxy is the actual proxy which your browser will connect to.
Your seleniumProxy variable is an object which represents your browser's proxy settings.
You are telling your browser to use "trustAllSSLCertificates" as the address for your proxy server, which is why you are getting an error. Instead, you should tell browsermob (proxy) to trustAllSSLCertificates, and your sslProxy needs to reference your browsermob proxy.
Start the proxy like so:
public void startProxy() {
proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
proxy.start(9091);
}
Start the driver like so:
public void startBrowserWithProxy() {
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
seleniumProxy.setSslProxy("localhost:" + proxy.getPort());
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
}

I managed to get it to work. After adding log4j and debugging the browsermob logs the issue was caused by
Caught an exception on ClientToProxyConnection
java.lang.NoSuchMethodError: com.google.common.net.HostAndPort.fromHost(Ljava/lang/String;)Lcom/google/common/net/HostAndPort;
In order to make it to work I had to add a dependency to my maven project. This fixed this issue and I was able to see the capture the traffic on https sites aswell http sites.
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>20.0</version>
</dependency>

I hade a lot for capabilities, options and etc but it did not work
In my case, I changed exist dependency in pom
<artifactId>browsermob-core-littleproxy</artifactId>
to
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.5</version>
</dependency>
and up "guava" version.
After that everything became good

Slow Selenium tests when using proxy with BrowserMob

I am setting up a ChromeDriver using BrowserMob(http://bmp.lightbody.net/) for intercepting HTTP responses.
ProxyServer proxyServer = null;
proxyServer = new ProxyServer(9101);
proxyServer.start();
proxyServer.setCaptureContent(true);
proxyServer.setCaptureHeaders(true);
Proxy proxy = proxyServer.seleniumProxy();
proxy.setHttpProxy("localhost:9101");
proxyServer.addResponseInterceptor(new ResponseInterceptor() {
#Override
public void process(BrowserMobHttpResponse response, Har har) {
if (response.getRawResponse().getStatusLine().getStatusCode() == 500) {
// do something
}
}
});
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability(CapabilityType.PROXY, proxy);
System.setProperty("webdriver.chrome.driver", "seleniumdrivers/chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--lang=nl");
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
this.driver = new ChromeDriver(capabilities);
When running a Selenium test every page load is extremely slow. Without the proxy it works fine. Anyone knows the reason/ solution for this?
In the log console the following message appears: jan 10, 2014 12:58:06 PM net.sf.uadetector.datastore.AbstractUpdateOperation isUpdateAvailable
INFO: Can not check for an updated version. Are you sure you have an established internet connection?
No idea if this is related.
Running Selenium tests on an online website (not local server), which means I have internet connection

Selenium 2: Unable to set PROXY for Internet Explorer

I want to set proxy and open IE (version 9, selenium 2.25) but this is not working for IE, though it is working for Firefox.
Here is my code:
String PROXY = "somehost.com" + ":" + "80";
org.openqa.selenium.Proxy proxy = new org.openqa.selenium.Proxy();
proxy.setHttpProxy(PROXY);
proxy.setFtpProxy(PROXY);
proxy.setSslProxy(PROXY);
DesiredCapabilities ieCapabilities = DesiredCapabilities.internetExplorer();
ieCapabilities.setCapability(InternetExplorerDriver.INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS, true);
ieCapabilities.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new InternetExplorerDriver(ieCapabilities);
driver.get("http://www.cnn.com/");
Following run time error occurred:
Aug 16, 2012 2:00:13 PM org.openqa.selenium.browserlaunchers.WindowsProxyManager backupRegistrySettings
INFO: Backing up registry settings...

The message you get is just right - it is no exception (but an informational message). To use IE via Selenium, some specific settings will be changed in the Registry (only for Selenium runs, BTW - and the registry will be restored afterwards).
Do you have the rights to change your registry (check if you can run regedit)?