Develop Reference

Selenium & Java - specified proxy ignored

I am learning selenium testing and have encountered my first problem after few hours :D
Objective: add a proxy into my selenium program so that it will not access the internet straight from my router.
(Proxy- running tor browser on localhost. Works with other web scrapers smoothly)
The problem is, it seems like program does not see proxy settings or goes around it. Navigating program to "check ip" websites shows original ip, not changed one by proxy.
My code: [as in documentation https://www.selenium.dev/documentation/webdriver/http_proxies/]
Proxy proxy = new Proxy();
proxy.setHttpProxy("127.0.0.1:9150");
ChromeOptions options = new ChromeOptions();
options.setCapability("proxy", proxy);
Then I tried several other approaches found online (including a deprecated one)
/*String proxy = "127.0.0.1:9150";
ChromeOptions options = new ChromeOptions().addArguments("--proxy-server=http://" +proxy);
*/
/*String proxy = "localhost:9150";
Proxy p = new Proxy();
p.setHttpProxy(proxy);
DesiredCapabilities cap = new DesiredCapabilities();
cap.setCapability(CapabilityType.PROXY, p);
*/
and then it goes into driver parameter (commented ones are using options and cap)
WebDriver driver = new ChromeDriver(options);
Using the documentation code I do not get any error. It just does not change ip.
I tried everything I could think of but without success.
Notes:
ip & port are correct
websites are accessed after setting up proxy.
using commented code (first block, two lines) returns "This site can’t be reached" in chrome and " org.openqa.selenium.WebDriverException: unknown error: net::ERR_TUNNEL_CONNECTION_FAILED" in console
same code as note 3) but protocol changed to https "...https://" +proxy);" returns "No internet" in browser and "org.openqa.selenium.WebDriverException: unknown error: net::ERR_PROXY_CONNECTION_FAILED" in console

Your troubleshooting, may have highlighted the cause. 3) shows that it could not create the tunnel with your proxy using http. 4) shows tunnel created using https, but proxy connection failed. So it is probably looking for some type of creds.
Update your code to add SSL proxy:
Proxy proxy = new Proxy();
proxy.setHttpProxy("127.0.0.1:9150");
proxy.setSslProxy("127.0.0.1:9150");
ChromeOptions options = new ChromeOptions();
options.setCapability("proxy", proxy);

Cannot Setup SOCKS Proxy with Firefoxdriver

I have a problem with SOCKS5 proxy with firefox driver in Java.
final FirefoxOptions FIREFOX_OPTIONS = new FirefoxOptions();
final Proxy PROXY = new Proxy().setProxyType(Proxy.ProxyType.MANUAL).setSocksVersion(5).setSocksProxy
("host:port");
FIREFOX_OPTIONS.setHeadless(true);
desiredCapabilities = DesiredCapabilities.firefox();
desiredCapabilities.setCapability(CapabilityType.PROXY, PROXY);
webDriver = new FirefoxDriver(FIREFOX_OPTIONS.merge(desiredCapabilities));`
After starting session I receive this error message.
Caused by: org.openqa.selenium.remote.UnreachableBrowserException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure.
Without setCapability, firefox driver seems working fine, but without proxy connection.
desiredCapabilities.setCapability(CapabilityType.PROXY, PROXY);
Thanks for replies.

It might be something like:
FIREFOX_OPTIONS.addArguments(--firefox.preference "network.proxy.socks:host" + ":" + --firefox.preference "network.proxy.socks_port:port");
The syntax may be incorrect because I'm not familiar with firefox or java but I believe the arguments may have to be added. In chrome it is:
.addArguments("--proxy-server=socks5://" + host + ":" + port);

Getting Request And Response Using BrowserMobProxy, Selenium, Firefox, marionette/gecko

I'm trying to get response and request using BMP's RequestFilter and ResponseFilter. However, when the webpage loads, nothing gets printed in the console.
Everything else seems to work though. Maybe BMP is not watching GeckoDriver?
I'm using Firefox 50.0, BrowserMobProxy 2.1.2, Selenium 3.0.1, and GeckoDriver 0.11.1
The testing code is below. Could someone please help me?
Thank you very much!
BrowserMobProxy server = new BrowserMobProxyServer();
server.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
server.start();
int port = server.getPort();
server.addRequestFilter((request, content, info) -> {
String q = URLDecoder.decode(info.getOriginalUrl(), "UTF-8");
System.out.println("Request: "+q);
return null;
});
server.addResponseFilter((response, content, info) -> {
String type = response.headers().get("Content-Type");
System.out.println("Response: "+info.getOriginalRequest());
System.out.println(type);
});
Proxy proxy = ClientUtil.createSeleniumProxy(server);
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(CapabilityType.PROXY, proxy);
capabilities.setCapability("marionette", true);
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
capabilities.setCapability(CapabilityType.SUPPORTS_JAVASCRIPT, true);
FirefoxProfile fp = new FirefoxProfile();
capabilities.setCapability(FirefoxDriver.PROFILE, fp);
String gecko = "d:/Programming/java/geckodriver.exe";
System.setProperty("webdriver.gecko.driver", gecko);
driver = new FirefoxDriver(capabilities);
driver.get("https://google.com");;

In Firefox 51 and lower, there is a bug/missing feature in Selenium 3's GeckoDriver that prevents Firefox from picking up the proxy settings when setting CapabilityType.PROXY on the DesiredCapabilities object.
However, you can still set the proxy settings directly on the FirefoxProfile. There's an example of this in one of BMP's tests. Since you're already using a FirefoxProfile object, this would probably be a sensible solution for you. It would look something like this (replace localhost with a hostname/ip address as appropriate):
FirefoxProfile fp = new FirefoxProfile();
fp.setPreference("network.proxy.http", "localhost");
fp.setPreference("network.proxy.http_port", server.getPort());
fp.setPreference("network.proxy.ssl", "localhost");
fp.setPreference("network.proxy.ssl_port", server.getPort());
fp.setPreference("network.proxy.type", 1);
fp.setPreference("network.proxy.no_proxies_on", "");
This geckodriver issue also discusses a few other alternatives to using CapabilityType.PROXY on the DesiredCapabilities object.
UPDATE
According to the mozilla bug report, this issue is fixed in Firefox 52, which is scheduled to be released on March 7, 2017. In the meantime, the solution with the FirefoxProfile should work with 51 (and lower), and should also continue to work with 52+.

Firefox Error: "Your connection is not secure" while launching driver with Selenium 3.0.1 using Java

My Firefox version is 46.0.1 and Selenium version is 3.0.1.
I am getting error:
Your connection is not secure
while executing following code:
#Test
public void test() {
ProfilesIni profile = new ProfilesIni();
FirefoxProfile ffProfile = profile.getProfile("newCretedProfile");
ffProfile.setAcceptUntrustedCertificates(true);
ffProfile.setAssumeUntrustedCertificateIssuer(false);
System.setProperty("webdriver.gecko.driver", "D:\\SELENUIUM\\Drivers\\geckodriver.exe");
FirefoxDriver driver = new FirefoxDriver(ffProfile);
driver.get("http://www.google.com");
driver.quit();
}
I have created new firefox profile and followed steps from this url
Nevertheless it's not working and giving me same error while I launching any site.

Download Firefox 55 beta and set
capabilities.setCapability("acceptInsecureCerts", true);
Here is my code that works for Firefox 55 beta:
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setBrowserName("firefox");
capabilities.setCapability("acceptInsecureCerts", true);
RemoteWebDriver driver = new RemoteWebDriver(Environment.remoteWebDriverURL, capabilities);

I have tried this approach and it worked well for me.
Create new firefox profile by following below step.
Close all your firefox windows
In the Run dialog box, type in: ‘firefox.exe -p' and then Click OK.
Click “Create Profile”
Create a name for your new profile(say Selenium)
Click “Choose Folder”
Pick location something easy to find — like “C:\NewFirefoxProfile”
Click Finish
Now after selecting newly created profile, start Firefox. Open the specific url you were getting 'Secure Connection Issue', accept SSL certificates for this profile.
Now use the newly created firefox profile to run your selenium test. Modify below code as per your requirement.
System.setProperty("webdriver.firefox.marionette","D:\\SELENUIUM\\Drivers\\geckodriver.exe");
ProfilesIni profile = new ProfilesIni();
FirefoxProfile myprofile = profile.getProfile("C:\\NewFirefoxProfile");//location of your new firefox profile
WebDriver driver = new FirefoxDriver(myprofile);
driver.get("https://cacert.org/");

With FF v53+ and Se 3.x (summer 2017), advice from before (May?) 2017 is no longer true.
You have to use Marionette and set capability to True.
Took me few days to sort out all old and obsolete advice, yours for free. :-)

Looks like it is not supported yet by geckodriver/Marionette.
You can check below bugs for more information:-
https://github.com/mozilla/geckodriver/issues/93
https://bugzilla.mozilla.org/show_bug.cgi?id=1103196

If you want to run the tests on Firefox with Selenium 3.0 set the Firefox driver capability “marionette” to false.
#Test
public void test() {
DesiredCapabilities d = new DesiredCapabilities();
d.setCapability("marionette", false);
WebDriver driver = new FirefoxDriver(d);
driver.get("http://www.google.com");
driver.quit();
}

How to set BrowserMob with WebDriver?

I am trying to intercept the requests made by Webdriver using BrowserMobProxy.
But the below code is not working..It is not able to open the site google.com.
It says the "Internet Explorer cannot open the site"
proxyServer = new ProxyServer(9101);
proxyServer.start();
proxyServer.setCaptureHeaders(true);
proxyServer.setCaptureContent(true);
proxyServer.addRequestInterceptor(new RequestInterceptor() {
#Override
public void process(BrowserMobHttpRequest request, Har har) {
System.out.println("From Process method");
}
});
seleniumProxy = proxy.seleniumProxy();
seleniumProxy.setHttpProxy("localhost:9101");
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
capabilities.setCapability("ie.setProxyByServer", true);
File file = new File("C:\\path\\IEDriverServer.exe");
System.setProperty("webdriver.ie.driver", file.getAbsolutePath());
driver = new InternetExplorerDriver(capabilities);
driver.get("www.google.com");
I get the below error when trying to access google.com from webdrviver
From Process method
Nov 09, 2014 2:07:58 AM net.lightbody.bmp.proxy.util.Log info
INFO: java.net.UnknownHostException: www.google.com when requesting http://www.google.com/

Browsermob uses xbill DNS instead of regular Java/native DNS resolution, which may not play nice with your VPN. The latest browsermob snapshot allows you to enable native DNS fallback by setting the system property bmp.allowNativeDnsFallback to true:
System.setProperty("bmp.allowNativeDnsFallback", "true");
proxyServer = new ProxyServer(9101);
proxyServer.start();
You can get the latest snapshot at the browsermob github page.