Develop Reference

Selenium opening browser but not loading websites in Java

I am using Selenium in a Burp plugin but I can't load pages with the get method. Browsers open correctly, both Firefox and Chrome, but they don't load the page. Chrome address bar shows "data;.", while Firefox has no text in it. I am using the last driver available, Chrome 81.0.4044.183 and the driver for this exact version, while Firefox is 76.0.1 and I am using GeckoDriver 0.24 (since 0.25+ have a known bug) and it works with the last version of Firefox.
The code is the following
void runBrowserAutomatization(File fileDriver, String seleniumTrack, boolean isHeadless) {
WebDriver driver;
if (gui.usedBrowser().toLowerCase().contains("chrome")) {
ChromeOptions options = new ChromeOptions();
Proxy proxy = new Proxy();
proxy.setHttpProxy("localhost:8080");
proxy.setSslProxy("localhost:8080");
options.setCapability(CapabilityType.PROXY, proxy);
options.setHeadless(isHeadless);
System.setProperty("webdriver.chrome.driver", fileDriver.getPath());
driver = new ChromeDriver(options);
} else if (gui.usedBrowser().toLowerCase().contains("firefox")) {
FirefoxOptions options = new FirefoxOptions();
Proxy proxy = new Proxy();
proxy.setHttpProxy("localhost:8080");
proxy.setSslProxy("localhost:8080");
options.setCapability(CapabilityType.PROXY, proxy);
options.setHeadless(isHeadless);
System.setProperty("webdriver.gecko.driver", fileDriver.getPath());
driver = new FirefoxDriver(options);
} else {
PrintMsg("No browser selected...");
return;
}
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.get("https://www.nytimes.com/");
driver.quit();
}
I may also think it is a Proxy misconfiguration, Burp certificate is installed in Firefox and Windows (where Chrome gets Certificate Authorities) but is not shown in the settings of the instance started by Selenium. Any help or suggestion is highly appreciated, thnaks.

Getting Request And Response Using BrowserMobProxy, Selenium, Firefox, marionette/gecko

I'm trying to get response and request using BMP's RequestFilter and ResponseFilter. However, when the webpage loads, nothing gets printed in the console.
Everything else seems to work though. Maybe BMP is not watching GeckoDriver?
I'm using Firefox 50.0, BrowserMobProxy 2.1.2, Selenium 3.0.1, and GeckoDriver 0.11.1
The testing code is below. Could someone please help me?
Thank you very much!
BrowserMobProxy server = new BrowserMobProxyServer();
server.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
server.start();
int port = server.getPort();
server.addRequestFilter((request, content, info) -> {
String q = URLDecoder.decode(info.getOriginalUrl(), "UTF-8");
System.out.println("Request: "+q);
return null;
});
server.addResponseFilter((response, content, info) -> {
String type = response.headers().get("Content-Type");
System.out.println("Response: "+info.getOriginalRequest());
System.out.println(type);
});
Proxy proxy = ClientUtil.createSeleniumProxy(server);
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(CapabilityType.PROXY, proxy);
capabilities.setCapability("marionette", true);
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
capabilities.setCapability(CapabilityType.SUPPORTS_JAVASCRIPT, true);
FirefoxProfile fp = new FirefoxProfile();
capabilities.setCapability(FirefoxDriver.PROFILE, fp);
String gecko = "d:/Programming/java/geckodriver.exe";
System.setProperty("webdriver.gecko.driver", gecko);
driver = new FirefoxDriver(capabilities);
driver.get("https://google.com");;

In Firefox 51 and lower, there is a bug/missing feature in Selenium 3's GeckoDriver that prevents Firefox from picking up the proxy settings when setting CapabilityType.PROXY on the DesiredCapabilities object.
However, you can still set the proxy settings directly on the FirefoxProfile. There's an example of this in one of BMP's tests. Since you're already using a FirefoxProfile object, this would probably be a sensible solution for you. It would look something like this (replace localhost with a hostname/ip address as appropriate):
FirefoxProfile fp = new FirefoxProfile();
fp.setPreference("network.proxy.http", "localhost");
fp.setPreference("network.proxy.http_port", server.getPort());
fp.setPreference("network.proxy.ssl", "localhost");
fp.setPreference("network.proxy.ssl_port", server.getPort());
fp.setPreference("network.proxy.type", 1);
fp.setPreference("network.proxy.no_proxies_on", "");
This geckodriver issue also discusses a few other alternatives to using CapabilityType.PROXY on the DesiredCapabilities object.
UPDATE
According to the mozilla bug report, this issue is fixed in Firefox 52, which is scheduled to be released on March 7, 2017. In the meantime, the solution with the FirefoxProfile should work with 51 (and lower), and should also continue to work with 52+.

Firefox Error: "Your connection is not secure" while launching driver with Selenium 3.0.1 using Java

My Firefox version is 46.0.1 and Selenium version is 3.0.1.
I am getting error:
Your connection is not secure
while executing following code:
#Test
public void test() {
ProfilesIni profile = new ProfilesIni();
FirefoxProfile ffProfile = profile.getProfile("newCretedProfile");
ffProfile.setAcceptUntrustedCertificates(true);
ffProfile.setAssumeUntrustedCertificateIssuer(false);
System.setProperty("webdriver.gecko.driver", "D:\\SELENUIUM\\Drivers\\geckodriver.exe");
FirefoxDriver driver = new FirefoxDriver(ffProfile);
driver.get("http://www.google.com");
driver.quit();
}
I have created new firefox profile and followed steps from this url
Nevertheless it's not working and giving me same error while I launching any site.

Download Firefox 55 beta and set
capabilities.setCapability("acceptInsecureCerts", true);
Here is my code that works for Firefox 55 beta:
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setBrowserName("firefox");
capabilities.setCapability("acceptInsecureCerts", true);
RemoteWebDriver driver = new RemoteWebDriver(Environment.remoteWebDriverURL, capabilities);

I have tried this approach and it worked well for me.
Create new firefox profile by following below step.
Close all your firefox windows
In the Run dialog box, type in: ‘firefox.exe -p' and then Click OK.
Click “Create Profile”
Create a name for your new profile(say Selenium)
Click “Choose Folder”
Pick location something easy to find — like “C:\NewFirefoxProfile”
Click Finish
Now after selecting newly created profile, start Firefox. Open the specific url you were getting 'Secure Connection Issue', accept SSL certificates for this profile.
Now use the newly created firefox profile to run your selenium test. Modify below code as per your requirement.
System.setProperty("webdriver.firefox.marionette","D:\\SELENUIUM\\Drivers\\geckodriver.exe");
ProfilesIni profile = new ProfilesIni();
FirefoxProfile myprofile = profile.getProfile("C:\\NewFirefoxProfile");//location of your new firefox profile
WebDriver driver = new FirefoxDriver(myprofile);
driver.get("https://cacert.org/");

With FF v53+ and Se 3.x (summer 2017), advice from before (May?) 2017 is no longer true.
You have to use Marionette and set capability to True.
Took me few days to sort out all old and obsolete advice, yours for free. :-)

Looks like it is not supported yet by geckodriver/Marionette.
You can check below bugs for more information:-
https://github.com/mozilla/geckodriver/issues/93
https://bugzilla.mozilla.org/show_bug.cgi?id=1103196

If you want to run the tests on Firefox with Selenium 3.0 set the Firefox driver capability “marionette” to false.
#Test
public void test() {
DesiredCapabilities d = new DesiredCapabilities();
d.setCapability("marionette", false);
WebDriver driver = new FirefoxDriver(d);
driver.get("http://www.google.com");
driver.quit();
}

Set Selenium proxy programmatically

In my automation project I need to set proxy server. I tried with system variable settings and profile setting for firefox browser. But those technique does not work for me. Please any one help me in this regard.
Note: I also tried with executing shell command using java but I got stuck when password is asked.

You definitely don't need to set any System-level properties. This is one way to do it in Firefox:
FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("network.proxy.type", 1); // Manual proxy config
profile.setPreference("network.proxy.http", "proxy3.proxy.net");
profile.setPreference("network.proxy.http_port", 3128);
profile.setPreference("network.proxy.ssl", "proxy3.proxy.net");
profile.setPreference("network.proxy.ssl_port", 3128);
WebDriver driver = new FirefoxDriver(profile);
Or a more flexible, less browser-specific alternative:
org.openqa.selenium.Proxy proxy = new org.openqa.selenium.Proxy();
proxy.setHttpProxy("proxy3.proxy.net:3128");
proxy.setSslProxy("proxy3.proxy.net:3128");
DesiredCapabilities caps = DesiredCapabilities.firefox(); // or chrome() etc.
caps.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new FirefoxDriver(caps);

How to turn off the proxy settings on chrome browser through selenium webdriver?

When I navigate to my url on chrome, I get The system cannot find the file specified." . I thought it might be due to automatic proxy settings on chrome.
I want to explicitly turn off the proxy setting before starting chrome browser in selenium. I tried below, it isn't working. Can anyone help me
ChromeOptions options = new ChromeOptions();
DesiredCapabilities dc = DesiredCapabilities.chrome();
dc.setCapability("chrome.setProxyByServer", false);
System.setProperty("webdriver.chrome.driver",sChromeDriverPath);
WebDriver driver = new ChromeDriver();
No errors are thrown at any point of time but URL doesn't open up
Tia
Anjana

You need to pass the options object to the chrome driver when you initialize it. If you use a specific capability then pass it to the chromeDriver(), so that chrome knows what to start with. Also there is no JSON object as setProxyByServer in chrome, instead use noProxy JSON object. Check this out. Here's how -
Proxy proxy=startProxy();
proxy.setProxyType(ProxyType.MANUAL);
proxy.setNoProxy("");
ChromeOptions options = new ChromeOptions();
DesiredCapabilities dc = DesiredCapabilities.chrome();
dc.setCapability(CapabilityType.PROXY, proxy);
System.setProperty("webdriver.chrome.driver",sChromeDriverPath);
dc.setCapability(ChromeOptions.CAPABILITY, options);
WebDriver driver = new ChromeDriver(dc);
More info about chrome capabilities. Hope it helps.