Develop Reference

Headless chrome using Selenium-Java running tests in normal UI mode of browser

I am trying to run my application in headless mode using Chrome Browser, Selenium and Java. But it open a new chrome instance and start running the script in normal UI mode(can see the execution)
Is there anything wrong in code or browser compatibility.
OS - Windows 10,
Chrome Version - 85.0.4183.121
Below is my code -
package XYZ;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
public class Test{
public static void main(String args[]) {
System.setProperty("webdriver.chrome.driver",
"C:\\Users\\TestUser\\Desktop\\SeleniumWorkspace\\ABC\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("headless");
options.addArguments("window-size=1400,800");
options.addArguments("disable-gpu")
//options.addArguments("--headless", "--disable-gpu", "--window-size=1400,800","--ignore-certificate-errors");
WebDriver driver = new ChromeDriver(options);
driver.get("https://www.google.com");
System.out.println(driver.getCurrentUrl());
}
}

I can see you are missing "--" before passing the arguments in the code. Below is the correct code to use headless chrome while running your automated cases:
package XYZ;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
public class Test{
public static void main(String args[]) {
System.setProperty("webdriver.chrome.driver",
"C:\\Users\\TestUser\\Desktop\\SeleniumWorkspace\\ABC\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--headless");
options.addArguments("--disable-gpu");
options.addArguments("--window-size=1400,800");
WebDriver driver = new ChromeDriver(options);
driver.get("https://www.google.com");
System.out.println(driver.getCurrentUrl());
}
}
For me this is working fine. I hope this solves your probelm.

Timeout exception when using dev tools with selenium-java-4.0.0 and chromedriver v85

I'm trying to use selenium dev tools java API, and for multiple API methods I'm getting java.util.concurrent.TimeoutException.
For example I'm trying to use Network.clearBrowserCache, which should work accroding to chromedriver docs: https://chromedevtools.github.io/devtools-protocol/tot/Network/
I'm calling clearBrowserCache using following code:
chromeDriver.getDevTools().send(Network.clearBrowserCache())
It fails, but at the same time if I use other devTools commands like this:
chromeDriver.getDevTools().send(Browser.getVersion())
It returns data properly.
Chrome version is: 85.0.4183.39
Chromedriver version is: 85.0.4183.87
Selenium-java version is: 4.0.0-alpha-6

Try calling createSession before calling clearBrowserCache.
Using your setup, this works:
chromeDriver.getDevTools().createSession();
chromeDriver.getDevTools().send(Network.clearBrowserCache())
and this produces java.util.concurrent.TimeoutException:
chromeDriver.getDevTools().send(Network.clearBrowserCache())
You can verify that the browser cache has been cleared with this snippet:
ChromeDriver driver = new ChromeDriver();
driver.get("https://refreshyourcache.com/en/cache-test/");
Thread.sleep(2000);
driver.getDevTools().createSession();
driver.getDevTools().send(Network.clearBrowserCache());
driver.get("https://refreshyourcache.com/en/cache-test/");
Thread.sleep(5000);
If you run the code, the pages displayed in the test browser will show these images:
If you commment out the line driver.getDevTools().send(Network.clearBrowserCache()); then you get a different result:

Using Selenium 4.0.0-alpha-6, Chrome v85 and ChromeDriver v85.0 through google-chrome-devtools you must be able to use getVersion() method as follows:
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.browser.Browser;
public class BrowserGetVersion {
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver","C:\\WebDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-maximized");
ChromeDriver driver = new ChromeDriver(options);
DevTools devTools = driver.getDevTools();
devTools.createSession();
devTools.send(Browser.getVersion());
}
}
Similarly, using the clearBrowserCache() method you should be able to clear the browser cache using the following code block:
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.network.Network;
public class ClearChromeCache {
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver","C:\\WebDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-maximized");
ChromeDriver driver = new ChromeDriver(options);
DevTools devTools = driver.getDevTools();
devTools.createSession();
devTools.send(Network.clearBrowserCache());
driver.get("https://www.google.com/");
}
}
Additional Consideration
Additionally, you can also use setCacheDisabled(true) to completely disable the cache as follows:
Code Block:
import java.util.Collections;
import java.util.Optional;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.network.Network;
import org.testng.Assert;
import org.testng.annotations.Test;
public class testngBasic {
#Test
public void foo() {
System.setProperty("webdriver.chrome.driver","C:\\WebDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-maximized");
options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
options.setExperimentalOption("useAutomationExtension", false);
ChromeDriver driver = new ChromeDriver(options);
DevTools devTools = driver.getDevTools();
devTools.createSession();
devTools.send(Network.clearBrowserCache());
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.of(100000000)));
devTools.send(Network.setCacheDisabled(true));
devTools.addListener(Network.responseReceived(), responseReceived -> Assert.assertEquals(false, responseReceived.getResponse().getFromDiskCache()));
driver.get("https://www.google.com/");
}
}
This usecase
Possibly your code have nothing to do with java.util.concurrent.TimeoutException error and the real issue is either with the:
jdk version
guava version
Solution
Ensure that:
JDK is upgraded to current levels JDK 8u252.
guava is upgraded to guava-29.0-jre.
Outro
Disable cache in Selenium Chrome Driver

It worked fine
public void testCdt {
final ChromeLauncher launcher = new ChromeLauncher();
final ChromeService chromeService = launcher.launch(false);
final ChromeTab tab = chromeService.createTab();
final ChromeDevToolsService devToolsService = chromeService.createDevToolsService(tab);
final Page page = devToolsService.getPage();
Network network = devToolsService.getNetwork();
// Clear browser cached
network.clearBrowserCache();
// Log requests with onRequestWillBeSent event handler.
network.onRequestWillBeSent(
event ->
System.out.printf(
"request: %s %s%s",
event.getRequest().getMethod(),
event.getRequest().getUrl(),
System.lineSeparator()));
network.onLoadingFinished(
event -> {
chromeService.closeTab(tab);
launcher.close();
});
network.enable();
page.navigate("http://github.com");
devToolsService.waitUntilClosed();
}

Standalone vs embedded Browsermob proxy for cloud-based tests (Sauce Labs/Browserstack)

I made a working test locally with embedded Browsermob proxy server. Nothing new, but still here is the sample code.
_server = new BrowserMobProxyServer()
_server.start();
Proxy proxy = ClientUtil.createSeleniumProxy(_server);
ChromeOptions options = new ChromeOptions();
options.setCapability("proxy", proxy);
_driver = new ChromeDriver(options);
Now we're looking into options to integrate such tests into our CI pipeline and to execute these tests in the cloud (Browserstack/Sauce Labs). I'm trying to figure out what the setup will look like in this case. Right now my understanding is that the code (which sets up the proxy and actually contains the tests) will run on our server. This means that the embedded proxy will also run on our server which is not necessarily accessible from the outside. So the questions are:
Will I have to switch to a standalone browsermob proxy and make it accessible?
If yes, then is there any actual code sample of using the standalone proxy from code? (This options doesn't look particularly appealing since we'll have to write boilerplate code to wrap the REST API)
If no, then am I correct assuming the remote Selenium Webdriver will connect to the website under test through the newly set up embedded proxy by means of tunnelling (Sauce Connect and alike)?
What is the best practice of using Browsermob with CI server with cloud-based testing platforms?

If the test/webdriver instance will be running on a remote machine (browserstack or sauce) in your case, it is essential that the proxy generated by your proxy server should be authenticated on the remote machine to intercept traffic. I had a similar requirement and I set it up using a standalone BrowserMob instance. Below is a working sample code for browserstack with their local testing binary:
This will need the following Dependencies:
<dependency>
<groupId>com.browserstack</groupId>
<artifactId>browserstack-local-java</artifactId>
<version>1.0.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.5</version>
<scope>test</scope>
</dependency>
Code snippet:
import com.browserstack.local.Local;
import net.lightbody.bmp.BrowserMobProxy;
import net.lightbody.bmp.BrowserMobProxyServer;
import net.lightbody.bmp.client.ClientUtil;
import net.lightbody.bmp.core.har.Har;
import net.lightbody.bmp.proxy.CaptureType;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.ie.InternetExplorerDriver;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import java.io.File;
import java.net.URL;
import java.util.HashMap;
public class InterceptProxy {
public static final String USERNAME = <BrowserStack Username>;
public static final String AUTOMATE_KEY = <BrowserStack Key>;
public static final String URL = "https://" + USERNAME + ":" + AUTOMATE_KEY + "#hub-cloud.browserstack.com/wd/hub";
public static void main(String[] args) throws Exception {
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.start(0);
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
Local browserStackLocal = new Local();
HashMap<String, String> browserStackLocalArgs = new HashMap<String, String>();
browserStackLocalArgs.put("key", AUTOMATE_KEY);
browserStackLocalArgs.put("forcelocal", "true");
browserStackLocalArgs.put("forceproxy","true");
browserStackLocalArgs.put("force","true");
browserStackLocalArgs.put("v", "true");
String host=seleniumProxy.getHttpProxy().substring(0,seleniumProxy.getHttpProxy().indexOf(":"));
String port=seleniumProxy.getHttpProxy().substring(seleniumProxy.getHttpProxy().indexOf(":")+1,seleniumProxy.getHttpProxy().length());
browserStackLocalArgs.put("-local-proxy-host", host);
browserStackLocalArgs.put("-local-proxy-port", port);
browserStackLocal.start(browserStackLocalArgs);
DesiredCapabilities caps = new DesiredCapabilities();
caps.setCapability("browser", "Chrome");
caps.setCapability("browser_version", "62.0");
caps.setCapability("os", "Windows");
//caps.setCapability(CapabilityType.PROXY, seleniumProxy);
caps.setCapability("os_version", "10");
caps.setCapability("browserstack.local",true);
WebDriver driver = new RemoteWebDriver(new URL(URL), caps);
driver.manage().deleteAllCookies();
driver.manage().window().maximize();
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
// create a new HAR with the label "yahoo.com"
proxy.newHar("yahoo.com");
// open yahoo.com
driver.get("http://yahoo.com");
// get the HAR data
Har har = proxy.getHar();
//Writing Har to file
har.writeTo(new File("/Users/MyUser/Desktop/HAR.txt"));
driver.quit();
browserStackLocal.stop();
proxy.stop();
}
}

How to effectively change the browser name of the WebDriver object?

I intend to perform some tests by using selenium with multiple web browsers. To distinguish between the different web drivers, I use the following line of code:
((RemoteWebDriver) driver).getCapabilities().getBrowserName();
This will return a String indicating the web browser that is used by the driver object. However, for my Opera WebDriver object this will give me the String 'chrome'. I have tried changing this by explicitly setting the browser name to 'opera' using DesiredCapabilities:
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setBrowserName("opera");
WebDriver driver = new OperaDriver(capabilities);
Unfortunately, this does not fix my problem. How do I effectively change the web browser name?

Your basic requirement is to identify browser initialization if I'm right, which can be done by getting user-agent from your browser using JavascriptExecutor as following:
String userAgent = (String) ((JavascriptExecutor) driver).executeScript("return navigator.userAgent;");
//following is for identifying opera browser initialization
if(userAgent.contains("OPR/"){
System.out.println("Browser currently in use is Opera");
}
Similarly you can identify other browser initilisation by referring this link

Unfortunately , you would not be able to change the BrowserName.
What instead you can try is to create function for specifically handling multiple browsers: -
package multiBrowser;
import org.testng.annotations.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxProfile;
import org.openqa.selenium.firefox.internal.ProfilesIni;
import org.openqa.selenium.ie.InternetExplorerDriver;
import org.openqa.selenium.opera.OperaDriver;
import org.testng.annotations.Parameters;
public class MultiBrowserClass {
WebDriver driver;
#Test
#Parameters("browser")
public void multiBrowsers(String browserName) throws InterruptedException{
if(browserName.equalsIgnoreCase("firefox")){
System.setProperty("webdriver.firefox.marionette","D:\\My Work\\Setup\\JAR\\geckodriver.exe");
ProfilesIni profile = new ProfilesIni();
FirefoxProfile myprofile = profile.getProfile("default");
driver = new FirefoxDriver(myprofile);
}
if(browserName.equalsIgnoreCase("chrome")){
System.setProperty("webdriver.chrome.driver", "D:\\My Work\\Setup\\JAR\\driver\\chromedriver.exe");
driver = new ChromeDriver();
}
else if(browserName.equalsIgnoreCase("IE")){
System.setProperty("webdriver.ie.driver", "D:\\My Work\\Setup\\JAR\\driver\\IEDriverServer.exe");
driver = new InternetExplorerDriver();
}
else if(browserName.equalsIgnoreCase("opera")){
System.setProperty("webdriver.opera.driver", "D:\\My Work\\Setup\\JAR\\driver\\operadriver.exe");
driver = new OperaDriver();
}
driver.manage().window().maximize();
driver.navigate().to("https://");
System.out.println(driver.getTitle());
driver.findElement(By.xpath("//div[#id='navbar-main']/ul/li[5]/a")).click();
driver.findElement(By.xpath("//div[#id='navbar-main']/ul/li[5]/ul/li/a")).click();
Thread.sleep(3000);
driver.findElement(By.name("email")).clear();
driver.findElement(By.name("email")).sendKeys("abc#mm.kk");
driver.findElement(By.name("password")).clear();
driver.findElement(By.name("password")).sendKeys("1qaz2wsx");
Thread.sleep(3000);
driver.findElement(By.xpath("//form[#id='loginform']/div[8]/button")).click();
Thread.sleep(5000);
if(driver.getPageSource().contains("Welcome abc#mm.kk")){
System.out.println("User Successfully logged in");
}else{
System.out.println("Username or password you entered is incorrect");
}
driver.quit();
}
}
=======

How to use SSL certificates in Selenium Web Driver?

I'm using Selenium Web Driver on Windows 7.
I'm trying to test a website that use authentication and I need to use SSL certificates.
When I use Firefox out of Selenium all works fine but I've noted that the Firefox browser session opened by Selenium doesn't have any certificates registered and so it's clear that it doesn't work.
Here you are the Advanced Preferences when I use Firefox "out" of Selenium
and here you are the same when I use the Firefox sessione opened by Selenium
I've tried to keep the session opened and to register manually the certificate but the browser session doesn't register the certificate.
Here you are my code if could be useful
package myTestProjects;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
public class GAMOPERA_Test_01 {
private static WebDriver driver = null;
public static void main(String[] args) throws InterruptedException {
// Create a new instance of the Firefox driver
System.out.println("Creo una nuova sessione del browser Firefox ...");
driver = new FirefoxDriver();
//Put a Implicit wait, this means that any search for elements on the page could take the time the implicit wait is set for before throwing exception
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
// It is always advisable to Maximize the window before performing DragNDrop action
System.out.println("Massimizzo la finestra del browser ...");
driver.manage().window().maximize();
Thread.sleep(3000L);
//Launch the Sistema Piemonte Home Page
System.out.println("Mi collego a Sistema Piemonte ...");
driver.get("http://<my_site_url>");
Thread.sleep(3000L);
// Find the element Accedi o
System.out.println("Accesso tramite certificato digitale ...");
driver.findElement(By.xpath("/html/body/div[6]/div/div/div[2]/form/table/tbody/tr[3]/td/input")).click();
//driver.findElement(By.className("loginbutton")).click();
Thread.sleep(3000L);
// Print TEST = OK!!
System.out.println("TEST = OK !!");
//driver.quit();
}
}
Any suggestions?

I've solved!
Surfing on the web I've found this post http://seleniummonk.blogspot.it/p/how-to-handle-ssl-cerificates.html that gave me the solution.
I need to use the "Firefox profile" (I use the default one ...), so I can have all the certificates I need to.
Here you're the new code that works
package myTestProjects;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxProfile;
import org.openqa.selenium.firefox.internal.ProfilesIni;
public class GAMOPERA_Test_01 {
private static WebDriver driver = null;
public static void main(String[] args) throws InterruptedException {
ProfilesIni profile = new ProfilesIni();
FirefoxProfile ffProfile = profile.getProfile("default");
// Create a new instance of the Firefox driver
System.out.println("Creo una nuova sessione del browser Firefox ...");
driver = new FirefoxDriver(ffProfile);
//Put a Implicit wait, this means that any search for elements on the page could take the time the implicit wait is set for before throwing exception
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
// It is always advisable to Maximize the window before performing DragNDrop action
System.out.println("Massimizzo la finestra del browser ...");
driver.manage().window().maximize();
Thread.sleep(3000L);
//Launch the Sistema Piemonte Home Page
System.out.println("Mi collego a Sistema Piemonte ...");
driver.get("<my_site_url>");
Thread.sleep(3000L);
// Find the element Accedi o
System.out.println("Accesso tramite certificato digitale ...");
driver.findElement(By.xpath("/html/body/div[6]/div/div/div[2]/form/table/tbody/tr[3]/td/input")).click();
//driver.findElement(By.className("loginbutton")).click();
Thread.sleep(3000L);
// Print TEST = OK!!
System.out.println("TEST = OK !!");
//driver.quit();
}
}
I hope this could be useful!

You can do that using Proxy. Through DesiredCapabilities you can configure the browser accordingly.
String PROXY = "localhost:8080";
org.openqa.selenium.Proxy proxy = new org.openqa.selenium.Proxy();
proxy.setHttpProxy(PROXY)
.setFtpProxy(PROXY)
.setSslProxy(PROXY);
DesiredCapabilities cap = new DesiredCapabilities();
cap.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new InternetExplorerDriver(cap);
Code taken from SeleniumHQ

First create a profile as I created with "Test" in Firefox using below command in CMD:
"C:\Program Files\Mozilla Firefox\firefox.exe" -P
Then import your certificate in this newly created Firefox Profile with Password used to generate the certificate.
In my case this was a P12 extenstion Certificate.
Once done you can use below code in Selenium and Firefox will not popup for Certificate and you will be logged into the Portal or Website.
ProfilesIni allProfiles = new ProfilesIni();
FirefoxProfile myProfile = allProfiles.getProfile("Test");
myProfile.setPreference("security.default_personal_cert", "Select Automatically");
FirefoxOptions firefoxoptions = new FirefoxOptions();
firefoxoptions.setProfile(myProfile);
WebDriver driver = new FirefoxDriver(firefoxoptions);
Background I am using Firefox 56.0 and Windows 7