Develop Reference

Selenium & Java - specified proxy ignored

I am learning selenium testing and have encountered my first problem after few hours :D
Objective: add a proxy into my selenium program so that it will not access the internet straight from my router.
(Proxy- running tor browser on localhost. Works with other web scrapers smoothly)
The problem is, it seems like program does not see proxy settings or goes around it. Navigating program to "check ip" websites shows original ip, not changed one by proxy.
My code: [as in documentation https://www.selenium.dev/documentation/webdriver/http_proxies/]
Proxy proxy = new Proxy();
proxy.setHttpProxy("127.0.0.1:9150");
ChromeOptions options = new ChromeOptions();
options.setCapability("proxy", proxy);
Then I tried several other approaches found online (including a deprecated one)
/*String proxy = "127.0.0.1:9150";
ChromeOptions options = new ChromeOptions().addArguments("--proxy-server=http://" +proxy);
*/
/*String proxy = "localhost:9150";
Proxy p = new Proxy();
p.setHttpProxy(proxy);
DesiredCapabilities cap = new DesiredCapabilities();
cap.setCapability(CapabilityType.PROXY, p);
*/
and then it goes into driver parameter (commented ones are using options and cap)
WebDriver driver = new ChromeDriver(options);
Using the documentation code I do not get any error. It just does not change ip.
I tried everything I could think of but without success.
Notes:
ip & port are correct
websites are accessed after setting up proxy.
using commented code (first block, two lines) returns "This site can’t be reached" in chrome and " org.openqa.selenium.WebDriverException: unknown error: net::ERR_TUNNEL_CONNECTION_FAILED" in console
same code as note 3) but protocol changed to https "...https://" +proxy);" returns "No internet" in browser and "org.openqa.selenium.WebDriverException: unknown error: net::ERR_PROXY_CONNECTION_FAILED" in console

Your troubleshooting, may have highlighted the cause. 3) shows that it could not create the tunnel with your proxy using http. 4) shows tunnel created using https, but proxy connection failed. So it is probably looking for some type of creds.
Update your code to add SSL proxy:
Proxy proxy = new Proxy();
proxy.setHttpProxy("127.0.0.1:9150");
proxy.setSslProxy("127.0.0.1:9150");
ChromeOptions options = new ChromeOptions();
options.setCapability("proxy", proxy);

How to check if there is a Chrome instance already open with a specific port in selenium

I have the following code
try {
System.setProperty("webdriver.chrome.driver", Sources.SOURCE_PATH + "chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("debuggerAddress", "127.0.0.1:9222");
driver = new ChromeDriver(options);
ex = (JavascriptExecutor)driver;
} catch (Exception e) {
e.printStackTrace();
}
I would like to check if there is already an instance of Chrome open with the port 9222 before executing. Is this possible? If so how would I go about doing this. Any help would be appreciated.

You can try the following:
Make an HTTP request to the url below
import requests
res = requests.get('http://localhost:9222/selenium-server/driver/?cmd=shutDownSeleniumServer')
If selenium is not running on this port 9222 then by hitting above URL it will give you
Unable to connect
If selenium server is already running on port 9222 then it will shut down the server and will give you
OKOK.
You can then use these to handle execution as you desire.

Selenium (Chrome) and BrowserMob doesn't work for https

I have been trying to integrate BrowserMob to my selenium tests. It works fine with website that work on http, but with https websites the browsers stop working and the HAR file doesn't contain any requests.
When navigating to a https site I get this error on the browser.
"There is something wrong with the proxy server or the address is incorrect."
Here is my code.
public class Browsermob {
BrowserMobProxy proxy = new BrowserMobProxyServer();
#Test
public void browsermobtest() {
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
// enable more detailed HAR capture, if desired (see CaptureType for the complete list)
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
// create a new HAR with the label "google.com"
proxy.newHar("http://www.google.com/");
// open google.com
driver.get("https://www.google.ee/#gfe_rd=cr");
driver.findElement(By.cssSelector("#gb_70")).click();
}
#AfterMethod
public void Afterthetest() {
// get the HAR data
Har har = proxy.getHar();
File harFile = new File("C:/Users/Madis/Documents/har.har");
try {
har.writeTo(harFile);
} catch (IOException e) {
e.printStackTrace();
}
}
}

You don't need to specify the sslProxy on the Selenium Proxy object. ClientUtil.createSeleniumProxy does this for you, and in most simple cases it chooses a suitable default value (using InetAddress.getLocalHost(); if that's working for HTTP, it will work for HTTPS as well).
A few things to keep in mind:
You'll receive SSL warnings in the browser unless you either tell the browser to ignore cert errors (on Chrome, use the --ignore-certificate-errors command-line flag), or install the BMP CA in the browser's trust store (for Chrome on Windows, you must install it in the Windows trust store).
Depending on your version of Chrome and OS, you may need to specify an alternate user-data-dir using a command line option. For example, --user-data-dir=/tmp/insecurechrome.
BMP has its own source of trusted certificates (Java trust store + a recent list from Mozilla), so if you're trying to connect to internal websites with certificates issued by a private CA, you need to tell BMP to either trust the private CA or skip certificate validation using .setTrustAllServers(true).
The proxy must be started using .start(...) before calling createSeleniumProxy().
Combining all these things, your code would look something like this:
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// NOTE: there is no call to .setSslProxy() here
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArgument("--ignore-certificate-errors");
// replace 'somedirectory' with a suitable temp dir on your filesystem
options.addArgument("--user-data-dir=somedirectory");
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
WebDriver driver = new ChromeDriver(capabilities);
// [...]

I had this problem. After numerous trials, I got to know we have to add setmitmManager and upstream proxy if you are connected to corporate proxy. It worked for me.
Here is the example code.
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
//Add below line if you are under corporate proxy.
proxy.setChainedProxy(new InetSocketAddress("XXX.XXX.com", 8080));
proxy.setMitmManager(ImpersonatingMitmManager.builder().trustAllServers(true).build());
proxy.start(9091);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver","C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
// your code to start, get har

You're confusing the browser mob proxy object and the selenium proxy object.
Your proxy variable proxy is the actual proxy which your browser will connect to.
Your seleniumProxy variable is an object which represents your browser's proxy settings.
You are telling your browser to use "trustAllSSLCertificates" as the address for your proxy server, which is why you are getting an error. Instead, you should tell browsermob (proxy) to trustAllSSLCertificates, and your sslProxy needs to reference your browsermob proxy.
Start the proxy like so:
public void startProxy() {
proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
proxy.start(9091);
}
Start the driver like so:
public void startBrowserWithProxy() {
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
seleniumProxy.setSslProxy("localhost:" + proxy.getPort());
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "C:/Users/Madis/Documents/chromedriver.exe");
WebDriver driver = new ChromeDriver(capabilities);
}

I managed to get it to work. After adding log4j and debugging the browsermob logs the issue was caused by
Caught an exception on ClientToProxyConnection
java.lang.NoSuchMethodError: com.google.common.net.HostAndPort.fromHost(Ljava/lang/String;)Lcom/google/common/net/HostAndPort;
In order to make it to work I had to add a dependency to my maven project. This fixed this issue and I was able to see the capture the traffic on https sites aswell http sites.
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>20.0</version>
</dependency>

I hade a lot for capabilities, options and etc but it did not work
In my case, I changed exist dependency in pom
<artifactId>browsermob-core-littleproxy</artifactId>
to
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.5</version>
</dependency>
and up "guava" version.
After that everything became good

Open browser with proxy setting set from Java

I need to open a browser from Java code. I understand this can be done as follows :
java.awt.Desktop.getDesktop().browse(java.net.URI.create("http://google.com"));
But i need the browser to use certain proxy settings as well. (i.e. when the browser opens, its proxy settings must be set to certain values.) I tried using the follwoing code but it doesnt work :
public static void main(String asf[]){
System.setProperty("java.net.useSystemProxies", "true");
System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "8080");
try {
java.awt.Desktop.getDesktop().browse(java.net.URI.create("http://google.com"));
} catch (IOException e) {
e.printStackTrace();
}
System.out.println("done");
}
Setting the proxy from command line using
java -Dhttp.proxyHost=webcache.example.com -Dhttp.proxyPort=8080
is not an option for me. How do i accomplish this?

Your code is largely correct which deals with setting the proxy, but in case it is not working there is another way to set the proxy via Java code and that is via the proxy class.
SocketAddress addr = new InetSocketAddress("socks.example.com", 1080);
Proxy proxy = new Proxy(Proxy.Type.SOCKS, addr);
Socket socket = new Socket(proxy);
InetSocketAddress dest = new InetSocketAddress("server.example.org", 1234);
socket.connect(dest);
Here the socket will try to connect to its destination address (server.example.org:1234) through the specified SOCKS proxy.
For more detail you can go through the Standard Java Documentation for Proxies
Your solution for opening a browser can be improved by adding a check
if(Desktop.isDesktopSupported())
{
Desktop.getDesktop().browse(new URI("http://www.google.com"));
}
this is in addition to your solution .... maybe you can call it an alternate way
try {
Process p=Runtime.getRuntime().exec("cmd /c start http://www.google.com");
}
catch(IOException e1) {
System.out.println(e1);
}

The Google Chromes proxy switches can be useful here. We can just make a shortcut for the chrome browser whose target contains the switch --proxy-server=127.0.0.1:8080 . Now this shortcut can be opened from java code using the Runtime class' exec method. The arguments to exec will be "cmd /c start /d \"d:\" chrome.lnk" where d: is the path of my shortcut. A detailed description of this technique can be found here http://sleepingthreads.blogspot.in/2013/07/open-browser-with-proxy-settings-set.html
Note that Google states that the use of switches is not recommended. So use this as a temporary solution only.

Groovy URL UnknownHostException on Windows

I'm trying to build a groovy script that connects to a website. The webaddress ends in a non-standard format .abc.
I had this snippet of code working on a Linux box and now I am moving it over to a Windows box. The Windows box throws an UnknownHostException and fails.The website does render in browsers on both Linux and Windows.
def url = 'http://www.testURL.abc'
def connection = new URL(url).openConnection()
if (connection.responseCode != 200)
<<Error Handling>>
I believe it may be a proxy issue since both the Windows and Linux boxes are using different proxies to connect. I looked into this and configured Java on each box to use the proxy of the browser which didn't help either. At this point, I'm somewhat stuck. Any help would be greatly appreciated.
EDIT* Both proxies are using automatic configuration scripts (.pac files)
** Updated syntax errors from copying them over

I ended up finding a solution through the proxy issue. I had to download the automatic configuration script (.pac file) and find out which proxy host and port were being used for my URL.
I had to set the proxy host and port with the following code:
ProxySelector.setDefault(new ProxySelector() {
#Override
public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
throw new RuntimeException("Proxy connect failed", ioe);
}
#Override
public List select(URI uri) {
return Arrays
.asList(new Proxy(Proxy.Type.HTTP,
new InetSocketAddress(proxyHost,
proxyPort)));
}
});
This was code from unknown host exception