Develop Reference

Is there a way to capture network using BrowserStack with Selenium?

I use BrowserStack with Selenium-webdriver to run tests on different types of devices and browsers. So actually tests are running by RemoteWebDriver.
I know that it's possible to capture network within Selenium tests using BrowserMobProxy, but as i understand it's working only if test is running on local machine.
Is there a way to capture network while running test on cross-platform base like BrowserStack?
UPDATE
I managed to get capture of network in har file (from link "localhost:8080/proxy/8081/har"), using standalone BrowserMobProxy and standalone local BrowserStack, as I was advised.
I tried to do the same automatically from code:
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.start(8080);
System.out.println("Proxy port: " + port);
System.setProperty("java.net.useSystemProxies", "true");
System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "8080");
System.setProperty("https.proxyHost", "127.0.0.1");
System.setProperty("https.proxyPort", "8080");
Local l = new Local();
Map<String, String> options = new HashMap<String, String>();
options.put("key", accessKey);
options.put("forcelocal", "true");`
//when I uncomment it i get an exception:
//com.browserstack.local.LocalException: Could not connect to www.browserstack.com!
// options.put("forceproxy", "true");
// options.put("proxyHost", "localhost");
// options.put("proxyPort", "8080");
l.start(options);
}
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
driver = new RemoteWebDriver(new URL("http://"+username+":"+accessKey+"#"+config.get("server")+"/wd/hub"), capabilities);'
proxy.newHar("testHar.com");
driver.get(testUrl);
Thread.sleep(15000);
Har har = proxy.getHar();
FileOutputStream fos = new FileOutputStream("C:\\LoadingPage\\network\\testHar.har");
har.writeTo(fos);
The connection to the url is working, I could see it and make screenshouts.
BUT! In the har file I see only request to "hub-cloud.browserstack.com/wd/hub/...", not the requests from page itself.
How to get correct har from code?
What in the code is not correct?

From my experience I would like to add small modification to the binary command given in BrowserStack link (Shared by Mikhail). The cmd given in doc should work well for private URLs but may not work for public ones.
Steps for Standalone binary:
1 - Download the BrowserStackLocal binary from 'https://www.browserstack.com/local-testing#command-line'.
Launch the binary by running the below command to enable your proxy to monitor the traffic.
- BrowserStackLocal.exe --key
--local-proxy-host --local-proxy-port --local-proxy-user --local-proxy-pass --force-proxy --force-local
More details on all the modifiers are available at 'https://www.browserstack.com/local-testing#modifiers'.
2 - Include "browserstack.local" capability in your test script.
"browserstack.local" = true
Steps for Java (BrowserStack Local) bindings:
1 - Follow these steps for using local bindings.
2 - Using this you can use newer options available in latest versions of the binary. For instance if you wish to add --local-proxy-* options, for which there is no existing wrapper (like this which is internally mapped to this), try using below:
bsLocalArgs.put("-local-proxy-host", "Your BrowserMob proxy IP");
bsLocalArgs.put("-local-proxy-port", "Your BrowserMob proxy Port");
bsLocalArgs.put("-local-proxy-user", "Your BrowserMob proxy Username");
bsLocalArgs.put("-local-proxy-pass", "Your BrowserMob proxy Password");
3 - Include "browserstack.local" capability in your test script.
"browserstack.local" = true
How it works:
BrowserStack, by default, will resolve all the public URLs from their network.
Using --force-local option will force the binary to resolve all the traffic (even public URLs) via your network and not from BrowserStack's network.
Adding --local-proxy-* options will let the binary know that the traffic needs to be routed via your local proxy as well.
Now your local BrowserMob can capture all the traffic in HAR.

I see 2 solutions for this problem
BrowserMobProxy - there are 2 ways to run it: 1. from your code(adding library) 2. standalone proxy(controlled by REST API). In both cases you need to provide proxy to webdriver and control your proxy. One more improtant thing here to understand is that you need to redirect all the traffic from browsermob through the machine where proxy is located, please refer to this article for browserstack local execution. As I understand problem describes the case when proxy is being created on one machine and browserstack is simply not able to reach it.
Using browsermob you can get ALL required information: like request, params, response code, response time, etc. And even wait for requests to finish.
Examine performance logs. There is an option for ChromeDriver to capture performance logs. This option is easier since you don't need to care about proxy. However there are certain limitations of this approach as well: You won't be able to get request statistics like response time and response code and maybe some other that one may require. It would allow you to get only basic info like: request type, request url and of course you can parse params from url.

Using Proxy server for Java application

So i have a java app that uses Google Analytics API to gather some info. I am putting this application to run in my oracle cloud managed server which has a firewall and blocks any web calls to work. So, they setup a proxy for me to use....I've never set up a proxy to work with a java application before, I've been reading at tutorials like this: http://docs.oracle.com/javase/1.5.0/docs/guide/net/proxies.html
And i have no idea how to set this up...anyone want to point me in the right direction?

You must tell your application that there's a proxy somewhere.
As the documentation says, you must set some properties in your virtual machine. You can do it programatically:
//Set the http proxy to webcache.mydomain.com:8080
System.setProperty("http.proxyHost", "webcache.mydomain.com");
System.setPropery("http.proxyPort", "8080");
// Next connection will be through proxy.
URL url = new URL("http://java.sun.com/");
InputStream in = url.openStream();
// Now, let's 'unset' the proxy.
System.clearProperty("http.proxyHost");
// From now on http connections will be done directly.
Or use https.proxy... if the connection is HTTPS.
Besides, if you have access to the application server start script, you could add those properties as VM properties with -Dhttp.proxyHost....

The solution in my case was to configure the JVM with a HTTPS proxy:
System.setProperty("https.proxyHost", "proxy");
System.setProperty("https.proxyPort", "3128");

Proxy Implementation with WebLogic 11g

I am using WebLogic server 10.x (11g) to deploy and service my applications. However, our production environment consists of a proxy server which is causing troubles if I try to access certain sites.
Therefore, I would love to know if any feasible solution available to resolve this by configuring the proxy server details on our
WebLogic Environment
Application Level - I am using Java to develop my applications
Thanks in advance.

There are two possible ways to aaccomplish this.
Configuring WebLogic Env.
Programatically
Let's look at these.
Proxy Configuration in Application Server
Let's take WebLogic application server to illustrate the desired configuration
Configure WebLogic Server to Enable Proxy
1. Using WebLogic Administration Console
Login into Administration console
Go to Domain -> Environment -> Servers -> Admin Server -> Configuration -> Server Start
Inside Arguments section enter following and save.
-jvmargs -Dhttp.proxySet=true -Dhttp.proxyHost=server_url -Dhttp.proxyPort=port
Restart the WebLogic Server
2. Using WebLogic Startup Scripts
Open WebLogic domain configuration script at
Search for the following phrase "JAVA_OPTIONS"
Enter following details after that.
set JAVA_OPTIONS=%JAVA_OPTIONS% -Dhttp.proxyHost=server_url -Dhttp.proxyPort=port
Save script and start the WebLogic server
Using Programatically
There are several implementations available and let's discuss few approaches which I think would be useful.
Proxy Configuration Using Values
As highlights below you can configure a proxy using programatically defined values inside your application.
System.setProperty("http.proxyHost", "proxy_url");
System.setProperty("http.proxyPort", "port");
Configuration with User Credentials
Sometimes you may need to provide exact credentials to get through proxy server. Here is an implementation which will help you to achieve this.
import java.net.Authenticator;
import java.net.PasswordAuthentication;
public class MyAuthenticator extends Authenticator {
private String username;
private String password;
public MyAuthenticator(String username, String password){
this.username = username;
this.password = password;
}
public PasswordAuthentication getPasswordAuthentication () {
return new PasswordAuthentication (username, password.toCharArray());
}
}
The defined Authenticator class can be used to inject credentials to the proxy configuration as below.
System.setProperty("http.proxyHost", "proxy_url");
System.setProperty("http.proxyPort", "port");
Authenticator.setDefault (new MyAuthenticator("domain_name\\user_name","password"));
In either case you need to implement the usages at program level as given below. Otherwise, proxy communication will not success and end up throwing exceptions.
final URL url = new URL(null, urlString, new sun.net.www.protocol.http.Handler());

To configure WebLogic proxy file is modified setDomainEnv.cmd (Windows) that is located in the bin folder of the domain.
For example for the integrated server is here
C: \ Users \ [YOUT USER NAME] \ AppData \ Roaming \ JDeveloper \ system11.1.1.4.37.59.23 \ DefaultDomain \ bin
The following property must be placed on file with the details of your proxy
#REM Estableciendo proxi
set JAVA_OPTIONS=%JAVA_OPTIONS% -Dhttp.proxySet=true -Dhttp.proxyHost=192.168.101.11 -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=localhost
See the next link
Configure WebLogic Proxi

I had a similar problem in my developer environment running WebLogic 14.1.1.0.0 inside Eclipse IDE. I spend a day finding a working solution to configure a proxy server.
Finally I set the variable EXTRA_JAVA_PROPERTIES in startWebLogic.cmd
set EXTRA_JAVA_PROPERTIES=-Dhttp.proxyHost=proxy.example.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxy.example.com -Dhttps.proxyPort=3128
here is a tiny JSP page to test, if proxy setting are applied to the JVM:
<html>
<body>
<p><%=System.getProperty("http.proxyHost") %></p>
<p><%=System.getProperty("http.proxyPort") %></p>
<p><%=System.getProperty("https.proxyHost") %></p>
<p><%=System.getProperty("https.proxyPort") %></p>
</body>
</html>
There are solutions I tried, but which DID NOT WORK for me:
Setting EXTRA_JAVA_PROPERTIES in setDomainEnv.cmd as recommended by Oracle [1]
Setting JAVA_OPTIONS in startWebLogic.cmd as recommended by Aviro and Manish Kumar Gupta [2]
Setting Server Start Arguments over Administration Console
[1] Configure Proxy Settings for WebLogic Server
https://docs.oracle.com/cd/E53672_01/doc.111191/e53673/GUID-36A18CFA-55F1-4D19-B371-A6D15EBF2E24.htm
[2] Proxy server configuration for Weblogic Server !!!
https://kumarsoablog.blogspot.com/2016/10/proxy-server-configuration-for-weblogic.html

How to get default proxy settings in java program (not applet)?

I've just gone through the web searching how to get system proxy settings. I've found:
System.setProperty("java.net.useSystemProxies", "true");
but it does nothing. I have a proxy settings in my corpo network but the code that shows the proxy list:
ProxySelector.getDefault().select(new URI("http://foo/bar")))
says it's only one proxy "DIRECT". I don't want to provide the proxy settings by hand when it's already done. Is there a way to make JVM to provide proxy settings from OS/browser to the Java program (not applet)?
Ok,I think I got it: my browser proxy is set up by some script, defined in:
Internet Properties/Connections/LAN Settings/Use automatic
configuration script
Probably, that's why Java cannot list proxy properly, even it's used in the browser. Sad, that JVM cannot parse the script and provide these settings...

You have to set the property:
System.setProperty("java.net.useSystemProxies", "true");
in the main method, otherwise it get no effect, then call the getDefault() as you described.

How to configure freebase java client to connect through a proxy?

I am looking for a way to configure the freebase java api to connect to freebase through a proxy. I have set up sytem properties
System.setProperty("http.proxyHost", "proxy");
System.setProperty("http.proxyPort", "8080");
System.setProperty("http.proxySet", "true");
but it doesn't have any effect. Could anybody help?

It could be that an authentication is required, in this case please add
System.setProperty( "http.proxyUser", "user" );
System.setProperty( "http.proxyPassword", "password" );
or the hostname (proxy) isn't resolve this could be checked by ping hostname, you could add the IP-address instead. For further anaysis you should add the stacktrace of your exception.
HttpURLConnection has a method isUsingProxy() which could be useful for diagnostic log messages.
Further details you might find here

Nop, it does not work. I do not need to do authentication. I have found another solution. I have modified the freebase client -- the source code is freely available. The freebase client uses HttpClient4 , so you need to create HttpProxy object (and set your proxy details), and provide it to HttpClient.