Java Playwright using connect with Proxy for browserless - java

I want to use Playwright.connect() method using Proxy to consume Browserless. According to Browserless doc.
https://docs.browserless.io/docs/playwright.html
The standard connect method uses playwright's built-in browser-server
to handle the connection. This, generally, is a faster and more
fully-featured method since it supports most of the playwright
parameters (such as using a proxy and more). However, since this
requires the usage of playwright in our implementation, things like
ad-blocking and stealth aren't supported. In order to utilize those,
you'll need to see our integration with connectOverCDP.
I thought well connect will have a .setProxy(), Like launch()
browserType.launch(new BrowserType.LaunchOptions().setProxy(proxy));
But connect methods it has 2 variations
default Browser connect(String wsEndpoint) {
return connect(wsEndpoint, null);
}
Browser connect(String wsEndpoint, ConnectOptions options);
I thought well i will pick connect + ConnectOptions it sures has a .setProxy as well but it doesn't.
class ConnectOptions {
public Map<String, String> headers;
public Double slowMo;
public Double timeout;
public ConnectOptions setHeaders(Map<String, String> headers) {
this.headers = headers;
return this;
}
public ConnectOptions setSlowMo(double slowMo) {
this.slowMo = slowMo;
return this;
}
public ConnectOptions setTimeout(double timeout) {
this.timeout = timeout;
return this;
}
}
I have try this
final Browser.NewContextOptions browserContextOptions = new Browser.NewContextOptions().setProxy(proxy);
Browser browser = playwright.chromium()
.connect("wss://&--proxy-server=http://myproxyserver:1111")
.newContext(browserContextOptions)
.browser();
browser.newPage("resource");
But the proxy returns authentication is required.
I'm confused now Browserless says that .connect could provide a Proxy but how? Is browserless wrong? Or am I missing something? I'm new on this technology.
I have tried as well using page.setExtraHTTPHeaders.
private void applyProxyToPage(final Page page,final String
userPassCombination){
final String value = "Basic "+Base64.getEncoder().encodeToString(userPassCombination.getBytes(Charset.forName("UTF-8")));
page.setExtraHTTPHeaders(Collections.singletonMap("Authorization",value));
//page.setExtraHTTPHeaders(Collections.singletonMap("Proxy-Authorization",value));// Not working either
}

With the help of my friend Alejandro Loyola at Browserless, I am now able to connect. I will post the snippet:
private String navigateWithPlaywrightInBrowserlessWithProxy(final String token,final String proxyHost,final String userName,final String userPass,final String url){
final Browser.NewContextOptions browserContextOptions = new Browser.NewContextOptions()
.setProxy(new Proxy(proxyHost)
.setUsername(userName)
.setPassword(userPass));//Raw password not encoded in any way;
try (final Playwright playwright = Playwright.create(); Browser browser = playwright.chromium().connectOverCDP("wss://chrome.browserless.io?token=" + token);final BrowserContext context = browser.newContext(browserContextOptions);){
Page page = context.newPage();
page.route("**/*.svg", Route::abort);
page.route("**/*.png", Route::abort);
page.route("**/*.jpg", Route::abort);
page.route("**/*.jpeg", Route::abort);
page.route("**/*.css", Route::abort);
page.route("**/*.scss", Route::abort);
page.navigate(url, new Page.NavigateOptions()
.setWaitUntil(WaitUntilState.DOMCONTENTLOADED));
return page.innerHTML("body");
}
}
My gotchas were as follows.
I was using:
"wss://chrome.browserless.io/playwright?token=
Instead of:
"wss://chrome.browserless.io?token="
And use:
connectOverCDP

Related

Filtering a Flux based upon the contents of the Flux, without blocking

Requirement: given the list of Servers in a Flux, return the LOCAL server if it is present, OR return the DEV servers if the LOCAL server is not present, and must be done in a non-blocking way.
I don't know how to do that with the Flux.filter method since it only allows you to examine one element at a time, and to perform that filtering you have to know what is in the whole list ahead of time. This was easy with blocking code as I simply iterated over the entire list ahead of time to determine if it had LOCAL or DEV instances, etc. Not sure how to do this with a Flux though.
Server Class:
class Server {
public String environment;
public String hostName;
public Server(String environment, String hostName) {
this.environment = environment;
this.hostName = hostName;
}
}
Filter Method:
public Flux<Server> filterServersLocalOrDev(Flux<Server> serverFlux) {
Flux<Server> filteredServers = // do filtering here..how to?
return filteredServers;
}
Demo calling code:
Server server1 = new Server("DEV", "dev1.somecompany.com");
Server server2 = new Server("TEST", "test1.somecompany.com");
Server server3 = new Server("DEV", "dev2.somecompany.com");
Server server4 = new Server("LOCAL", "localhost");
Flux<Server> serverFlux = Flux.just(server1, server2, server3, server4);
Flux<Server> filteredServerFlux = filterServersLocalOrDev(serverFlux);
// filteredServerFlux should only contain LOCAL if it is
// present OR the two DEV servers if there is no LOCAL
You can use filter and switchIfEmpty. It will return Local servers if present otherwise returns dev servers like this
public Flux<Server> filterServersLocalOrDev(Flux<Server> serverFlux) {
Flux<Server> filteredServers = serverFlux
.filter(s -> "LOCAL".equals(s.environment))
.switchIfEmpty(serverFlux.filter(s -> "DEV".equals(s.environment)));
return filteredServers;
}

How to set up default browser that jdk/jre uses

I used code of programm that you can see below.The logic of class is to get some properties from html code from YouTube page.For long time it worked fine, but now not. The reason of problem is the next: jdk/jre uses Internet explorer as default browser and now YouTube not support ie (It returns the page with suggestion of updating browser).
The question is : how to change default browser taht java uses?
I switched the default browser of the system to Chrome and default browser of Intellij IDE to Chrome too, but it didn't give any result to me.
#Component(immediate = true, service = LastActualVideoService.class)
public class LastActualVideoServiceServiceImpl implements LastActualVideoService {
private final Logger logger = LoggerFactory.getLogger(getClass());
private static final String LINK_TO_YOU_TUBE = "https://www.youtube.com/embed/";
private static final String TRIGGER_FOR_VIDEO = "/watch?v=";
private static final String VIDEO_SELECTOR = "/videos";
private static final String HTML_SEPARATOR = "\\A";
private static final String ERROR_MASSAGE = "Incorrect input URL";
private static final String OPEN_TITLE_TAG = "<title>";
private static final String CLOSE_TITLE_TAG = "</title>";
#Override
public YouTubeChannelInfo getVideoBlob(String channelURL) {
channelURL = channelURL.concat(VIDEO_SELECTOR);
try (InputStream response = new URL(channelURL).openStream()) {
Scanner scanner = new Scanner(response);
String responseBody = scanner.useDelimiter(HTML_SEPARATOR).next();
String uniqueVideo = responseBody.substring(responseBody.indexOf(TRIGGER_FOR_VIDEO), responseBody.indexOf(TRIGGER_FOR_VIDEO) + 20);
String title = responseBody.substring(responseBody.indexOf(OPEN_TITLE_TAG) + 7, responseBody.indexOf(CLOSE_TITLE_TAG));
String linkToVideo = LINK_TO_YOU_TUBE.concat(uniqueVideo.substring(uniqueVideo.lastIndexOf('=') + 1));
return new YouTubeChannelInfo(linkToVideo, title, channelURL);
} catch (IOException e) {
logger.error(ERROR_MASSAGE, e);
return null;
}
}
}
URL.openStream does not "use the browser", your Java program acts as HTTP client itself. The way the remote server can know what type of browser is connecting is the user agent that the client sends with the request. It's possible that Youtube does not recognize or like whatever the default is.
Like Joachim Rohde commented, the solution is to manually set the user agent to something Youtube will recognize as supported.

Does Protocol.registerProtocol for apache HttpClient cause a global static change?

I've come across some code that's using
Protocol.registerProtocol
in an attempt to block some TLS ciphers for a request, and retry the request with it re-enabled at times, depending on other factors.
But is Protocol.registerProtocol causing a global change - ie will other threads be affected by this?
Here's the code in question:
protected static HostConfiguration buildTLSConfig(String uri, HostConfiguration config,
boolean blockTLS1)
throws MalformedURLException
{
scheme = "https";
if (baseHttps == null)
{
baseHttps = Protocol.getProtocol(scheme);
baseFactory = baseHttps.getSocketFactory();
}
URL newUrl = new URL(uri);
defaultPort = baseHttps.getDefaultPort();
if (blockTLS1)
{
ProtocolSocketFactory customFactory =
new CustomHttpsSocketFactory(baseFactory, TLS_PREFERRED_PROTOCOLS);
Protocol applyHttps = new Protocol(scheme, customFactory, defaultPort);
Protocol.registerProtocol(scheme, applyHttps);
config.setHost(newUrl.getHost(), defaultPort, applyHttps);
}
else
{
Protocol.registerProtocol(scheme, baseHttps);
config.setHost(newUrl.getHost(), defaultPort, baseHttps);
}
return config;
}
Yes, all threads will be affected by the change.
If we look at org.apache.commons.httpclient.protocol.Protocol, we see a global protocol Map:
/** The available protocols */
private static final Map PROTOCOLS = Collections.synchronizedMap(new HashMap());
And registerProtocol() simply modifying it:
public static void registerProtocol(String id, Protocol protocol) {
// . . .
PROTOCOLS.put(id, protocol);
}
At least it's synchronized, so there won't be a race during modification.

How can I hijack an opened browser session and attach it to the Webdriver than opening a browser by itself

When using Selenium webDriver, I always have to initiate the driver and open a browser, navigate all the way through the flow to validate something, and if it failed throwing an exception, has to do the same spending lot of time.
Always thought of keeping another browser session opened, and let webdriver attached to the opened session and work on it, so the need to go through the flow could be eliminated.
Does any one knows how to do this in Selenium Webdriver, using Java?
Thanks
I've tried this. It works for me.
public class xRemoteWebDriver : RemoteWebDriver
{
public static bool newSession;
public static string capPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "TestFiles", "tmp", "sessionCap");
public static string sessiodIdPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "TestFiles", "tmp", "sessionid");
public xRemoteWebDriver(Uri remoteAddress)
: base(remoteAddress, new DesiredCapabilities())
{
}
protected override Response Execute(DriverCommand driverCommandToExecute, Dictionary<string, object> parameters)
{
if (driverCommandToExecute == DriverCommand.NewSession)
{
if (!newSession)
{
var capText = File.ReadAllText(capPath);
var sidText = File.ReadAllText(sessiodIdPath);
var cap = JsonConvert.DeserializeObject<Dictionary<string, object>>(capText);
return new Response
{
SessionId = sidText,
Value = cap
};
}
else
{
var response = base.Execute(driverCommandToExecute, parameters);
var dictionary = (Dictionary<string, object>) response.Value;
File.WriteAllText(capPath, JsonConvert.SerializeObject(dictionary));
File.WriteAllText(sessiodIdPath, response.SessionId);
return response;
}
}
else
{
var response = base.Execute(driverCommandToExecute, parameters);
return response;
}
}
}
Hope it helps.

string escaping with non-nullable named parameters

I am trying to use the google cloud endpoints Java from android as such:
client:
Core.Builder coreBuilder = new Core.Builder(
AndroidHttp.newCompatibleTransport(), new GsonFactory(), null);
coreBuilder.setApplicationName("myapp");
if (MainActivity.ENDPOINTS_URL != null) {
coreBuilder.setRootUrl(MainActivity.ENDPOINTS_URL);
coreBuilder.setGoogleClientRequestInitializer(new GoogleClientRequestInitializer() {
public void initialize(AbstractGoogleClientRequest<?> request)
throws IOException {
request.setDisableGZipContent(true);
}
});
}
Core core = coreBuilder.build();
myList = core.asdf("x=&+x", myObject);
server:
#ApiMethod(name = "asdf")
public List<String> asdf(#Named("param1") String param1, MyObject myObject) {
if (param1.equals("x=&+x")) {
//should go here, but never does
}
...
While it mostly works, somehow the param1 string does not get correctly transmitted, meaning that "x=&+x" arrives at the server as "x=&%2Bx". Is this a known bug? Or do arguments have to be manually encoded somehow? Or is this somehow particular to my environment?
Appengine SDK V1.8.8 for java, google api 1.17.0-rc, using the dev environment.
Cheers,
Andres

Categories

Resources