Develop Reference

Java Playwright using connect with Proxy for browserless

I want to use Playwright.connect() method using Proxy to consume Browserless. According to Browserless doc.
https://docs.browserless.io/docs/playwright.html
The standard connect method uses playwright's built-in browser-server
to handle the connection. This, generally, is a faster and more
fully-featured method since it supports most of the playwright
parameters (such as using a proxy and more). However, since this
requires the usage of playwright in our implementation, things like
ad-blocking and stealth aren't supported. In order to utilize those,
you'll need to see our integration with connectOverCDP.
I thought well connect will have a .setProxy(), Like launch()
browserType.launch(new BrowserType.LaunchOptions().setProxy(proxy));
But connect methods it has 2 variations
default Browser connect(String wsEndpoint) {
return connect(wsEndpoint, null);
}
Browser connect(String wsEndpoint, ConnectOptions options);
I thought well i will pick connect + ConnectOptions it sures has a .setProxy as well but it doesn't.
class ConnectOptions {
public Map<String, String> headers;
public Double slowMo;
public Double timeout;
public ConnectOptions setHeaders(Map<String, String> headers) {
this.headers = headers;
return this;
}
public ConnectOptions setSlowMo(double slowMo) {
this.slowMo = slowMo;
return this;
}
public ConnectOptions setTimeout(double timeout) {
this.timeout = timeout;
return this;
}
}
I have try this
final Browser.NewContextOptions browserContextOptions = new Browser.NewContextOptions().setProxy(proxy);
Browser browser = playwright.chromium()
.connect("wss://&--proxy-server=http://myproxyserver:1111")
.newContext(browserContextOptions)
.browser();
browser.newPage("resource");
But the proxy returns authentication is required.
I'm confused now Browserless says that .connect could provide a Proxy but how? Is browserless wrong? Or am I missing something? I'm new on this technology.
I have tried as well using page.setExtraHTTPHeaders.
private void applyProxyToPage(final Page page,final String
userPassCombination){
final String value = "Basic "+Base64.getEncoder().encodeToString(userPassCombination.getBytes(Charset.forName("UTF-8")));
page.setExtraHTTPHeaders(Collections.singletonMap("Authorization",value));
//page.setExtraHTTPHeaders(Collections.singletonMap("Proxy-Authorization",value));// Not working either
}

With the help of my friend Alejandro Loyola at Browserless, I am now able to connect. I will post the snippet:
private String navigateWithPlaywrightInBrowserlessWithProxy(final String token,final String proxyHost,final String userName,final String userPass,final String url){
final Browser.NewContextOptions browserContextOptions = new Browser.NewContextOptions()
.setProxy(new Proxy(proxyHost)
.setUsername(userName)
.setPassword(userPass));//Raw password not encoded in any way;
try (final Playwright playwright = Playwright.create(); Browser browser = playwright.chromium().connectOverCDP("wss://chrome.browserless.io?token=" + token);final BrowserContext context = browser.newContext(browserContextOptions);){
Page page = context.newPage();
page.route("**/*.svg", Route::abort);
page.route("**/*.png", Route::abort);
page.route("**/*.jpg", Route::abort);
page.route("**/*.jpeg", Route::abort);
page.route("**/*.css", Route::abort);
page.route("**/*.scss", Route::abort);
page.navigate(url, new Page.NavigateOptions()
.setWaitUntil(WaitUntilState.DOMCONTENTLOADED));
return page.innerHTML("body");
}
}
My gotchas were as follows.
I was using:
"wss://chrome.browserless.io/playwright?token=
Instead of:
"wss://chrome.browserless.io?token="
And use:
connectOverCDP

Config-file for JavaFX-Project

I would like to add a file or class to my JavaFX project that only contains the configuration data of the project, e.g. the access data for the database, system paths etc. How would you do this?
Just write everything in a normal class? There is definitely a better way, right?

You're right, of course I'll be happy to do that.
First I created a property file in the project folder and call it app.properties:
db_url=jdbc:mysql://localhost:3306/db name
db_user=user name
db_pwd=secret password
instructions_folder=/home/username/documents/
Then I created a class that loads the properties and makes them available throughout the project.
public class AppProperties {
// FILENAME = Path to properties-file
// Store and protect it where ever you want
private final String FILENAME = "app.properties";
private static final AppProperties config_file = new AppProperties();
private Properties prop = new Properties();
private String msg = "";
private AppProperties(){
InputStream input = null;
try{
input = new FileInputStream(FILENAME);
// Load a properties
prop.load(input);
}catch(IOException ex){
msg = "Can't find/open property file";
ex.printStackTrace();
}finally{
if (input != null){
try{
input.close();
}catch(IOException e){
e.printStackTrace();
}
}
}
}
public String getProperty (String key){
return prop.getProperty(key);
}
public String getMsg () {
return msg;
}
// == Singleton design pattern == //
// Where ever you call this methode in application
// you always get the same and only instance (config_file)
public static AppProperties getInstance(){
return config_file;
}
}
In the DBUtilitis class, where I do my database queries, I now load the properties into final variables and use them in the query methods.
private static final String db_url = AppProperties.getInstance().getProperty("db_url");
private static final String db_user = AppProperties.getInstance().getProperty("db_user");
private static final String db_pwd = AppProperties.getInstance().getProperty("db_pwd");
If I have not completely misunderstood this, the advantage of property files is that they can be stored and protected somewhere on the server. I hope the solution is not completely wrong - it works well anyway. I am always happy to receive suggestions and / or improvements.

SMBJ and DFS and "Nested Session"

I have a project, where I am given an id, and then using that ID look up files paths and process them... these files are on various mounted drives, so I am using the SMBJ java libraries to access them.
The problem I am having is that some (most) of the files are using a DFS mountpoint... Now, this in and of itself is NOT a problem per se, but apparently the SMBJ libraries appear to create nested sessions for each distinct DFS location. So even though I am closing the actual FILE after I am done reading it the DiskSession object is holding onto all these nested sessions... and eventually either through the DFS config settings, or through these libraries I am hitting some point where it just blows up and stops allowing more sessions to be created.
I am processing hundreds of thousands of records, and the "crash" appears to happen somewhere around 500ish records(session) being processed. I do not see anything obvious looking at the code to explicitly close these nested sessions.. in fact I see no external access to them at all externally from the DiskShare object.
Is there some sort of setting I am missing that maximizes the sessions that this is holding onto? Other than me managing some sort of my own counter around this, and closing and reopening sessions/connections I am at a loss how to handle this.
Does anyone know what I am missing here?
Code below:
public class Smb {
private static SMBClient client;
private static String[] DFSMounts = {"DFS1","dfs1"};
private static final Logger Log = LoggerFactory.getLogger(Smb.class);
private static HashMap<String,DiskShare> shares = new HashMap<>();
private static HashMap<String,Connection> connections = new HashMap<>();
private static HashMap<Connection,Session> sessions = new HashMap<>();
private synchronized static SMBClient getClient(){
if (client == null){
SmbConfig cfg = SmbConfig.builder().withDfsEnabled(true).build();
client = new SMBClient(cfg);
}
return client;
}
private synchronized static Connection getConnection(String realDomainName) throws IOException{
Log.info("DOMAIN NAME "+realDomainName);
Connection connection = (connections.get(realDomainName) == null) ? client.connect(realDomainName) : connections.get(realDomainName);
if(!connection.isConnected()) {
connection.close();
sessions.remove(connection);
connection = client.connect(realDomainName);
}
// connection = client.connect(realDomainName);
connections.put(realDomainName,connection);
return connection;
}
private synchronized static Session getSession(Connection connection,SMBClient client){
Session session = sessions.get(connection);
if(session==null) {
PropertiesCache props = PropertiesCache.getInstance();
String sambaUsername = props.getProperty("smb.user");
String sambaPass = props.getProperty("smb.password");
String sambaDomain = props.getProperty("smb.domain");
Log.info("CLIENT " + client);
session = (sessions.get(connection) != null) ? sessions.get(connection) : connection.authenticate(new AuthenticationContext(sambaUsername, sambaPass.toCharArray(), sambaDomain));
sessions.put(connection, session);
}
return session;
}
#SuppressWarnings("UnusedReturnValue")
public synchronized static DiskShare getShare(String domainName, String shareName) throws SmbException
{
DiskShare share = shares.get(domainName+"/"+shareName);
if((share!=null)&&(!share.isConnected())) share=null;
if(share == null){
try {
PropertiesCache props = PropertiesCache.getInstance();
String sambaUsername = props.getProperty("smb.user");
String sambaPass = props.getProperty("smb.password");
String sambaDomain = props.getProperty("smb.domain");
String dfsIP = props.getProperty("smb.sambaIP");
SMBClient client = getClient();
String realDomainName = (Arrays.stream(DFSMounts).anyMatch(domainName::equals)) ? dfsIP: domainName;
Connection connection = getConnection(realDomainName);
Session session = getSession(connection,client);
share = (DiskShare) session.connectShare(shareName);
shares.put(domainName+"/"+shareName,share);
}
catch (Exception e){
Log.info("EXCEPTION E "+e);
Log.info("EX "+e.getMessage());
throw new SmbException();
}
}
return(share);
}
public static String fixFilename(String filename){
String[] parts = filename.split("\\\\");
ArrayList<String> partsList = new ArrayList<>(Arrays.asList(parts));
partsList.remove(0);
partsList.remove(0);
partsList.remove(0);
partsList.remove(0);
return String.join("/",partsList);
}
public static File open(String filename) throws SmbException {
String[] parts = filename.split("\\\\");
String domainName = parts[2];
String shareName = parts[3];
DiskShare share = getShare(domainName,shareName);
Set<SMB2ShareAccess> s = new HashSet<>();
s.add(SMB2ShareAccess.ALL.iterator().next());
filename = fixFilename(filename);
return(share.openFile(filename, EnumSet.of(AccessMask.GENERIC_READ), null, s, SMB2CreateDisposition.FILE_OPEN, null));
}
}
And here is how the OPEN is being used (to show it is closing the file after use):
String filename = documents.get(0).getUNCPath();
try (File f = Smb.open(filename)){
Process the file code...
f.closeSilently();
}
And:
while(i.hasNext()){
String filename = (String)i.next();
Log.info("FILENAME "+filename);
try(File f = Smb.open(filename)){
Process the file stuff here
}
}

I have created a PR for SMBJ which changes this. It will reuse the nested session for same host. I have successfully used it myself to avoid the exact same problem you are having. https://github.com/hierynomus/smbj/pull/489

How to use Authentication with JIRA REST API in Java

hi im creating a simple tool using java to create,update and delete issues(tickets) in jira. i am using rest api following code is im using to authenticate jira and issue tickets.
public class JiraConnection {
public static URI jiraServerUri = URI.create("http://localhost:8090/jira/rest/api/2/issue/HSP-1/");
public static void main(String args[]) throws IOException {
final AsynchronousJiraRestClientFactory factory = new AsynchronousJiraRestClientFactory();
final JiraRestClient restClient = factory.createWithBasicHttpAuthentication(jiraServerUri,"vinuvish92#gmail.com","vinu1994");
System.out.println("Sending issue creation requests...");
try {
final List<Promise<BasicIssue>> promises = Lists.newArrayList();
final IssueRestClient issueClient = restClient.getIssueClient();
System.out.println("Sending issue creation requests...");
for (int i = 0; i < 100; i++) {
final String summary = "NewIssue#" + i;
final IssueInput newIssue = new IssueInputBuilder("TST", 1L, summary).build();
System.out.println("\tCreating: " + summary);
promises.add(issueClient.createIssue(newIssue));
}
System.out.println("Collecting responses...");
final Iterable<BasicIssue> createdIssues = transform(promises, new Function<Promise<BasicIssue>, BasicIssue>() {
#Override
public BasicIssue apply(Promise<BasicIssue> promise) {
return promise.claim();
}
});
System.out.println("Created issues:\n" + Joiner.on("\n").join(createdIssues));
} finally {
restClient.close();
}
}
}
according this code i couldn't connect to the jira
**following exception i am getting **
please suggest me best solution to do my task

It seems to me that your error is clearly related to url parameter. The incriminated line and the fact that the error message is about not finding the resource are good indications of it.
You don't need to input the whole endpoint since you are using the JiraRestClient. Depending on the method that you call it will resolve the endpoint. Here is an example that works: as you can see I only input the base url

Connecting to Microsoft Dynamics CRM on-premise web service with Java?

Are there any online resources which show the basic steps to access the Microsoft CRM on-premise web service with a client written in Java?
Which web service toolkit should I use?
I tried it with JAXB but there is a conflict in the WSDL element naming which requires a class customization. If I find the correct binding fix, I will post it here.

The Microsoft Dynamics CRM application on premise version uses Active Directory authentication.
Although I never tried referencing the Microsoft Dynamics CRM web services from Java, I am sure it is feasible, as these are standard web services and therefor can be referenced from Java via SOAP, just like any other web service.
public class TestCRM {
private static String endpointURL = "http://server:port/MSCrmServices/2007/CrmService.asmx";
private static String userName = "username";
private static String password = "password";
private static String host = "server";
private static int portport = port;
//To make sure you are using the correct domain open ie and try to reach the service. The same domain you entered there is needed here
private static String domain = "DOMAIN";
private static String orgName = "THIS_IS_REQUIRED"; //this does the work....
public static void main(String[] args) {
CrmServiceStub stub;
try {
stub = new CrmServiceStub(endpointURL);
setOptions(stub._getServiceClient().getOptions());
RetrieveMultipleDocument rmd = RetrieveMultipleDocument.Factory.newInstance();
RetrieveMultiple rm = RetrieveMultiple.Factory.newInstance();
QueryExpression query = QueryExpression.Factory.newInstance();
query.setColumnSet(AllColumns.Factory.newInstance());
query.setEntityName(EntityName.######.toString());
//query.setFilter...
rm.setQuery(query);
rmd.setRetrieveMultiple(rm);
//Now this is required. Without it all i got was 401s errors
CrmAuthenticationTokenDocument catd = CrmAuthenticationTokenDocument.Factory.newInstance();
CrmAuthenticationToken token = CrmAuthenticationToken.Factory.newInstance();
token.setAuthenticationType(0);
token.setOrganizationName(orgName);
catd.setCrmAuthenticationToken(token);
boolean fetchNext = true;
while(fetchNext){
RetrieveMultipleResponseDocument rmrd = stub.RetrieveMultiple(rmd, catd, null, null);
RetrieveMultipleResponse rmr = rmrd.getRetrieveMultipleResponse();
BusinessEntityCollection bec = rmr.getRetrieveMultipleResult();
String pagingCookie = bec.getPagingCookie();
fetchNext = bec.getMoreRecords();
ArrayOfBusinessEntity aobe = bec.getBusinessEntities();
BusinessEntity[] myEntitiesAtLast = aobe.getBusinessEntityArray();
for(int i=0; i<myEntitiesAtLast.length; i++){
//cast to whatever you asked for...
### myEntity = (###) myEntitiesAtLast[i];
}
}
}
catch (Exception e) {
e.printStackTrace();
}
}
private static void setOptions(Options options){
HttpTransportProperties.Authenticator auth = new HttpTransportProperties.Authenticator();
List authSchemes = new ArrayList();
authSchemes.add(HttpTransportProperties.Authenticator.NTLM);
auth.setAuthSchemes(authSchemes);
auth.setUsername(userName);
auth.setPassword(password);
auth.setHost(host);
auth.setPort(port);
auth.setDomain(domain);
auth.setPreemptiveAuthentication(false); //it doesnt matter...
options.setProperty(HTTPConstants.AUTHENTICATE, auth);
options.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, "true"); //i think this is good.. not required though
}

Java -> SOAP -> MS CRM 2011 Online : http://zsvoboda.blogspot.com/2011/03/connecting-to-microsoft-crm-2011-online.html

The stub has been created with the Apache Axis2 framework.

You can find resources here. You can even work with an example is available in Dynamics CRM SDK. As Manuel Freiholz said, you have to use Axis2.
https://msdn.microsoft.com/en-us/library/jj602979(v=crm.5).aspx
http://blogs.msdn.com/b/dynamics-coe/archive/2013/09/21/integrating-microsoft-dynamics-crm-2011-online-with-java-and-other-non-net-clients.aspx
Alternatively, you can use RESTFul web services through the OData interface offered by Dynamics (https://msdn.microsoft.com/en-us/library/gg334279.aspx)