I am attempting to use HTMLUnit (first time) to extract data from specific pages.
Specifically, I am currently trying to grab an HTML element by ID (a search box).
But I am running into:
Exception in thread "main" com.gargoylesoftware.htmlunit.ElementNotFoundException: elementName=[*] attributeName=[id] attributeValue=[space_search_keyword]
at com.gargoylesoftware.htmlunit.html.HtmlPage.getHtmlElementById(HtmlPage.java:1547)
at com.gargoylesoftware.htmlunit.html.HtmlPage.getHtmlElementById(HtmlPage.java:1517)
at Test.main(Test.java:33)
This is my code:
import java.util.List;
import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HTMLParserListener;
import com.gargoylesoftware.htmlunit.html.HtmlElement;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
public class Test {
public static void main(String[] args) {
HtmlPage page = null;
WebClient client = new WebClient();
client.setCssEnabled(false);
client.setJavaScriptEnabled(false);
try {
String searchUrl = "https://25live.collegenet.com/umassd/#space_search[0]";
page = client.getPage(searchUrl);
}catch(Exception e){
e.printStackTrace();
}
//System.out.println(page.asXml());
HtmlElement searchBox = (HtmlElement)page.getHtmlElementById("space_search_keyword");
}
}
Upon further inspection using the page.asXML(), it seems that the page isn't properly loading and that's why it can't find the item? I'm not sure why it isn't loading for HTMLUnit. There's no need to login, you can see the page come up for yourself by entering it in a browser.
Any help with debugging HTMLUnit issues like this would be greatly appreciated.
The site is a SPA (Single-Page Application) written in Angular.
You need JavaScript to run it.
Unfortunately the JavaScript capability of HtmlUnit is insufficient to run Angular, so your approach won't work.
You can try:
Reverse-engineer the page and fetch the underlying resource that the SPA is accessing
Try Selenium ChromeDriver (it actually opens Chrome and simulates button clicks on the page)
Related
I am using headless PhantomJS browser to automate the application using phantomjs driver with selenium. (selenium java version 3.5.2 and phantomjs.exe)
I have a scenario where i will fill the form and submit it and then the browser gets closed and after closing the browser I am reusing the driver reference to get the URL. It works well when I am using the firefox driver with selenium 2.47.0.
Now I switched to the selenium phontamjsdriver and phantombrowser. Here when I make a call to the driver.get(url);after the browser gets closed it is throwing the nosuchwindowexception saying window is closed or inactive. But, the same code is working with the firefox driver
example:
driver.get(url);// first time works
submitForm(driver);//browser window gets closed.
driver.get(url);
The last line is throwing exception as:
nosuchwindowexception(selenium java with 3.5.2 version and phantomjs.exe).
But works well with the firefoxbrowser with selenium 2.4.7.
First of all, as you migrated from Selenium v2.47.0 to Selenium v3.5.2 it's worth to mention a lot have been changed with the availability of Selenium 3.x. Now Selenium-WebDriver is a W3C Recommendation Candidate and is compliant with WebDriver W3C Editor's Draft
NoSuchWindowException
NoSuchWindowException class extends NotFoundException and is majorly thrown while attempting:
WebDriver.switchTo().window(String windowName);
Now, a bit more details about your usecase, relevant HTML and your code block would have given us some more ideas what is going wrong. Perhaps the defination of submitForm(driver) holds the key to the solution of your question.
Best Practices
Here some of the best practices you can follow to avoid NoSuchWindowException:
Always strore the Parent Window Handle in a variable so that you can traverse back to the Parent Window as generally required.
Before invoking driver.switchTo().window(windowHandle); always induce WebDriverwait in-conjunction with ExpectedConditions method numberOfWindowsToBe(int).
Once you invoke driver.switchTo().window(windowHandle); induce WebDriverWait in-conjunction with ExpectedConditions method titleContains(java.lang.String title) to wait for the Page Loading to get completed to continue your Test Steps on the newly opened window.
To switch back to the parent window use the previously stored windowhandle.
Here is a sample code block to demonstrate Window/Tab handling:
package demo;
import java.util.Set;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class WINDOW_HANDLE_ITERATE_Firefox
{
public static void main(String[] args) throws Exception
{
System.setProperty("webdriver.gecko.driver", "C:\\Utility\\BrowserDrivers\\geckodriver.exe");
WebDriver driver = new FirefoxDriver();
driver.get("http://www.google.com");
String parent_window = driver.getWindowHandle();
System.out.println("Parent Window Handle is: "+driver.getWindowHandle());
System.out.println("Page Title is: "+driver.getTitle());
((JavascriptExecutor) driver).executeScript("window.open('http://facebook.com/');");
new WebDriverWait(driver,10).until(ExpectedConditions.numberOfWindowsToBe(2));
Set<String> allWindows_1 = driver.getWindowHandles();
System.out.println("Total Windows: "+allWindows_1.size());
for(String hand1:allWindows_1)
if(!parent_window.equals(hand1))
{
driver.switchTo().window(hand1);
new WebDriverWait(driver,10).until(ExpectedConditions.titleContains("Facebook"));
String first_child_window = driver.getWindowHandle();
System.out.println("First Child Window Handle is: "+first_child_window);
System.out.println("First Child Window Page Title is: "+driver.getTitle());
driver.close();
}
driver.switchTo().window(parent_window);
System.out.println("Current Window Handle is : "+driver.getWindowHandle()+ " which is same as "+parent_window +", which is the parent window handle" );
driver.quit();
}
}
Console Output:
1531917836983 geckodriver INFO geckodriver 0.20.1
1531917836993 geckodriver INFO Listening on 127.0.0.1:9993
1531917837915 mozrunner::runner INFO Running command: "C:\\Program Files\\Mozilla Firefox\\firefox.exe" "-marionette" "-profile" "C:\\Users\\ATECHM~1\\AppData\\Local\\Temp\\rust_mozprofile.W5WqVulBNm9x"
1531917842220 Marionette INFO Listening on port 35364
1531917843126 Marionette WARN TLS certificate errors will be ignored for this session
Jul 18, 2018 6:14:03 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Parent Window Handle is: 4294967297
Page Title is: Google
Total Windows: 2
First Child Window Handle is: 4294967301
First Child Window Page Title is: Facebook – log in or sign up
Current Window Handle is : 4294967297 which is same as 4294967297, which is the parent window handle
This may be an issue to bring to the PhantomJS team here https://github.com/ariya/phantomjs/issues
Unfortunately screenshots in Selenium only capture the DOM and not address bar. If you're able to save the pagesource, you may be able to extract the URL. When viewing page source of this page I see tags which list various 3rd party apps, inside there is a content source which lists the URL.
<meta name="twitter:app:url:googleplay" content="http://stackoverflow.com/questions/51255939/nosuchwindowexception-selenium-with-phantomjs-java">
This may not be true for every site, but could be somewhere to look. You could also try and add this tag in yourself if you're the site owner.
After my problem was solved yesterday (see link below), new problems have opened up.
First, the solution to the above problem works only for the IE. When I try chrome or firefox with the same code to feed (except, of course, the driver) I get back the error message as at the beginning (before I downgraded the IEDriverServer). see link.
Eclipse Java Selenium failed to connect to localhost
Secondly. does not happen when I use the findelement function. Error message "unable to find e
lement". however, this element is there. here the page that should be controlled.
https://accounts.google.com/signup/v2/webcreateaccount?continue=https%3A%2F%2Fwww.google.de%2F%3Fgws_rd%3Dssl&hl=de&flowName=GlifWebSignIn&flowEntry=SignUp
this problem is already frequently requested but unfortunately the answers do not help me. one solution was (as the findelement is faster than the browser) to install the following delay. driver.manage (). timeouts (). implicitlyWait (5, TimeUnit.SECONDS);
another solution was to search for id or xpath instead of name (that's why name and id are listed below as findelement). so far without success. none of the mentioned keys is entered in a field (password and first name, etc., I have already tried).
Here are the articles that come closest to my problem but have not helped me.
No such element exception | Internet explorer.
Same CSS Slector Not Working in Internet Explorer but works in firefox
anyway. the IE will open, but no other browser and no element will be filled. here code and error message.
import org.openqa.selenium.firefox.FirefoxBinary;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxProfile;
import org.openqa.selenium.ie.InternetExplorerDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.openqa.grid.selenium.*;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import java.awt.Desktop;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.concurrent.TimeUnit;
public class firefoxöffner {
public static void main(String[] args) throws IOException, URISyntaxException {
System.setProperty("webdriver.gecko.driver","./geckodriver");
String service = "D://IEDriverServer.exe";
System.setProperty("webdriver.ie.driver", service);
InternetExplorerDriver driver = new InternetExplorerDriver();
driver.get("https://accounts.google.com/signup/v2/webcreateaccount?flowName=GlifWebSignIn&flowEntry=SignUp");
driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);
WebElement myDynamicElement = driver.findElement(By.id("lastName"));
driver.findElement(By.id("lastName")).sendKeys("B");
driver.findElement(By.name("lastName")).sendKeys("A");
}
}
Started InternetExplorerDriver server (64-bit)
3.8.0.0
Listening on port 47620
Only local connections are allowed
Mai 06, 2018 11:29:28 NACHM. org.openqa.selenium.remote.ProtocolHandshake createSession
INFORMATION: Detected dialect: W3C
Exception in thread "main" org.openqa.selenium.NoSuchElementException: Unable to find element with css selector == #lastName
Thanks again.
If you do Web Interface tests with Java + Selenium, I advise you to use the NoraUi Open Source Framework.
This Framework manage IE webdriver, Chrome webdriver and FF webdriver.
When ran on IE Probably //*[#id="lastName"] doesn't quite give you the exact element you want to work with. Instead when Can you try giving Xpath like below
//*[#id="lastName"]//following-sibling::div"
I am using a Google Cloud SQL using Java-SQL connector. The issue I am facing is that the connection to database drops unexpectedly. While Googling I came across this question and tried the solution suggested in the same question.
In your console click the project, on the left side click Storage > CloudSQL then click on your database name. You will see an 'Edit' button on top. Click that and scroll down to Activation Policy, change it to Always On and then click save.
But I'm still facing the same issue. Fortunately I have been keeping the logs on Google App Engine and I have attached the snapshot of the exception that occurred while connecting to database.
Gist of the code that I've posted below is used to establish connection to the database.
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.StringTokenizer;
import java.util.logging.Logger;
import com.google.appengine.api.utils.SystemProperty;
import static com.google.appengine.api.utils.SystemProperty.environment;
import static com.google.appengine.api.utils.SystemProperty.Environment.Value.Development;
import static com.google.appengine.api.utils.SystemProperty.Environment.Value.Production;
Connection con=null;
SystemProperty.Environment.Value env = environment.value();
if(env == Production)
{
System.out.println("Inside Production Phase");
// Load the class that provides the new "jdbc:google:mysql://" prefix.
Class.forName("com.mysql.jdbc.GoogleDriver");
url = "jdbc:google:mysql://<my-project-id>:<cloud-sql-instance>/<database-name>?user=<user-name>&password=<database-password>&useUnicode=true&characterEncoding=UTF-8";
}//if
else if(env == Development)
{
System.out.println("Inside Development Phase");
// This will load the MySQL driver, each DB has its own driver
Class.forName("com.mysql.jdbc.Driver");
url = "jdbc:mysql://127.0.0.1:3306/<database-name>?user=root";
}//else if
con = DriverManager.getConnection(url);
Is anyone facing the same problem, Please help.
Got a temporary fix, used following parameters while making connection to Google Cloud SQL
url = "jdbc:google:mysql://my-app:mysql2/project-name?user=root&password=password&autoReconnect=true&failOverReadOnly=false&maxReconnects=10";
Reference URL: https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html
I have started to work with spring social and following the tutorial from here. and pages that follow.
My java file looks like this.
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
//import org.springframework.boot.SpringApplication;
//import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
import org.springframework.social.connect.Connection;
import org.springframework.social.connect.ConnectionFactory;
import org.springframework.social.connect.ConnectionFactoryLocator;
import org.springframework.social.connect.support.ConnectionFactoryRegistry;
import org.springframework.social.facebook.api.Comment;
import org.springframework.social.facebook.api.CommentOperations;
import org.springframework.social.facebook.connect.FacebookConnectionFactory;
import org.springframework.social.facebook.api.Facebook;
import org.springframework.social.facebook.api.impl.FacebookTemplate;
import org.springframework.social.oauth2.AccessGrant;
import org.springframework.social.oauth2.GrantType;
import org.springframework.social.oauth2.OAuth2Operations;
import org.springframework.social.oauth2.OAuth2Parameters;
//import org.springframework.social.UserIdSource;
//import org.springframework.social.connect.ConnectionFactoryLocator;
//import org.springframework.social.connect.ConnectionRepository;
//import org.springframework.social.connect.web.ConnectController;
#Configuration
#EnableAutoConfiguration
#Import(FacebookConfig.class)
#ComponentScan
public class App {
static private String accessToken = "accesstoken";
static private String secretKey = "secretkey";
static private String clientId = "clientid";
public static void main(String[] args) {
FacebookConnectionFactory connectionFactory = new FacebookConnectionFactory(clientId, secretKey);
OAuth2Operations oauthOperations = connectionFactory.getOAuthOperations();
OAuth2Parameters params = new OAuth2Parameters();
params.setRedirectUri("http://facebook.com");
String authorizeUrl = oauthOperations.buildAuthorizeUrl(GrantType.IMPLICIT_GRANT, params);
AccessGrant accessGrant = new AccessGrant(accessToken);
System.out.println(accessGrant.getAccessToken());
System.out.println(accessGrant.getExpireTime());
System.out.println(accessGrant.getScope());
ConnectionFactoryRegistry registry = new ConnectionFactoryRegistry();
registry.addConnectionFactory(connectionFactory);
Facebook facebook = new FacebookTemplate(accessToken);
}
}
When i run this code i get the error as stated.
Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (default-cli) on project mavenproject3: Command execution failed. Process exited with an error: 1(Exit value: 1) -> [Help 1]
To see the full stack trace of the errors, re-run Maven with the -e switch.
Re-run Maven using the -X switch to enable full debug logging.
For more information about the errors and possible solutions, please read the following articles:
[Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
When i comment out this line:
Facebook facebook = new FacebookTemplate(accessToken);
It works fine.
Can someone suggest me the requisite. I am a newbie so please bear with me.
The URL you mentioned isn't really a tutorial as much as it is a reference. Admittedly, it gets a bit detailed and isn't very helpful for the new user. Duly noted...expect a new tutorial to be written as soon as I get a moment to do so.
Where did you get the value of accessToken? If you didn't get it via an OAuth2 "dance" with Facebook, then it's not going to work.
First, I see you creating a FacebookConnectionFactory to obtain an OAuth2Operations, through which you set a redirect URI, etc, etc..and then build an authorization URL for IMPLICIT grant. There are several things out of sorts there:
Facebook doesn't support IMPLICIT grant. It only supports authorization code grant and client token grant. Even so, with implicit grant and authorization code grant your app must redirect to Facebook (in a web browser) to obtain permission from the user. Once that's granted, then it will redirect back to your app...speaking of which...
The redirect URI you set is http://facebook.com. That should be the URL of your application where Facebook will redirect back to after authorization.
After all of that, you never even use the authorizeUrl...it's just in a String. It wouldn't work even if you did use it, for the reasons already mentioned, but the first 5 or so lines are all for nothing.
You create a ConnectionFactoryRegistry and register the FacebookConnectionFactory with it...but then you do nothing with the ConnectionFactoryRegistry. That's okay...you almost never need to do anything with it anyway, because it primarily exists as a helper to ConnectController.
There's simply no good way of obtaining a user-oriented access token without the redirect "dance". It's important to get permission from the user you'll be accessing Facebook on behalf of. If it were any easier than that, it'd be way too easy to create an app that spams Facebook and essentially ruins the experience for everyone.
The work of obtaining an access token via that redirect "dance" is handled automatically by the framework using ConnectController. Sure, you can do it yourself if you'd rather, but ConnectController will handle all of that for you.
For lack of a proper tutorial at the moment, I recommend that you have a look at https://github.com/spring-projects/spring-social-samples/tree/master/spring-social-showcase. Also, there's a Spring Boot-oriented version of it at https://github.com/spring-projects/spring-social-samples/tree/master/spring-social-showcase-boot that simplifies the configuration more (albeit, it relies on changes that aren't in an official Spring Boot release yet).
Anyone know any? I need to send in a http request and make sure the http response i got back is not http 500
I believe Hyperic HQ meets all of your criteria. It is open source, I believe it is written at least partially in Java, and it is designed to do all kinds of server monitoring.
It should be able to handle not only the kind of monitoring you requested but other necessary monitoring like memory, CPU usage, and disk space on your servers as well.
You could use httpunit - web-centric unit testing
While you find it you can use this:
import java.net.HttpURLConnection;
import java.net.URL;
import java.io.IOException;
public class SimplisticMonitor {
public static void main( String [] args ) throws IOException {
HttpURLConnection c = ( HttpURLConnection )
new URL( "http://stackoverflow.com" ).openConnection();
System.out.println( c.getResponseCode() );
}
}
If you want to do this yourself, Apache HttpClient is an option:
GetMethod get = new GetMethod("http://www.stackoverflow.com");
try
{
int resultCode = client.executeMethod(get);
if (resultCode == 500)
{
//do something meaningful here
} // if
} // try
catch (Exception e)
{
e.printStackTrace();
}
finally
{
get.releaseConnection();
}
http-unit
Written in Java, HttpUnit emulates the relevant portions of browser behavior, including form submission, JavaScript, basic http authentication, cookies and automatic page redirection, and allows Java test code to examine returned pages either as text, an XML DOM, or containers of forms, tables, and links. When combined with a framework such as JUnit, it is fairly easy to write tests that very quickly verify the functioning of a web site.
or html-unit
HtmlUnit is a "GUI-Less browser for Java programs". It models HTML documents and provides an API that allows you to invoke pages, fill out forms, click links, etc... just like you do in your "normal" browser.
It has fairly good JavaScript support (which is constantly improving) and is able to work even with quite complex AJAX libraries, simulating either Firefox or Internet Explorer depending on the configuration you want to use.