I am trying to check whether a page contains next page to sroll. The idea is basic: get the current link, remove the index at the end and then put new index, try to connect to the new link, if it throws IOException that means it does not have next button so that can be scrolled. Here is my code, and the problem it returns false even to the links that Jsoup can connect:
public boolean checkNextButton(String linkToCheck) throws IOException {
boolean containsNextButton = true;
try {
//Here I am trying to connect to a link which is actually available but it still returns false
Document doc = Jsoup.connect(turnToNextPage(linkToCheck)).get();
specific.setPageChangeRate(specific.getPageChangeRate() - 1);
//Set back the counter to the previous one
} catch (IOException e) {
containsNextButton = false;
}
System.out.println("The page contains next button: " + containsNextButton);
return containsNextButton;
}
Solution provided by Pshemo:
It is hard to tell what exactly went wrong without more details.
Are you able to access resource from linkToCheck via browser? Did you need to log-in on that site first (maybe try accessing it via incognito mode to see if it is available for anyone without creating an account).
Other problem could be that some servers allow only specific browsers and they try to recognize them by using user-agent header browsers send, or if server is creating temporary session you may need to pass cookies from previous connection.
Related
I am trying to do the following thing using htmlunit:
Get on a page, and login.
AFTER a successful login, acquire a particular page so that I can work with its content.
Suppose that the server is located at mysite.mydomain.com.
First let me describe what happens using an actual browser, e.g. Chrome.
I type mysite.mydomain.com on the address bar.
I am arriving at
mysite.mydomain.com/blahb/blahb.exe/index?SOMETHING=0&SOMETHINGELSE=1
There I can fill in my info for logging in, i.e. I can enter my username and password and press the submit button.
Upon successful login, I end up looking at the following page:
mysite.mydomain.com/blahb/blahinfo.exe/index (blahb.exe became blahinfo.exe)
There I have options. Say, a menu. Clicking on the particular option I am interested in, will bring up a button in the main window frame. Let's call this button "SHOW".
Clicking SHOW, the URL remains unchanged but the screen is split in two frames. TOP and BOTTOM. The TOP frame is the one that contains the button SHOW that we just pressed as well as two new buttons that just appeared: one to PRINT something and one to save that something as a PDF.
At the BOTTOM frame is the content that one could print or save, and which I am interested in.
A simple inspection of the page's code shows that the content I am interested in and which is depicted at the bottom frame was taken from:
mysite.mydomain.com/blahb/blahinfo.exe/somethingBody
Indeed, If I simply login to the site, and instead of pressing any menu buttons after that, I simply visit:
mysite.mydomain.com/blahb/blahinfo.exe/somethingBody
Will result in me getting the content I need on my screen, in a single frame.
If I am logged in, I can open another tab, paste
mysite.mydomain.com/blahb/blahinfo.exe/somethingBody
And I will get the content.
So far pretty simple.
Now let's see what happens when I use htmlunit.
I can do the following:
private static HtmlPage loginAndGetPageOfInterest(WebClient webClient) throws Exception
{
//enable whatever I want to enable for the webClient
webClient.getOptions().setJavaScriptEnabled(true);
//deal with cookies
CookieManager cookieManager = new CookieManager(); //seems we need cookies
cookieManager = webClient.getCookieManager();
cookieManager.setCookiesEnabled(true);
String mainURL = "mysite.mydomain.com/blahb/blahb.exe/index?SOMETHING=0&SOMETHINGELSE=1";//want to go there
HtmlPage currentPage = webClient.getPage(mainURL) ;
//Fill the login info and press the button
HtmlTextInput nameInput = currentPage.getHtmlElementById("Id");//find the username field
nameInput.setValueAttribute("Iamtheuser"); //fill it in
HtmlInput passInput = currentPage.getHtmlElementById("Pass"); //likewise for password
passInput.setValueAttribute("andthisismypass");
HtmlButton submit = (HtmlButton) currentPage.getElementsById("login_button").get(0);//find the submit button
currentPage = submit.click();
Thread.sleep(2000); //wait some time to ensure that we have indeed logged in.
//if I were to print currentPage in a file NOW... ALL is well. I get exactly what I get in the browser. If I were to return the currentPage here, everything is fine.
String urlIActuallyWant = "mysite.mydomain.com/blahb/blahinfo.exe/somethingBody";
currentPage = webClient.getPage(urlIActuallyWant);//try to get what I actually want...
return currentPage;
}
Now I can do:
public static void main(String[] args) throws Exception
{
WebClient webClient = new WebClient(BrowserVersion.CHROME); //Could do without saying anything about client version
HtmlPage currentPage = loginAndGetPageOfInterest(webClient);
//save what we have so that we can look at it.
String pageSource = currentPage.asXml();
File file = new File("howInteresting.html");
try
{
Files.write(file.toPath(), pageSource.getBytes());
Desktop.getDesktop().browse(file.toURI());
}
catch (IOException e)
{ //end of the world }
}
However... when I attempt this, I am getting:
Server Error
500 - Internal server
error. There is a problem with the resource you are looking
for, and it cannot be displayed.
Exception in thread "main"
com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException: 500
Internal Server Error for
https://mysite.mydomain.com/blahb/blahinfo.exe/somethingBody at
com.gargoylesoftware.htmlunit.WebClient.throwFailingHttpStatusCodeExceptionIfNecessary(WebClient.java:595)
at
com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:410)
at
com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:317)
at
com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:469)
at
com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:450)
at TestWTH.loginAndGetPageOfInterest(TestWTH.java:46) at
TestWTH.main(TestWTH.java:54)
Now I understand that the server, is sending back a 500 Internal Server Error and I suspect that this is because it thinks that we are NOT logged in.
But why would that be?
The fact that if I log in and then go directly to the page of interest works fine on the browser but not when using htmlunit makes me suspect that it is an issue with cookie handling. But why would there be an issue with the cookies? Any ideas?
I've a webapp running on TOMCAT 7.0.5 - JDK 1.7 - and connecting to SQL Server 2012.
The hibernate version is 4.3.5. Browser used is IE 11.0.
The issue Is I have a webpage: A normal FORM submit (submit.jsp). It shows a readonly page (submitConfirm.jsp) and has two buttons.
Submit - Saves the application and redirects the page to the home page to start subscription again
Print - opens a new tab via javascript to print a PDF form that likely takes 2 to 5 seconds to render in a "new" tab.
The submit button is disabled by defualt and enabled only when user clicks on the PRINT Button. While the item is still rendering If someone clicks on Continue button - all actions perform well and at the I get both the print and subscriptions saved.
The next subscriptions entered would throw a http 500 error indicating the "Connection is already closed".
I use the Standard HibernateUtil paradigm to begin/end/commit/rollback transactions.
The models are all XML based (not entity decorator based).
For normal workflows where if the user waits till the PDF is rendered all is fine.
But If the user is pressing the button of continue - Seems there is an issue.
Attached is the screen-grab of the error is someone finds it useful provide some hints.
Based on the user feedback (the concerned method):
public String reportSessionId(String userId) throws InfrastructureException {
String sid = null;
try {
sid = (String) HibernateUtil.getSession()
.getNamedQuery("com.ttsme.ims.model.IMSUser.reportSessionId")
.setString("userId", userId).uniqueResult();
}
catch (HibernateException ex) {
throw new InfrastructureException(ex);
}
catch (Exception exp)
{
exp.printStackTrace();
}
return sid;
}
I am implementing FB in one of my app.I am using jar 0.8.25. Its working fine on all simulators from 5 to 7.1.And for devices works only for OS 5 and 6 but not working on device 7 and 7.1.For OS 7 after log in success it remains on FB page it doesn't redirect back. and when i press back button, i get error encountered unable to refresh access token with try again button.
When analyzing on console it never finds access token single time for OS 7.while for 5 and 6 its working perfectly.
Please tell what may cause the issue.
Thanks,
This isn't a solution to your specific problem. I mentioned in the comments that I'm using an interface. So I'm posting here as its too much for the comment section. It is also not the COMPLETE solution, you will need to handle the flow and expired tokens, this is just to show you the logic of how I did this.
For my interface I open a browserfield to the Oauth url:
https://www.facebook.com/dialog/oauth?client_id=<APP_ID>&response_type=token&redirect_uri=http://www.facebook.com/connect/login_success.html&scope=publish_actions
And I add a listener to this browser to listen for the redirects after login. Once you have the access token, you should persist it and close the browserfield.
private class OAuthScreen extends MainScreen
{
BrowserField browser_field;
LoadingDialog loading_dialog;
public OAuthScreen(final Command task)
{
super(VERTICAL_SCROLL | HORIZONTAL_SCROLL);
BrowserFieldConfig browserConfig = new BrowserFieldConfig();
browserConfig.setProperty(BrowserFieldConfig.VIEWPORT_WIDTH, new Integer(Display.getWidth()));
browser_field = new BrowserField(browserConfig);
browser_field.addListener(new BrowserFieldListener()
{
public void documentCreated(BrowserField browserField, ScriptEngine scriptEngine, Document document) throws Exception
{
int index = browserField.getDocumentUrl().indexOf("#access_token=");
if (index == -1)
{
super.documentCreated(browserField, scriptEngine, document);
}
else
{
access_token = browserField.getDocumentUrl().substring(index + "#access_token=".length(), browserField.getDocumentUrl().indexOf("&"));
PersistentObject store = PersistentStore.getPersistentObject(STORE_KEY);
FacebookTokens store_tokens = new FacebookTokens();
store_tokens.access_token = access_token;
store.setContents(store_tokens);
store.commit();
if (task != null) task.execute();
OAuthScreen.this.close();
}
}
public void documentLoaded(BrowserField browserField, Document document) throws Exception
{
super.documentLoaded(browserField, document);
loading_dialog.close();
}
});
// whatever loading dialog you want, this sometimes takes a while to open
loading_dialog = LoadingDialog.push(loading_field);
add(browser_field);
browser_field.requestContent("https://www.facebook.com/dialog/oauth?client_id="+APP_ID+"&response_type=token&redirect_uri=http://www.facebook.com/connect/login_success.html&scope=publish_actions");
}
}
The callback task is just for if I want to perform a call directly after login.
Now just perform API calls as you need them. API methods here https://developers.facebook.com/docs/graph-api/reference/v2.0/
Methods that require the access token, should have it appended to the url such as, https://graph.facebook.com/me/feed?access_token=" + access_token
Be aware that clearing your access token won't clear the token stored in the browser field. And will mean that you can't login next time (because the browser is still logged in).
So if you want to logout you need to open this link in a browserfield before clearing your local access token "https://www.facebook.com/logout.php?next=http://www.facebook.com/connect/login_success.html&access_token=" + access_token
Clearing the cookies of the browser should suffice, but I haven't found a way to do this.
I'm making a web application for blackberry and I really need the current URL
In the description of documentUrl, it says
This method will return the URL of the currently loaded page of this BrowserField Instance
My code is:
_bf2.requestContent("google.com";);
add(_bf2);
Global.c = _bf2.getDocumentUrl();
Global.be=new BasicEditField("URL: "+Global.c,Global.c);
add(Global.be);
and the weird thing is that www.google.com gets loaded in the BrowserField and the documentUrl returns null.
This is my current code:
BrowserField _bf2 = new BrowserField();
MYBrowserFieldListener _listener = new MYBrowserFieldListener();
_bf2.requestContent("google.com";);
_bf2.addListener(_listener);
String url=_bf2.getDocumentUrl();
Global.be=new BasicEditField("URL: "+url,url);
add(Global.be);
add(_bf2);
I changed it to
final BrowserField _bf2 = new BrowserField();
_bf2.requestContent("google.com";);
//_bf2.addListener(listener);
Global.be=new BasicEditField("URL: "+Global.c,Global.c);
add(Global.be);
add(_bf2);
_bf2.addListener(new BrowserFieldListener(){
public void documentLoaded(BrowserField _bf2, Document document) throws Exception {
Global.c=_bf2.getDocumentUrl();
}
});
But it still returns null. Can someone please tell me how to fix this? Thanks in advance!
I would say that Arhimed has answered your question. An HTTP request is a very time consuming process (from a CPU perspective) and will block until the server responds. I suspect that RIM programmers have coded the requestContent() method as per their own recommendations and are fetching the web content on a separate thread. So, requestContent() will return immediately, when you call getDocumentUrl() it is still null since the fetch thread has probably not even connected to the server at this point.
You will need to implement a BrowserFieldListener and listen for documentLoaded().
I'm trying to create an application for checking in and out devices. So far I've had success but I'm trying to have the main index.jsp page that is called display a message like "connecting" until a connection with the database can actually be made and then display the actual login form. So something to the effect of.
Connection con = null;
String driver = "com.mysql.jdbc.Driver";
Class.forName(driver).newInstance();
boolean connected = false;
while(!connected) {
try{
String url = "jdbc:mysql://localhost:3306/test";
con = DriverManager.getConnection(url,"username","password");
connected = true;
//Display form allowing user to authenticate login
} catch(Exception e) {
//Display Message "Attempting to connect to database"
}
}
The problem I have is the same message will get repeated over and over on the web page but i just want it to display once and stay there until the connection is found and then be removed and replace with the login form. Any thoughts?
I suggest you use connection pooling.... it is quite easy to manage connections and the overheads will be reduced making your app efficient.
see the following:
Setup Connection pooling in jsp/servlets application based on MVC?
Am I Using JDBC Connection Pooling?
Setup Connection pooling in jsp/servlets application based on MVC?
Change your code in this way
Connection con = null;
String driver = "com.mysql.jdbc.Driver";
Class.forName(driver).newInstance();
boolean connected = false;
int errors = 0;
while(!connected) {
try{
String url = "jdbc:mysql://localhost:3306/test";
con = DriverManager.getConnection(url,"root","ikkakumon");
connected = true;
//Display form allowing user to authenticate login
} catch(Exception e) {
if (errors == 0) {
//Display Message "Attempting to connect to database"
}
++errors;
Thread.sleep(1000); // wait a little before the next attempts
}
}
So you display the error message only the first time.
Then wait between one attempt and another.
However the loop is not the usual way to manage connections. Why do you choose this solution? What kind of exception do you receive?
Any thoughts?
Yes.
You need to get your head around HOW stuff gets displayed to the user. The normal model is that browser sends an HTTP request to the server, the server creates a response consisting of an HTML page, and the browser displays the page. In this model, if you want the user to see the results, the server has to finish and send the response. Until it does, the user sees a blank screen (or the page he / she was on previously).
So to get a page that updates you have two options:
You can create a page that contains a <meta> tag that requests that the browser resends the request after a fixed delay; see http://en.wikipedia.org/wiki/Meta_refresh
You can embed javascript in the page to send an AJAX request to the server asking for an update. On getting a response, the javascript then needs to modify the DOM of the displayed page with the new information.
But in either case, it should be clear to you that you cannot do this in a single call to a single JSP. So you don't want a while loop in the JSP, or in the controller that dispatches to the JSP. The looping has to happen at the browser end.