Develop Reference

How to find CurrentPage in HtmlUnit from webclient

When I used HttpUnit, I would invoke getCurrentPage() method of HttpUnit to get the current page. How can I do that in HtmlUnit? I tried webclient.getHomePage(), but it seem to return wesite of htmlunit.
One suggestion I got is use getPage using previous URL, but that doesn't work for me because I need to refactor a code that is earlier written in a code which makes it impossible reexecute previous request.

You can use the following approach to get HtmlPage object from webclient assuming you already navigaed to a page either by using wc.getPage(url) or submitting a form in previous page or using any other method. Assuming that wc is the WebClient object.
HtmlPage currentPage = (HtmlPage) wc.getCurrentWindow().getEnclosedPage();

POSTing a request to the correct URL once HTMLUnit is ignoring the form.setActionAttribute and fom.setAttribute

I'm trying to submit a form using HTMLUnit but it seems that the action attribute of the form is ignored once the http post is going to the same page.
I'm getting the form on this URL:
http://www.tjse.jus.br/tjnet/consultas/internet/consnomeparte.wsp
And in the source code of this URL we can find that the action attribute is set to this URL:
http://www.tjse.jus.br/tjnet/consultas/internet/respconsnomeparte.wsp
But HTMLUnit always post to the first URL.
I'm using fiddler to analyse the request through a real web browser and through HTMLUnit and comparing the two HTTP POST it's easy to see that HTMLUnit is POSTing to the same site, i.e, the first URL mentioned.
I need that HTMLUnit POST to the second URL.
If anyone could help me I'll appreciate.

Problem solved.
Instead of using:
HtmlPage page2 = button.click();
I used:
button.click().getWebResponse().getContentAsString();

I would use something simular to the following.
// Enter your username in feild
searchForm.getInputByName("Username").setValueAttribute(schoolID);
//Submit the form and get the result page
HtmlPage pageResult = (HtmlPage) searchForm.getInputByValue("Search").click();
//Page results in raw html source code
String html = pageResult.asXml();
/*
* filter source code if needed to collect desired data
*/
//login via another server url
page = (HtmlPage) webClient.getPage("https://"+url);
HtmlForm LoginForm = page.getFormByName("Form1");
// login to web portal
LoginForm.getInputByName("txtUserName").setValueAttribute(username);
LoginForm.getInputByName("txtPassword").setValueAttribute(password);
//Submit the form and get the result page
HtmlPage pageResult = (HtmlPage) LoginForm.getInputByName("btnLogin").click();
Note: this htmlUnit code complys with htmlunit 2.15 API

Posting data using web client

I am creating a submit button using a web client, but it's not working.
This is the code, which I am using:
HtmlElement button = firstPage.createElement("button");
button.setAttribute("type", "submit");
button.setAttribute("name", "submit");
form.appendChild(button);
System.out.println(form.asXml());
HtmlPage pageAfterLogin = button.click();

Simple question, Do you have HTML FORM tag included in the page?
Also see below link, not sure if it helps you
HtmlUnit, how to post form without clicking submit button?

Getting Final HTML with Javascript rendered Java as String

I want to fetch data from an HTML page(scrape it). But it contains reviews in javascript. In normal java url fetch I am only getting the HTML(actual one) without Javascript executed. I want the final page with Javascript executed.
Example :- http://www.glamsham.com/movies/reviews/rowdy-rathore-movie-review-cheers-for-rowdy-akki-051207.asp
This page has comments as a facebook plugin which are fetched as Javascript.
Also similar to this even on this.
http://www.imdb.com/title/tt0848228/reviews
What should I do?

Use phantomjs: http://phantomjs.org
var page = require('webpage').create();
page.open("http://www.glamsham.com/movies/reviews/rowdy-rathore-movie-review-cheers-for-rowdy-akki-051207.asp")
setTimeout(function(){
// Where you want to save it
page.render("screenshoot.png")
// You can access its content using jQuery
var fbcomments = page.evaluate(function(){
return $(".fb-comments iframe").contents().find(".postContainer")
})
},10000)
You have to use the option in phantom --web-security=no to allow cross-domain interaction (ie for facebook iframe)
To communicate with other applications from phantomjs you can use a web server or make a POST request: https://github.com/ariya/phantomjs/blob/master/examples/post.js

You can use HTML Unit, A java based "GUI LESS Browser". You can easily get the final rendered output of any page because this loads the page as a web browser do so and returns the final rendered output. You can disable this behaviour though.
UPDATE: You were asking for example? You don't have to do anything extra for doing that:
Example:
WebClient webClient = new WebClient();
HtmlPage myPage = ((HtmlPage) webClient.getPage(myUrl));
UPDATE 2: You can get iframe as follows:
HtmlPage myFrame = (HtmlPage) myPage.getFrameByName(myIframeName).getEnclosedPage();
Please read the documentation from above link. There is nothing you can't do about getting page content in HTMLUnit

The simple way to solve that problem.
Hello, you can use HtmlUnit is java API, i think it can help you to access the executed js content, as a simple html.
WebClient webClient = new WebClient();
HtmlPage myPage = (HtmlPage) webClient.getPage(new URL("YourURL"));
System.out.println(myPage.getVisibleText());

How to programmatically access web page in java

There is a web page from which I want to retrieve a certain string. In order to do so, I need to login, click some buttons, fill a text box, click another button - and then the string appears.
How can I write a java program to do that automatically? Are there any useful libraries for that purpose?
Thanks

Try HtmlUnit
HtmlUnit is a "GUI-Less browser for
Java programs". It models HTML
documents and provides an API that
allows you to invoke pages, fill out
forms, click links, etc... just like
you do in your "normal" browser.
Example code for submiting form:
#Test
public void submittingForm() throws Exception {
final WebClient webClient = new WebClient();
// Get the first page
final HtmlPage page1 = webClient.getPage("http://some_url");
// Get the form that we are dealing with and within that form,
// find the submit button and the field that we want to change.
final HtmlForm form = page1.getFormByName("myform");
final HtmlSubmitInput button = form.getInputByName("submitbutton");
final HtmlTextInput textField = form.getInputByName("userid");
// Change the value of the text field
textField.setValueAttribute("root");
// Now submit the form by clicking the button and get back the second page.
final HtmlPage page2 = button.click();
webClient.closeAllWindows();
}
For more details check:
http://htmlunit.sourceforge.net/gettingStarted.html

The super simple way to do this is using HtmlUnit here:
http://htmlunit.sourceforge.net/
and what you want to do can be as simple as:
#Test
public void homePage() throws Exception {
final WebClient webClient = new WebClient();
final HtmlPage page = webClient.getPage("http://htmlunit.sourceforge.net");
assertEquals("HtmlUnit - Welcome to HtmlUnit", page.getTitleText());
}

Take a look at the apache HttpClient project, or if you need to run Javascript on the page, try HttpUnit.

Well when you press a button usually you do a request via a HTTP POST method, so you should use HttpClient to handle request and HtmlParser to handle the response page with the string you need.

Yes:
java.net.URL#openConnection() will allow you to make http requests and get the http responses
Apache HttpComponents is a library that makes it easier to work with HTTP.