Develop Reference

How to get html of fully loaded page (with javascript) as input in java?

I need to parse page, everything is ok except some elements on page are loaded dynamically. I used jsoup for static elements, then when I realized that I really need dynamic elements I tried javafx. I read a lot of answeres on stackoverflow and there were many recommendations to use javafx WebEngine. So I ended with this code.
#Override
public void start(Stage primaryStage) {
WebView webview = new WebView();
final WebEngine webengine = webview.getEngine();
webengine.getLoadWorker().stateProperty().addListener(
new ChangeListener<State>() {
public void changed(ObservableValue ov, State oldState, State newState) {
if (newState == Worker.State.SUCCEEDED) {
Document doc = webengine.getDocument();
//Serialize DOM
OutputFormat format = new OutputFormat (doc);
// as a String
StringWriter stringOut = new StringWriter ();
XMLSerializer serial = new XMLSerializer (stringOut, format);
try {
serial.serialize(doc);
} catch (IOException e) {
e.printStackTrace();
}
// Display the XML
System.out.println(stringOut.toString());
}
}
});
webengine.load("http://detail.tmall.com/item.htm?spm=a220o.1000855.0.0.PZSbaQ&id=19378327658");
primaryStage.setScene(new Scene(webview, 800, 800));
primaryStage.show();
}
I made string from org.w3c.dom.Document and printed it. But it was useless too. primaryStage.show() showed me fully loaded page (with element I need rendered on page), but there was no element I need in html code (in output).
This is the third day I'm working on that issue, of course lack of experience is my main problem, nevertheless I have to say: I'm stuck. This is my first java project after reading java complete reference. I make it to get some real-world experience (and for fun). I want to make parser of chinese "ebay".
Here is the problem and my test cases:
http://detail.tmall.com/item.htm?spm=a220o.1000855.0.0.PZSbaQ&id=19378327658
need to get dynamically loaded discount "129.00"
http://item.taobao.com/item.htm?spm=a230r.1.14.67.MNq30d&id=22794120348
need "15.20"
As you can see, if you view this pages with browser at first you see original price and after a second or so - discount.
Is it even possible to get this dynamic discounts from html page? Other elements I need to parse are static. What to try next: another library to render html with javascript or maybe smth else? I really need some advice, don't want to give up.

DOM model returned after Worker.State.SUCCEEDED shoulb be already processed by javascript.
Your code worked for me with tested with FX 7u40 and 8.0 dev. I see next output in the log:
<DIV id="J_PromoBox"><EM class="tb-promo-price-type">夏季新品</EM><EM class="tm-yen">¥</EM>
<STRONG class="J_CurPrice">129.00</STRONG></DIV>
which is dynamically loaded box with data (129.00) you looked for.
You may want to upgrade your JDK to 7u40 or revisit your log parsing algorithm.

It sounds like you want the rendered DOM from a dynamic page after the Javascript on the page has finished modifying the original HTML. This would not be easy to do in Java as you would need to implement browser-like functionality with an embedded Javascript engine. If you only care about reading a web page from Java, you might want to look into Selenium since it takes control of a browser and allows you to pull the rendered HTML into Java.
This answer might also help:
Render JavaScript and HTML in (any) Java Program (Access rendered DOM Tree)?

Getting started with developing for Chrome

I have been asked by my friend to make an application for Chrome and it requires me to have context-sensitive menus as below:
I have never really made anything for Chrome before and I have a few questions regarding it:
I will have to develop a plug-in, right ?
If so, is there a specific set of rules I have to follow ?
I know I can use GWT to compile Java to JavaScript
3. This context sensitive menu is the same as JPopupMenu ?
The application I want to develop is simple:
Copy some text,
right-click, click on the context sensitive menu
apply simple Caesar's cipher to the text
open a new JFrame with JtextArea in it to display the encrypted text.

What you're creating is called an "extension", not a "plug-in". A browser extension is written using HTML, CSS and Javascript, and got access to APIs for direct interaction with the browser.
Plug-ins, on the other hand, are compiled binaries such as Flash and Java.
Drop the idea of using GWT for Chrome extensions. It makes development of the extension harder, not easier (open issue).
Especially because you'll find plenty of vanilla JavaScript examples and tutorials in the documentation and Stack Overflow.
You just have to know the relevant APIs:
Copy some text,
right-click, click on the context sensitive menu
Use chrome.contextMenus. There's no need to copy, the selected text is available in the callback (examples).
apply simple Caesar's cipher to the text
Create a JavaScript function to achieve this.
open a new JFrame with JtextArea in it to display the encrypted text.
Create a new window using chrome.windows.create. You could include an extra HTML page in your extension, and use the message passing APIs to populate the text field, but since you appear to be a complete newbie, I show a simple copy-paste method to create and populate this window:
function displayText(title, text) {
var escapeHTML = function(s) { return (s+'').replace(/</g, '<'); };
var style = '*{width:100%;height:100%;box-sizing:border-box}';
style += 'html,body{margin:0;padding:0;}';
style += 'textarea{display:block;}';
var html = '<!DOCTYPE html>';
html += '<html><head><title>';
html += escapeHTML(title);
html += '</title>';
html += '<style>' + style + '</style>';
html += '</head><body><textarea>';
html += escapeHTML(text);
html += '</body></html>'
var url = 'data:text/html,' + encodeURIComponent(html);
chrome.windows.create({
url: url,
focused: true
});
}
Don't forget to read Getting started to learn more about the extension's infrastructure.

Check out Google Chrome Extensions Chrome Extensions
The Getting Started will help you Getting Started
You will find a section on how to use Context Menus.

Java SE: Open Web Page and Click a Button

I have a Java 7 program (using WebStart technology, for Windows 7/8 computers only).
I need to add a function so that my program clicks a button on a page with known URL (https).
Some people suggest WebKit SWT, but I went to their site and they say that the project was discontinued. (http://www.genuitec.com/about/labs.html)
Other people say that JxBrowser is the only option but it looks like it's over $1,300 which is crazy. (http://www.teamdev.com/jxbrowser/onlinedemo/)
I'm looking for something simple, free, lightweight, and able to open HTTPS link, parse HTML, access a button through DOM and click it. Perhaps some JavaScript too, in case there are JS handlers.
Thanks for your help.

You may be looking for HtmlUnit -- a "GUI-Less browser for Java programs".
Here's a sample code that opens google.com, searches for "htmlunit" using the form and prints the number of results.
import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.*;
public class HtmlUnitFormExample {
public static void main(String[] args) throws Exception {
WebClient webClient = new WebClient();
HtmlPage page = webClient.getPage("http://www.google.com");
HtmlInput searchBox = page.getElementByName("q");
searchBox.setValueAttribute("htmlunit");
HtmlSubmitInput googleSearchSubmitButton =
page.getElementByName("btnG"); // sometimes it's "btnK"
page=googleSearchSubmitButton.click();
HtmlDivision resultStatsDiv =
page.getFirstByXPath("//div[#id='resultStats']");
System.out.println(resultStatsDiv.asText()); // About 309,000 results
webClient.closeAllWindows();
}
}
Other options are:
Selenium: Will open a browser like Firefox and operate it.
Watij: Also will open a browser, but in its own window.
Jsoup: Good parser. No JavaScript, though.

Your question is kind of difficult to understand what you want. If you have a webstart app and want to open a link in the browser, you can use the java.awt.Desktop.getDesktop().browse(URI) method.
public void openLinkInBrowser(ActionEvent event){
try {
URI uri = new URI(WEB_ADDRESS);
java.awt.Desktop.getDesktop().browse(uri);
} catch (URISyntaxException | IOException e) {
//System.out.println("THROW::: make sure we handle browser error");
e.printStackTrace();
}
}

How to show a captcha in a java swing application

I need to add a captcha validator in a java swing application. I have been searching some libraries (JCaptcha and SimpleCatcha) but they are for web development.
Is there any library to use captcha on swing? and if it's not, is there a web page or repository with some captcha caracters to implement my own captcha?
I really appreciate your time and your help.
Thanks in advance.

JCaptcha can return a BufferedImage. From there it is not much difficult to get the image visible using a JLabel:
BufferedImage captcha = // Get the captcha
// See also com.octo.captcha.service.image.AbstractManageableImageCaptchaService.getImageChallengeForID(String)
JLabel label = new JLabel(new ImageIcon(captcha));
// ... add that label to a visible container of your Swing application
In version 1.0, you can use this: http://jcaptcha.sourceforge.net/apidocs/1.0/com/octo/captcha/service/image/AbstractManageableImageCaptchaService.html
In 2.0-alpha1, there is this: http://jcaptcha.sourceforge.net/apidocs/2.0-alpha1/com/octo/captcha/service/image/AbstractManageableImageCaptchaService.html#getImageChallengeForID(java.lang.String)
You can also check the overloaded version of those methods with an extra Locale argument.
In each case, there is a default implementing class DefaultManageableImageCaptchaService.

BufferedImage captcha = // Get the captcha
// See also
com.octo.captcha.service.image.AbstractManageableImageCaptchaService.getImageChallengeForID(String)
JLabel label = new JLabel(new ImageIcon(captcha));
// ... add that label to a visible container of your Swing application

How to use HTML and CSS as a Java application GUI?

I want to design new Git client with a clean GUI.
Is it possible to use the power of HTML, CSS and JavaScript in a java application?
I would like to use Java + JGit for models, Java for controllers and HTML + CSS + JavaScript for views.
I don't want a client-server model. I would like to integrate Java and HTML nicely. A DOM event would fire events directly to a Java controller. This way it would be possible to create rich offline application.

You can embed web browser component into your Java Swing/JavaFX Desktop application that displays GUI built with HTML5+CSS+JavaScript. You can see an article that describes how to do this at https://jxbrowser-support.teamdev.com/docs/tutorials/cross-desktop-apps.html
One of the Java Swing/JavaFX libraries that allows embedding Chromium into Java applications is JxBrowser. Using JxBrowser API you can load any web page and work with its DOM and JavaScript. You can even call Java methods from JavaScript code and vice versa. For example:
import com.teamdev.jxbrowser.chromium.Browser;
import com.teamdev.jxbrowser.chromium.JSFunctionCallback;
import com.teamdev.jxbrowser.chromium.JSObject;
import com.teamdev.jxbrowser.chromium.JSValue;
import com.teamdev.jxbrowser.chromium.events.FinishLoadingEvent;
import com.teamdev.jxbrowser.chromium.events.LoadAdapter;
public class JavaScriptJavaSample {
public static void main(String[] args) {
Browser browser = new Browser();
browser.addLoadListener(new LoadAdapter() {
#Override
public void onFinishLoadingFrame(FinishLoadingEvent event) {
if (event.isMainFrame()) {
Browser browser = event.getBrowser();
JSObject window = (JSObject)
browser.executeJavaScriptAndReturnValue("window");
window.setProperty("MyFunction", new JSFunctionCallback() {
#Override
public Object invoke(Object... args) {
for (Object arg : args) {
System.out.println("arg = " + arg);
}
return "Hello!";
}
});
JSValue returnValue = browser.executeJavaScriptAndReturnValue(
"MyFunction('Hello JxBrowser!', 1, 2, 3, true);");
System.out.println("return value = " + returnValue);
}
}
});
browser.loadURL("about:blank");
}
}

It's not really feasible. Rich clients in Java are done using Swing or SWT.
If you want to use HTML/CSS for your user interface, you need to use the server/client model. It can be as simple as creating a local server and launching a browser that connects to it, but it would still be that model.
If you absolutely need to have HTML/CSS as your UI framework and can't go to a server/client model, your best bet is probably looking at something like Google Native Client, but that uses C/C++ bindings on the backend. I haven't used Native Client so I can't personally give much more information on that front.
Edit to add:
One option is to embed a native browser into your Swing app using something like: http://djproject.sourceforge.net/ns/
There are some pure Java HTML renderers, however, they most likely won't be fully HTML5/CSS3 compliant, let alone possibly have Javascript bugs as well.
See here for some of those options: Pure Java HTML viewer/renderer for use in a Scrollable pane

Like #Reverand Gonzo says, you will need some form of server/client. But you could easily embed a Jetty server into a Java app and then use GWT for your client code.

You can bring in the power of HTML,CSS,JavaScript into your Swing app using JFXPanel to embed JavaFX WebView. Have a look at the SimpleSwingBrowser demo in this link:https://docs.oracle.com/javase/8/javafx/interoperability-tutorial/swing-fx-interoperability.htm
WebView allows to call JavaScript functions from Java and vice versa. It is also a nice way to enhance your legacy Java app with web technologies.

JavaFX 2.2 brought this functionality to providing a user interface component (GUI) that has web view and full browsing functionality.
For more details, see Adding HTML Content to JavaFX Applications.

Use Angular.js with HTML and rest of the things as same in Java, just use classes for business logic, no need to write code for awt/swing. Angular with spring boot are rapid development in Java for webapp with less code in Java without swing use to create best webapp .