Develop Reference

Scraping from website that needs authentication using Java

I found this code on an old form and I am trying to get it to work but am getting this error:
File: /net/home/f13/dlschnettler/Desktop/javaScraper/RedditClient.java [line: 46]
Error: cannot access org.w3c.dom.ElementTraversal
class file for org.w3c.dom.ElementTraversal not found
Here's the code:
import java.io.IOException;
import java.net.MalformedURLException;
import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlForm;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
public class RedditClient {
//Create a new WebClient with any BrowserVersion. WebClient belongs to the
//HtmlUnit library.
private final WebClient WEB_CLIENT = new WebClient(BrowserVersion.CHROME);
//This is pretty self explanatory, these are your Reddit credentials.
private final String username;
private final String password;
//Our constructor. Sets our username and password and does some client config.
RedditClient(String username, String password){
this.username = username;
this.password = password;
//Retreives our WebClient's cookie manager and enables cookies.
//This is what allows us to view pages that require login.
//If this were set to false, the login session wouldn't persist.
WEB_CLIENT.getCookieManager().setCookiesEnabled(true);
}
public void login(){
//This is the URL where we log in, easy.
String loginURL = "https://www.reddit.com/login";
try {
//Okay, bare with me here. This part is simple but it can be tricky
//to understand at first. Reference the login form above and follow
//along.
//Create an HtmlPage and get the login page.
HtmlPage loginPage = WEB_CLIENT.getPage(loginURL);
//Create an HtmlForm by locating the form that pertains to logging in.
//"//form[#id='login-form']" means "Hey, look for a <form> tag with the
//id attribute 'login-form'" Sound familiar?
//<form id="login-form" method="post" ...
HtmlForm loginForm = loginPage.getFirstByXPath("//form[#id='login-form']");
//This is where we modify the form. The getInputByName method looks
//for an <input> tag with some name attribute. For example, user or passwd.
//If we take a look at the form, it all makes sense.
//<input value="" name="user" id="user_login" ...
//After we locate the input tag, we set the value to what belongs.
//So we're saying, "Find the <input> tags with the names "user" and "passwd"
//and throw in our username and password in the text fields.
loginForm.getInputByName("user").setValueAttribute(username);
loginForm.getInputByName("passwd").setValueAttribute(password);
//<button type="submit" class="c-btn c-btn-primary c-pull-right" ...
//Okay, you may have noticed the button has no name. What the line
//below does is locate all of the <button>s in the login form and
//clicks the first and only one. (.get(0)) This is something that
//you can do if you come across inputs without names, ids, etc.
loginForm.getElementsByTagName("button").get(0).click();
} catch (FailingHttpStatusCodeException e) {
e.printStackTrace();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
public String get(String URL){
try {
//All this method does is return the HTML response for some URL.
//We'll call this after we log in!
return WEB_CLIENT.getPage(URL).getWebResponse().getContentAsString();
} catch (FailingHttpStatusCodeException e) {
e.printStackTrace();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}
import org.jsoup.Jsoup;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class Main {
public static void main(String[] args) {
//Create a new RedditClient and log us in!
RedditClient client = new RedditClient("hutsboR", "MyPassword!");
client.login();
//Let's scrape our messages, information behind a login.
//https://www.reddit.com/message/messages/ is the URL where messages are located.
String page = client.get("https://www.reddit.com/message/messages/");
//"div.md" selects all divs with the class name "md", that's where message
//bodies are stored. You'll find "<div class="md">" before each message.
Elements messages = Jsoup.parse(page).select("div.md");
//For each message in messages, let's print out message and a new line.
for(Element message : messages){
System.out.println(message.text() + "\n");
}
}
}
Not really sure how to fix it since I'm not very familiar with scraping in the first place.

Try to add xml-apis to your classpath

How to run a Java applet in the browser: "Class Not Found exception"

I'm trying to get a Java applet to display in the browser - I know this question has been asked a number of times but I can't seem to find the answer that works specifically for this case - over the past few days I've tried everything from moving the HTML file to various places in the directory structure to using <applet> vs. the deployJava() API.
The code runs fine as a standalone applet in Eclipse, but when I try to run it in the browser I get either a "ClassNotFound" or "ClassDefNotFound" exception. I've packaged the code into a .jar and placed the .jar within the same folder as the HTML file, with my java code as follows:
package myPackage;
import java.awt.*;
import java.applet.*;
public class myClass extends java.applet.Applet{
public void init(){
String latLong = getParameter("unUsedParameter");
}
public void paint(Graphics g){
g.drawString("Hello World",50,25);
}
}
and the Javascript code is as follows:
<script src="https://www.java.com/js/deployJava.js"></script>
<section id = "java">
<script type="text/javascript">
var attributes = {
code:'myClass.class',
archive: 'myApplet.jar',
width:500, height:500
};
var parameters = {latLong: total_path};
var version = '1.8';
deployJava.runApplet(attributes, parameters, version);
</script>
</section>
I also tried using codebase: 'myApplet.jar' instead of archive: but that didn't work either - I keep getting one of the same two exceptions. HELP!
EDIT: First off, the code: attribute was incorrect in my original post, it should have read 'myClass.class' (this is corrected above). The answer that got it working was changing the code: attribute to code: 'myApplet/myClass' - thanks for your help!

Change your 'code' parameter to 'myPackage.myClass', instead of 'myApplet.class'.
You have more insights about the declaration in this post, which I've shown a way that works Angular.js and Java Applet
ie:
<script>
<!-- applet id can be used to get a reference to the applet object -->
var attributes = { id:'cdigApplet', code:'cdig.CDigApplet', archive:'cdig-applet-1.0.jar', width:1, height:1, classloader_cache:'false'} ;
var parameters = {persistState: false, cache_option:'no' } ;
deployJava.runApplet(attributes, parameters, '1.8');
</script>
My Applet
package cdig;
import java.applet.Applet;
import java.security.AccessController;
import java.security.PrivilegedAction;
import java.util.Base64;
/**
*
* #author Ulysses Marins
*/
public class CDigApplet extends Applet
{
private static final long serialVersionUID = 1L;
String ret;
CDigApplet applet = this;
#SuppressWarnings({ "rawtypes", "unchecked" })
public String signFile(String fileID, String pin, String token)
{
AccessController.doPrivileged(new PrivilegedAction()
{
#Override
public Object run()
{
try
{
File objFile = new File(token);
System.out.println("Iniciando processo de assinatura.");
objFile.sign("json", sig);
System.out.println(ret);
} else {
throw new IllegalArgumentException("Não foi possível iniciar processo de assinatura.");
}
}
catch (Exception e)
{
String sl = "{\"success\":false," + "\"message\":\"" + e.getMessage() + "\"}";
ret = sl;
System.out.println(sl);
}
return null;
}
});
return ret;
}
public void init(){
}
public void destroy(){
}
}

Browsers are becoming more and more reluctant to run applets. Chrome itself won't support the java plugin anymore very soon. Additionally, Java itself is requiring more or more secured applications. Your version of Java matters a lot. Versions >=7 require signed applications.
What's more, the error messages when applets fail to run for these reasons are generally very cryptic or inexistant.
What's clear is that you at least need to self-sign your applet, and declare some properties in the jar's manifest, such as the fact that you want to run the code in a sandbox (which will relax a bit the security restrictions).

How to recal unique text using java div class

I'd like to make one text for multiple html files, something like greating. Let's say greating is:
"Hello, if you have any questions please conatact me."
What I want is to recall that text on every html page. And later if I change it, the change would appear on all the html pages.
I am weak on java, but I think I need to create some javascript and recall the text with div class function, like the facebook button is made.
P.S. Facebook button recall:
<div class="fb-like" data-href="https://developers.facebook.com/docs/plugins/" data- layout="standard" data-action="like" data-show-faces="true" data-share="true">

with javascript you can change the content of a tag with html() function, or you could include the resource, i guess it depends on the technology being used

In the simplest form, you can create a function in your javascript master copy and make a document.write call. You would need to call that script file on every page.
function greetingMessage() {
document.write('your message);
};
Then
call greetingMessage();
you can also put the javascript in a master file and then have the div in each HTML page:
function greetingMessage(){
document.getElementById('Message').innerHTML = 'Your Message';
};
HTML:
<body onload="greetingMessage();">
<div id="Message" style="color:red;"></div>

If you are using JSP's or Servlets you can have a resource/properties file that contains many Strings being used throughout your application. The properties file would contain key=value pairs. You could then simply reference a particular key in the properties file, for instance:
greeting=Hello, if you have any questions please contact me
The key is "greeting", the value is "Hello, if you have any questions please contact me"
To read in the properties file you would use the Properties class like so:
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Properties;
public class MyWebPage extends HttpServlet
{
public void doGet(HttpServletRequest req, HttpServletResponse res)
{
PrintWriter out = response.getWriter();
response.setContentType("text/html");
out.print("<html><head></head><body><div class=\"someclass\">" +
getGreeting() + "</div>"
"</body></html>"
);
}
public String getGreeting()
{
String greeting = "";
try{
Properties prop = new Properties();
InputStream input = new FileInputStream("config.properties");
// load a properties file
prop.load(input);
greeting = prop.getProperty("greeting");
input.close();
}
catch(IOException ioe){ioe.printStackTrace();}
finally{
if (input != null)
{
try {
input.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return greeting;
}
}
The same sort of thing can be effectively used in Java Server Pages. Hope this helps.

Malformed URL when connecting applet to servlet

I'm trying to access a servlet from a java applet and set the servlet's response in the applet's text field.
I'm using tomcat 7.0 and my jre/jdk are fully updated.
The servlet runs fine (correct output in the browser) when invoked from the browser as localhost:8080/hello/hello?query=select * from airports
(where airports is the name of the database)
However when i run the applet in appletviewer, i get a Malformed URL exception thrown..
Code for Applet:
import java.applet.*;
import java.awt.*;
import java.awt.event.*;
import java.net.*;
import java.io.*;
/*
<applet code="lab101" width=500 height=270>
</applet>
*/
public class lab101 extends Applet implements ActionListener{
TextArea t;
Panel p,q,r;
CheckboxGroup c;
Checkbox ins,dis,del,update; //Checkboxes are included just for testing purposes.
TextField f;
Label l1;
Button b;
public void init(){
setLayout(new FlowLayout());
b=new Button("Run");
l1=new Label("Query:");
c=new CheckboxGroup();
t=new TextArea("",10,50);
p=new Panel();
q=new Panel();
r=new Panel();
p.add(t);
ins=new Checkbox("Insert",c,false);
dis=new Checkbox("Display",c,true);
del=new Checkbox("Delete",c,false);
update=new Checkbox("Update",c,false);
f=new TextField(50);
q.add(ins);
q.add(dis);
q.add(del);
q.add(update);
r.add(l1);
r.add(f);
r.add(b);
b.addActionListener(this);
add(p);
add(q);
add(r);
try{
URL url=new URL("127.0.0.1:8080/hello/hello?query=select * from airports");
URLConnection servletconnection=url.openConnection();
servletconnection.setDoInput(true);
InputStream in=servletconnection.getInputStream();
String s="";
int ch;
loop:while(1>0){
ch=in.read();
if(ch==-1) break loop;
else s+=(char)ch;
}
t.setText(s);
}//try close
catch(MalformedURLException e){
t.setText("Malformed URL Exception occured.");}
catch(IOException e){
t.setText("IO exception occured");}
}
public void actionPerformed(ActionEvent ae){
}
public void start(){
}
public void paint(Graphics g){
}
}//class ends
Code for servlet:
import java.io.*;
import javax.servlet.ServletException;
import javax.servlet.http.*;
import java.sql.*;
public class hello extends HttpServlet{
public void doGet(HttpServletRequest request,HttpServletResponse response)throws ServletException, IOException {
response.setContentType("text/plain");
PrintWriter out=response.getWriter();
String query=request.getParameter("query");
Connection link=null;
Statement statement=null;
ResultSet results=null;
try{
Class.forName("org.postgresql.Driver");
String url = "jdbc:postgresql://localhost:5432/test";
link=DriverManager.getConnection(url,"postgres","hooligan");
out.println("Successful connection");
}
catch(ClassNotFoundException e){
out.println("Unable to load driver");
}
catch(SQLException e){
out.println("Cannot connect to database");
}
try{
statement=link.createStatement();
//String select="select * from airports";
results=statement.executeQuery(query);
}
catch(SQLException e){
out.println("Cannot execute query");
e.printStackTrace();
}
try{
out.println();
while(results.next()){
out.println("Name: " + results.getString(1));
out.println("Location: " + results.getString(2));
//System.out.println("Account no: " + results.getInt(3));
System.out.println();}
}
catch(SQLException e){
out.println("Error retrieving data");
}
try{
link.close();}
catch(SQLException e){
out.println("Unable to disconnect");}
out.close();
out.flush();
}}
Any thoughts?
P.S. i also noticed that if i use localhost instead of 127.0.0.1 i get a Security Exception thrown (Probably because the applet is unsigned?)

There are 2 (actually 3) problems:
First, an applet is only allowed to fire HTTP requests on the exact URL base as where the applet is been served from. You can obtain it by Applet#getCodeBase() which needs to be used as follows:
URL url = new URL(getCodeBase(), "hello?query=select * from airports");
URLConnection connection = url.openConnection();
// ...
Second, your query string contains illegal characters for use in URLs (space, asterisk). You need to use URLEncoder#encode() to URL-encode the query string.
String query = URLEncoder.encode("select * from airports", "UTF-8");
URL url = new URL(getCodeBase(), "hello?query=" + query);
URLConnection connection = url.openConnection();
// ...
You also need to ensure that you open the HTML/JSP page with the applet in the browser on the same base URL as where the servlet runs. E.g. http://localhost:8080/hello/pagewithapplet.html and thus not from commandline or by an appletviewer or something. The applet really needs to be served from the same webserver as where the servlet runs.
Unrelated to the concrete problem as stated in the question, your third problem is that sending a plain SQL statement as request parameter is a very bad idea. What if a hacker decompiles your applet and figures how the applet-servlet communication is done and then modifies the SQL statement into something else, such as delete from airports?
Do not do the SQL in the applet, do it in the servlet only and let the applet send specific commands only, such as hello?query=list_airports (which is actually still open for further optimization, think of a REST webservice, but that's left up to you as an exercise).

URL url=new URL("127.0.0.1:8080/hello/hello?query=select * from airports")
is not a valid URL.

how to deploy a java applet for local development

I simply put the class file and the html file in the same directory. And I call the applet with this:
<p align="center">
<applet code="/ShowImage.class" width="200" height="200">
</applet>
</p>
Obviously this doesn't work. What is the most convenient way to setup local development?
edit:
My applet code:
import java.awt.image.BufferedImage;
import java.io.IOException;
import java.net.URL;
import javax.imageio.ImageIO;
/**
*
* #author PCKhoi
*/
public class ShowImage extends Applet {
private BufferedImage img;
public void init() {
try {
URL url = new URL(getCodeBase(), "what_I_think.jpg");
img = ImageIO.read(url);
} catch (IOException e) {
}
}
public void paint(Graphics g){
g.drawImage(img,20,20, null);
}
}

Try this
<HTML>
<HEAD>
</HEAD>
<BODY>
<APPLET ALIGN="CENTER" CODE="ShowImage.class" WIDTH="800" HEIGHT="500"></APPLET>
</BODY>
</HTML>
and please post your applet code

Some notes:
The code as published (at this instant) does not compile
Do not swallow exceptions in broken code
To compile and run..
prompt> javac ShowImage.java
prompt> appletviewer ShowImage.java
Code (note that the image name will need to be changed back).
//<applet code="ShowImage" width="200" height="200"></applet>
import java.applet.*;
import java.awt.*;
import java.awt.image.BufferedImage;
import javax.imageio.ImageIO;
import java.io.IOException;
import java.net.URL;
/**
* #author PCKhoi
*/
public class ShowImage extends Applet {
private BufferedImage img;
public void init() {
try {
URL url = new URL(getCodeBase(), "icon.png");
img = ImageIO.read(url);
} catch (IOException e) {
e.printStackTrace();
}
}
public void paint(Graphics g){
g.drawImage(img,20,20, null);
}
}
The important line of the source, in relation to your question, is the first, commented line. It supplies an HTML element that the Applet Viewer will parse and use as a pseudo-HTML.

I think I know why it doesn't load now. The applet was wrapped inside a complex netbeans project, that's why putting the class file and the html file inside the same directory didn't work.
My solution is to use a simple IDE such as DrJava if you don't need project functionality.