I want to generate xPath from html file. So far, I have been succeded to store Html source in a String and generating basic xpath using matcher regex as follows:-
String text = "<html><body><table><tr id=\"x\"><td>abc</td><td></td><td>xyz</td></tr></table></body></html>";
//I want xpath till label "xyz"
String unwanted= "xyz";
//so splitting and storing needed String
String[] neededString=text.split(unwanted);
String a="";
//pattern for extracting tags
String patternString1 = "<(.+?)>";
Pattern pattern = Pattern.compile(patternString1);
Matcher matcher = pattern.matcher(neededString[0]);
while(matcher.find()) {
a=a.concat(matcher.group(1)+"/");
System.out.println(a);
}
This code works for basic tag Structure without multiple child nodes like multiple <td>'s in <tr>. Can anyone improve my above code to include xpath generation for multiple childs and also for capturing attrributes like Ids,Class etc.
Any help is much appreciated.
Thanks in advance.
Regex is not so Accurate for Extracting the Html content.
Use Jsoup Html Parser
public static void main(String[] args){
String html = "<html><body><table><tr id=\"x\"><td>abc</td><td></td>" +
"<td>xyz</td></tr></table></body></html>";
Document doc = Jsoup.parse(html);
for (Element table : doc.select("table")) {
for (Element row : table.select("tr[id=x]")) {
Elements tds = row.select("td)");
System.out.println(tds.get(2).text());
}
}
}
Related
I want to get the String between (not including): alt=" and "
Here is a small sample of my code:
Pattern p2 = compile("alt=\"(.*?)\");
Matcher m2 = p2.matcher(result);
while (m2.find()) {
names.add(m2.group());
}
The output is for example: alt="Harry Potter"
when I want the output to be just: Harry Potter
Your code has a typo (a missing double quote in compile) and the group you need to access is Group 1 (use compile("alt=\"(.*?)\"") and m2.group(1)).
You should think about using an HTML parser for getting values from HTML, like jsoup. Here is a way to get what you need with it:
Document doc = Jsoup.parse(html_contents);
for (Element element : doc.getAllElements())
{
for (Attribute attribute : element.attributes())
{
if(attribute.getKey().equalsIgnoreCase("alt"))
{
names.add(attribute.getValue());
}
}
}
<b>Topic1</b><ul>asdasd</ul><br/><b>Topic2</b><ul>....
I want to extract everything that comes after <b>Topic1</b> and the next <b> starting tag. Which in this case would be: <ul>asdasd</ul><br/>.
Problem: it must not necessairly be the <b> tag, but could be any other repeating tag.
So my question is: how can I dynamically extract those text? The only static thinks are:
The signal keyword to look for is always "Topic1". I'd like to take the surrounding tags as the one to look for.
The tag is always repeated. In this case it's always <b>, it might as well be <i> or <strong> or <h1> etc.
I know how to write the java code, but what would the regex be like?
String regex = ">Topic1<";
Matcher m = Pattern.compile(regex).matcher(text);
while (m.find()) {
for (int i = 1; i <= m.groupCount(); i++) {
System.out.println(m.group(i));
}
}
The following should work
Topic1</(.+?)>(.*?)<\\1>
Input: <b>Topic1</b><ul>asdasd</ul><br/><b>Topic2</b><ul>
Output: <ul>asdasd</ul><br/>
Code:
Pattern p = Pattern.compile("Topic1</(.+?)>(.*?)<\\1>");
// get a matcher object
Matcher m = p.matcher("<b>Topic1</b><ul>asdasd</ul><br/><b>Topic2</b><ul>");
while(m.find()) {
System.out.println(m.group(2)); // <ul>asdasd</ul><br/>
}
Try this
String pattern = "\\<.*?\\>Topic1\\<.*?\\>"; // this will see the tag no matter what tag it is
String text = "<b>Topic1</b><ul>asdasd</ul><br/><b>Topic2</b>"; // your string to be split
String[] attributes = text.split(pattern);
for(String atr : attributes)
{
System.out.println(atr);
}
Will print out:
<ul>asdasd</ul><br/><b>Topic2</b>
I am reading the content from a web page and then I am parsing it with the help of Jsoup parser to get only the hyperlinks that exists in the body section. I am getting the output as:
<font color="#0000FF">Sports</font>
<font color="#0000FF">Titanic</font>
license plates
miracle cars
Clear
and even more hyperlinks.
From all of them, all I am interested in is data like
/sports/sports.asp
/titanic/titanic.asp
gastheft.asp
miracle.asp
/crime/warnings/clear.asp
How can I do this using Strings or is there any other way or method to extract this information usinf Jsoup Parser itself?
You can try this, its works.
public class AttributeParsing {
/**
* #param args
*/
public static void main(String[] args) {
final String html = "<font color=\"#0000FF\">Sports</font>";
Document doc = Jsoup.parse(html, "", Parser.xmlParser());
Element th = doc.select("a[href]").first();
String href = th.attr("href");
System.out.println(th);
System.out.println(href);
}
}
Output :
th : <font color="#0000FF">Sports</font>
href : /sports/sports.asp
Try this it may help
String html = "<p>An <a href='http://example.com/'><b>example</b></a> link.</p>";
Document doc = Jsoup.parse(html);
Element link = doc.select("a").first();
String text = doc.body().text(); // "An example link"
String linkHref = link.attr("href"); // "http://example.com/"
String nextIndex = linkHref .indexOf ("\"", linkHref );
This should be a basic bit of parsign using
String.indexOf
as in
index = jsoupOutput.indexOf ("href=\"");
and
nextIndex = jsoupOutput.indexOf ("\"", index);
with the necessary checks in place.
Let's assume that String anchor contains one of these links then the beginning index of the substring will after href=" and the end index will be the first quotation mark after index 9 this way:
String anchor = "<font color=\"#0000FF\">Sports</font>";
int beginIndex = anchor.indexOf("href=\"") + 6; //To start after <a href="
int endIndex = anchor.indexOf("\"", beginIndex);
String desiredPart = anchor.substring(beginIndex, endIndex);
And that's it if the shape of the anchor is going to always be that way.. better options are using regular expressions and best would be using an XML parser.
Use this as reference
import java.util.regex.*;
public class HelloWorld{
public static void main(String []args){
String s = "<font color=\"#0000FF\">Sports</font>"+
"<font color=\"#0000FF\">Titanic</font>"+
"license plates"+
"miracle cars"+
"Clear";
Pattern p = Pattern.compile("href=\".+?\"");
Matcher m = p.matcher(s);
while(m.find())
{
System.out.println(m.group().split("=")[1].replace("\"",""));
}
}
}
Output
/sports/sports.asp
/titanic/titanic.asp
gastheft.asp
miracle.asp
/crime/warnings/clear.asp
You can do it in one line:
String[] paths = str.replaceAll("(?m)^.*?\"(.*?)\".*?$", "$1").split("(?ms)$.*?^");
The first method call removes everything except the target from each line, and the second splits on newlines (will work on all OS terminators).
FYI (?m) turns on "multiline mode" and (?ms) also turns on the "dotall" flag.
What regular expression can be used to extract the value of src attribute in the iframe tag?
If you really are using Java (not JavaScript) and you only have the iframe, you can try the regular expression:
(?<=src=")[^"]*(?<!")
e.g.:
private static final Pattern REGEX_PATTERN =
Pattern.compile("(?<=src=\")[^\"]*(?<!\")");
public static void main(String[] args) {
String input = "<iframe name=\"I1\" id=\"I1\" marginwidth=\"1\" marginheight=\"1\" height=\"430px\" width=\"100%\" border=\"0\" frameborder=\"0\" scrolling=\"no\" src=\"report.htm?view=country=us\">";
System.out.println(
REGEX_PATTERN.matcher(input).matches()
); // prints "false"
Matcher matcher = REGEX_PATTERN.matcher(input);
while (matcher.find()) {
System.out.println(matcher.group());
}
}
Output:
report.htm?view=country=us
I would say look into dom parsing. from there it would be extremely similar to the javascript answer.
Dom parser will turn the html into a document from there you can do:
iframe = document.getElementById("I1");
src = iframe.getAttribute("src");
Regex is little bit costlier do not use it until you have other simple solution, in java try this
String src="<iframe name='I1' id='I1' marginwidth='1' marginheight='1'" +
" height='430px' width='100%' border='0' frameborder='0' scrolling='no'" +
" src='report.htm?view=country=us'>";
int position1 = src.indexOf("src") + 5;
System.out.println(position1);
int position2 = src.indexOf("\'", position1);
System.out.println(position2);
System.out.println(src.substring(position1, position2));
Output:
134
160
report.htm?view=country=us
In case you meant javascript instead of java:
var iframe = document.getElementById("I1");
var src = iframe.getAttribute("src");
alert(src); //outputs the value of the src attribute
src="(.*?)"
The regular expression will match src="report.htm?view=country=us", but you will find only the part between the " in the first (and only) submatch.
When you only want to match src-attributes when they are in an iframe, do this:
<iframe.*?src="(.*?)".*?>
but there are certain corner-cases where this could fail due to the inherently non-regular nature of HTML. See the top answer to RegEx match open tags except XHTML self-contained tags for an amusing rant about this problem.
I have this string containing a large chunk of html and am trying to extract the link from href="..." portion of the string. The href could be in one of the following forms:
<a href="..." />
<a class="..." href="..." />
I don't really have a problem with regex but for some reason when I use the following code:
String innerHTML = getHTML();
Pattern p = Pattern.compile("href=\"(.*)\"", Pattern.DOTALL);
Matcher m = p.matcher(innerHTML);
if (m.find()) {
// Get all groups for this match
for (int i=0; i<=m.groupCount(); i++) {
String groupStr = m.group(i);
System.out.println(groupStr);
}
}
Can someone tell me what is wrong with my code? I did this stuff in php but in Java I am somehow doing something wrong... What is happening is that it prints the whole html string whenever I try to print it...
EDIT: Just so that everyone knows what kind of a string I am dealing with:
<a class="Wrap" href="item.php?id=43241"><input type="button">
<span class="chevron"></span>
</a>
<div class="menu"></div>
Everytime I run the code, it prints the whole string... That's the problem...
And about using jTidy... I'm on it but it would be interesting to know what went wrong in this case as well...
.*
This is an greedy operation that will take any character including the quotes.
Try something like:
"href=\"([^\"]*)\""
There are two problems with the code you've posted:
Firstly the .* in your regular expression is greedy. This will cause it to match all characters until the last " character that can be found. You can make this match be non-greedy by changing this to .*?.
Secondly, to pick up all the matches, you need to keep iterating with Matcher.find rather than looking for groups. Groups give you access to each parenthesized section of the regex. You however, are looking for each time the whole regular expression matches.
Putting these together gives you the following code which should do what you need:
Pattern p = Pattern.compile("href=\"(.*?)\"", Pattern.DOTALL);
Matcher m = p.matcher(innerHTML);
while (m.find())
{
System.out.println(m.group(1));
}
Regex is great but not the right tool for this particular purpose. Normally you want to use a stackbased parser for this. Have a look at Java HTML parser API's like jTidy.
Use a built in parser. Something like:
EditorKit kit = new HTMLEditorKit();
HTMLDocument doc = (HTMLDocument)kit.createDefaultDocument();
doc.putProperty("IgnoreCharsetDirective", Boolean.TRUE);
kit.read(reader, doc, 0);
HTMLDocument.Iterator it = doc.getIterator(HTML.Tag.A);
while (it.isValid())
{
SimpleAttributeSet s = (SimpleAttributeSet)it.getAttributes();
String href = (String)s.getAttribute(HTML.Attribute.HREF);
System.out.println( href );
it.next();
}
Or use the ParserCallback:
import java.io.*;
import java.net.*;
import javax.swing.text.*;
import javax.swing.text.html.parser.*;
import javax.swing.text.html.*;
public class ParserCallbackText extends HTMLEditorKit.ParserCallback
{
public void handleStartTag(HTML.Tag tag, MutableAttributeSet a, int pos)
{
if (tag.equals(HTML.Tag.A))
{
String href = (String)a.getAttribute(HTML.Attribute.HREF);
System.out.println(href);
}
}
public static void main(String[] args)
throws Exception
{
Reader reader = getReader(args[0]);
ParserCallbackText parser = new ParserCallbackText();
new ParserDelegator().parse(reader, parser, true);
}
static Reader getReader(String uri)
throws IOException
{
// Retrieve from Internet.
if (uri.startsWith("http:"))
{
URLConnection conn = new URL(uri).openConnection();
return new InputStreamReader(conn.getInputStream());
}
// Retrieve from file.
else
{
return new FileReader(uri);
}
}
}
The Reader could be a StringReader.
Another easy and reliable way to do it is by using Jsoup
Document doc = Jsoup.connect("http://example.com/").get();
Elements links = doc.select("a[href]");
for (Element link : links){
System.out.println(link.attr("abs:href"));
}
you may use a html parser library. jtidy for example gives you a DOM model of the html, from wich you can extract all "a" elements and read their "href" attribute
"href=\"(.*?)\"" should also work, but I think Kugel's answer will work faster.