Trying to practice extracting data from tables using JSoup. Can't figure out why I can't pull the "Shares Outstanding" field from
https://finance.yahoo.com/q/ks?s=AAPL+Key+Statistics
Here's two attempts where 's' is AAPL:
public class YahooStatistics {
String sharesOutstanding = "Shares Outstanding:";
public YahooStatistics(String s) {
String keyStatisticsURL = ("https://finance.yahoo.com/q/ks?s="+s+"+Key+Statistics");
//Attempt 1
try {
Document doc = Jsoup.connect(keyStatisticsURL).get();
for (Element table : doc.select("table.yfnc_datamodoutline1")) {
for (Element row : table.select("tr")) {
Elements tds = row.select("td");
for (Element td : tds.select(sharesOutstanding)) {
System.out.println(td.ownText());
}
}
}
}
catch (IOException ex) {
ex.printStackTrace();
}
//Attempt 2
try {
Document doc = Jsoup.connect(keyStatisticsURL).get();
for (Element table : doc.select("table.yfnc_datamodoutline1")) {
for (Element row : table.select("tr")) {
Elements tds = row.select("td");
for (int j = 0; j < tds.size() - 1; j++) {
Element td = tds.get(j);
if ((td.ownText()).equals(sharesOutstanding)) {
System.out.println(tds.get(j+1).ownText());
}
}
}
}
}
catch(IOException ex) {
ex.printStackTrace();
}
The attempts return: BUILD SUCCESSFUL and nothing else.
I've disabled JavaScript on my browser and the table still shows, so I'm assuming this is not written in JavaScript but HTML.
Any suggestions are appreciated.
Notes about your source after the edit:
You should compare ownText() rather than text(). text() gives you the combined text of all the element and all its sub-elements. In this case the element contains Shares Outstanding<font size="-1"><sup>5</sup></font>:, so its combined text is "Shares Outstanding5:". If you use ownText it will just be "Shares Outstanding:".
Note the colon (:). Update the value in sharesOutstanding accordingly.
You are passing it the wrong URL. There should be a + following the AAPL.
Your current query (at least the second attempt) is returning the element twice, because there is a nested table so it finds the TDs twice.
You can either break from your loops once you found a match, go back to your original version (with corrections as above) - see note - or you can try using a more sophisticated query which will only match once:
Elements elems = doc.select("td.yfnc_tablehead1:containsOwn("+sharesOutstanding+") + td.yfnc_tabledata1");
if ( ! elems.isEmpty() ) {
System.out.println( elems.get(0).owntext() );
}
This selector gives you all the td elements whose class is yfnc_tabledata1, whose immediate preceding sibling is a td element whose class is yfnc_tablehead1 and whose own text contains the "Shares Outstanding:" string. This should basically select the exact TD you need.
Note: the previous version of this answer was a long rattle about the difference between Elements.select() and Element.select(). It turns out that I was dead wrong and your original version should have worked - if you had corrected the four points above. So to set the record straight: select() on an Elements actually does look inside each element and the resulting list may contain descendents of any of the elements in the original list that match the selection. Sorry about that.
Related
I am trying to get text between tags and save into some variable, for example:
Here I want to save value return which is between em tags. Also I need the rest of the text which is in p tags,
em tag value is assigned with return and
p tag value should return only --> an item, cancel an order, print a receipt, track your purchases or reorder items.
if some value is before em tag, even that value should be in different variable basically one p if it has multiple tags within then it should be split and save into different variables. If I know how can I get rest of text which are not in inner tags I can retrieve the rest.
I have written below: the below is returning just "return" which is in "'em' tags.
Here ep is basically doc.select(p), selecting p tag and then iterating, not sure if I am doing right way, any other approaches are highly appreciated.
String text ="\<p><em>return </em>an item, cancel an order, print a receipt, track your purchases or reorder items.</p>"
Elements italic_tags = ep.select("em");
for(Element em:italic_tags) {
if(em.tagName().equals("em")) {
System.out.println( em.select("em").text());
}
}
If you need to select each sub text and text enclosed by different tags you need to try selecting Node instead of Element. I modified your HTML to include more tags so the example is more complete:
String text = "<p><em>return </em>an item, <em>cancel</em> an order, <em>print</em> a receipt, <em>track</em> your purchases or reorder items.</p>";
Document doc = Jsoup.parse(text);
Element ep = doc.selectFirst("p");
List<Node> childNodes = ep.childNodes();
for (Node node : childNodes) {
if (node instanceof TextNode) {
// if it's a text, just display it
System.out.println(node);
} else {
// if it's another element, then display its first
// child which in this case is a text
System.out.println(node.childNode(0));
}
}
output:
return
an item,
cancel
an order,
print
a receipt,
track
your purchases or reorder items.
I was having trouble getting just the href from a rows of table data. Although I was able to get it working, I am wondering if anyone has an explanation for why my code here works.
for (Element element : result.select("tr")) {
if (element.select("tr.header.left").isEmpty()) {
Elements tds = element.select("td");
//The line below is what I don't understand
String link = tds.get(0).getElementsByAttribute("href").first().attr("href");
String position = tds.get(1).text();
}
}
The line that I was using before, that did not work is below:
String link = tds.get(0).attr("href");
Why does this line return an empty string? I'm assuming it has to do with how I am iterating through the elements as I've selected by "tr". However, I'm not familiar with how Elements vs Element are structured.
Thanks for your help!
Elements is simply an ArrayList<Element>
The reason you're having to write that extra code is because <td> doesn't have an href attribute, so tds.get(0).attr("href"); won't work. You're presumably trying to capture the href from an <a> within the cell. The longer, working code is saying:
For the first cell in the row, get the first element with an #href attribute (i.e. a link), and get
its #href attribute
Try the following example (with example document) to show how to access the child links more clearly:
Element result = Jsoup.parse("<html><body><table><tr><td><a href=\"http://a.com\" /</td><td>Label1</td></tr><tr><td><a href=\"http://b.com\" /></td><td>Label2</td></tr></table></body></html>");
for (Element element : result.select("tr")) {
if (element.select("tr.header.left").isEmpty()) {
Elements tds = element.select("td");
String link = tds.get(0).getElementsByTag("a").attr("href");
String position = tds.get(1).text();
System.out.println(link + ", " + position);
}
}
If you have a HTML page stored in String ArrayList, and you want to for example read the whole <div> tag of certain class type, how do you read the next lines so that it would reach the end of div tag?
for (String l : line) {
if (l.contains("<div class=\"somne_class\">"){
//read the next n strings in ArrayList until </div> tag is reached
}
Generally, it's bad idea to store HTML file as list of raw strings. Why do you store it in such way?
Imagine you have string like <div id="outer_div"><div id="inner_div">Hei!</div></div>. Here you have multiple nested HTML tags in a single line, so you won't easily get the closing tag.
Consider using HTML parser, then you can get desired tag(s) by type or attribute. There are plenty of HTML parsers implemented in Java. One of the most popular is jsoup.
I agree with Vladimir, you're probably looking for an HTML parser.
To answer the exact question in the post: to simply find the next </div> tag, you can use a for loop instead of a foreach loop.
for (int i = 0; i < line.size(); ++i) {
String l = line.get(i);
if (l.contains("<div class=\"somne_class\">") {
for (int j = i; j < line.size(); ++j) {
String l2 = line.get(j);
if (l2.contains("</div>")) {
// l2 is the next line that contains a </div> tag
}
}
}
}
Note that this might not be the matching closing tag for the opening tag, even if you assume that every tag is in a different line.
I recommend you to use jsoup
It is nice for parsing an writing html file.Althought i hadn't
yet digged to much on it here is an example of taking all the elements
with tag div:
Document htmlFile = null;
// Read the html file
try {
htmlFile = Jsoup.parse(new File("path"),"UTF-8");//path,encoding
} catch (IOException e) {
e.printStackTrace();
}
Elements images = htmlFile.getElementsByTag("div");
You can do much more read here
I have managed to successfully grab the href links using JSoup. I have also managed to grab the relative value and absolute value of a href for a single link. As shown below:
//works perfectly, website: bbc.co.uk
Document document = Jsoup.connect(url).get();
Element link = document.select("a").last();
String relHref = testlink.attr("href");
String absHref = testlink.attr("abs:href");
System.out.println(relHref);
System.out.println(absHref);
//output:
relHref: /help/web/links/
absHref: http://www.bbc.co.uk/help/web/links/
I can even use Element link = document.select("a").first(); and this also works. However, when I try and add this in a loop to iterate through all of the grabbed links and print out each link, it doesn't give me the expected results. Here is my code:
//not working
Elements links = document.select("a");
for(int i=0; i<links.size(); i++){
String relHref = links.attr("href");
String absHref = links.attr("abs:href");
System.out.println(relHref);
System.out.println(absHref);
}
//output
http://m.bbc.co.uk
http://m.bbc.co.uk
http://m.bbc.co.uk
....
I know the links array of type Elements has the correct data, and if I try and print the elements in the links array it displays all of the href tags i.e.
for (Element link : links) {
System.out.println(link);
}
//output 116 links:
mobile site
<img src="http://static.bbci.co.uk/frameworks/barlesque/2.72.5/orb/4/img/bbc-blocks-dark.png" width="84" height="24" alt="BBC">
Skip to content
<a id="orb-accessibility-help" href="/accessibility/">Accessibility Help</a>
....
But how do I get the relHref and absHref for an array to work? Instead my code just prints out the first link over and over again. I've been going at this for hours, so I'm probably making a silly mistake somewhere but help is appreciated!
Thanks.
On this line:
String relHref = links.attr("href");
...how is it supposed to know you're talking about the ith link? (It doesn't: Elements#attr always returns the value for the first entry in the Elements collection.)
You want
String relHref = links.get(i).attr("href");
...which gets the specific link you're interested in via Elements#get, then uses Node#attr on it.
That said, though, I would just use the enhanced for loop:
for (Element link : document.select("a")) {
String relHref = link.attr("href");
String absHref = link.attr("abs:href");
System.out.println(relHref);
System.out.println(absHref);
}
...unless you need i for something.
You need to use the Elements method, get(int index) inside of your for loop to get each Element held by your Elements.
e.g.,
Elements links = document.select("a");
for(int i=0; i < links.size(); i++) {
Element ele = links.get(i);
/// use ele here to extract info from each Element
}
I really tried a lot and also searched a lot of websites...
I tried to parse a price from a website with jsoup, but it didn't work.
What I tried out is this:
try {
String str1 = "https://www.google.de/shopping/product/3996339592576509511?hl=de&q=4250155834791&oq=4250155834791&gs_l=products-cc.3...4306.7625.0.8037.13.6.0.7.0.0.60.314.6.6.0...0.0...1ac.1.LgJKDfZQvls&sa=X&ei=eeqlUY2zFNT54QSyloCoDw&ved=0CFIQgggwAA&prds=scoring:p";
doc = Jsoup.connect(str3).get();
final Elements elements = doc.select("td:lt(1)");
String price = doc.select("span").first().text();
System.out.println(price);
System.out.println("Ende");
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
The goal should be to extract the lowest price of the product.
example-page:
https://www.google.de/shopping/product/3996339592576509511?hl=de&q=4250155834791&oq=4250155834791&gs_l=products-cc.3...4306.7625.0.8037.13.6.0.7.0.0.60.314.6.6.0...0.0...1ac.1.LgJKDfZQvls&sa=X&ei=eeqlUY2zFNT54QSyloCoDw&ved=0CFIQgggwAA&prds=scoring:p
I would like to parse the first row that shows me the results.
in this case: ebay 24-trade365.
I need the article's price and the link to the vendor.
Can anyone help, please?
You'd do better extracting by class, also when you select for "span" you are selecting in the original doc, not the elements you have extracted. Try something like:
// get all column entries for price
final Elements elements = doc.getElementsByClass("os-price-col");
int lowest_price = Integer.MAX_VALUE;
// foreach entry
for(Element element : elements){
// get the price in text form
String text_price = element.text();
// convert to an integer
text_price = text_price.replaceAll("[^0-9.]", "");
int price = Integer.parseInt(text_price);
// check if it's the lowest
if(price < lowest_price) lowest_price = price;
}
System.out.println(lowest_price);
Obviously updating slightly to get the output in a format you want.
EDIT: Just saw that you wanted the vendor link as well. In this case I would extract a row at a time i.e.
Elements rows = doc.getElementsByClass("os-row");
Then iterate through each row and pick out the price as before, but this time do
row.getElementsByClass("os-price-col").first();
And if it's the lowest you can pick out the vendor url with something like
row.getElementsByClass("os-seller-name").first().select("a").attr("href");
If your table is already sorted and all you want is the first row:
Element table=doc.getElementsByClass("os-main-table").first();
Element firstRow=table.select("tr[class=os-row").first();
Element seller=firstRow.select("td[class=os-seller-name]").first();
String sellerName=seller.text().trim();
String sellerLink=seller.getElementsByTag("a").first().attr("href");
String price=firstRow.select("td[class=os-price-col").first().getElementsByClass("os-base_price").text();
You can find a tutorial on Jsoup navigation at http://jsoup.org/cookbook/extracting-data/dom-navigation