I want to extract all the text there is between all paragraphs on an unknown site (meaning i do not know the structure of the site).
So far i've got:
Elements paragraphEmail = doc.select("p");
Where doc = Jsoup.connect(url).get();
for (Element e : paragraphEmail) {
}
How to achieve this?
doc.select("p") will give you all the paragraph elements as a collection Elements.
Use a for each loop to get the text:
for(Element e : paragraphEmail){
System.out.println(e.text());
}
I suggest you take a look at the Jsoup cookbook and the API reference to get more familiar with the methods in Jsoup.
Cookbook
API Reference
Related
So I am using JSoup to scrape a website that creates a bunch of divs with dynamic class names (they change every reload), but the same attribute names. E.g:
<div class="[random text here that changes] js_resultTile" data-listing-number="[some number]">
<div class="a12_regularTile js_rollover_container " itemscope itemtype="http://schema.org/Product" data-listing-number="[same number here]">
<a href...
I've tried multiple approaches to selecting those divs and saving them in elements, but I can't seem to get it right. I've tried by attribute:
Document doc = Jsoup.connect([theUrl]).get();
Elements myEls = doc.select("div[data-listing-number]");
I've tried by class:
Document doc = Jsoup.connect([theUrl]).get();
Elements myEls = doc.getElementsByClass("a12_regularTile")
And:
Document doc = Jsoup.connect([theUrl]).get();
Elements myEls = doc.select("div[class*=js_resultTile]")
I've tried another attribute method:
Document doc = Jsoup.connect([theUrl]).get();
Elements myEls = new Elements();
for (Element element : doc.getAllElements() )
{
for ( Attribute attribute : element.attributes() )
{
if ( attribute.getKey().equalsIgnoreCase("data-listing-number"))
{
myEls.add(element);
}
}
}
None of these work. I can select the doc that gets me all the HTML, but my myEls object is always empty. What can I use to select these elements?
Are you sure these elements are present in HTML returned by server? They may be added later by JavaScript. If JavaScript is involved in page presentation then you won't be able to use Jsoup. More details in my answer to similar question here: JSoup: Difficulty extracting a single element
And one more tip. Instead of using your for-for-if construction you can use this:
for (Element element : doc.getAllElements()) {
if (element.dataset().containsKey("listing-number")) {
myEls.add(element);
}
}
I have just started to explore Jsoup and faced the following problem: when I'm trying to extract links from https://en.wikipedia.org/wiki/Knowledge that belong only to the English version of Wikipedia everything works correctly.
Document document = Jsoup.connect("https://en.wikipedia.org/wiki/Knowledge").timeout(6000).get();
Elements linksOnPage = document.select( "a[href^=\"/wiki/\"]");
for (Element link : linksOnPage) {
System.out.println("link : " + link.attr("abs:href"));
}
}
However I'm also getting the links that do not belong to the text of the current article such as:
link : https://en.wikipedia.org/wiki/Main_Page
link : https://en.wikipedia.org/wiki/Portal:Contents
link : https://en.wikipedia.org/wiki/Portal:Featured_content
link : https://en.wikipedia.org/wiki/Portal:Current_events
link : https://en.wikipedia.org/wiki/Special:Random
link : https://en.wikipedia.org/wiki/Help:Contents
link : https://en.wikipedia.org/wiki/Wikipedia:About
link : https://en.wikipedia.org/wiki/Wikipedia:Community_portal
What is the proper way to get only the links from the text leading to other Wikipedia articles with Jsoup?
links that I do not need are located in the div id="mw-panel"
Therefore the correct selector would be:
div:not(#mw-panel) a[href^="/wiki/"]
Which will select <a> elements that:
are not inside a <div> element with mw-panel ID
and their href attribute starts with "/wiki/".
EDIT:
I need only the links from an article without links from the side panels and without any links such as https://en.wikipedia.org/wiki/Special:BookSources/978-1-4200-5940-3 https://en.wikipedia.org/wiki/Special:BookSources/1-58450-460-9
Then you may try:
#bodyContent a[href^="/wiki/"]
This will parse links that:
are inside the article (<div> with ID of bodyContent)
their href attribute starts with "/wiki/"
div#bodyContent does not have "/wiki/...Special:..." links. (If you want to exclude links with some other word, append this to the end of the above selector without any space or separator: :not([href*="something"]))
You can also try to combine selectors to achieve the best pattern based on my tryings above and by reading about Jsoup selectors.
Example code:
String url = "https://en.wikipedia.org/wiki/Knowledge";
Document document = Jsoup.connect(url).timeout(6000).get();
Elements links = document.select("#bodyContent a[href^=\"/wiki/\"]");
for (Element e : links) {
System.out.println(e.attr("href"));
}
System.out.println("Links found: " + links.size());
This prints out following:
/wiki/Knowledge_(disambiguation)
/wiki/Fact
/wiki/Information
...
/wiki/Category:Articles_with_unsourced_statements_from_September_2007
/wiki/Category:Articles_with_unsourced_statements_from_May_2009
/wiki/Category:Wikipedia_articles_with_GND_identifiers
Links found: 826
I am working on a project where I have to be able to know that an element have repeated children .For example in that DOM, I want to know that the element tbody has similar children
My goal is to extract data- and store it in a database -from pages that I ignore their structure.
Use Jquery to get your td elements and iterate with each over them.
you can use JSOUP for this. its very easy to use as well
for example you want to get all td tag in within your document:
String html=... //your html string
Document doc = JSoup.parse(html);
Elements elements = doc.select("tbody").select("td");
System.out.println(elements.size()); //prints number of td within tbody REGARDLESS of where in the DOM tree they live.
Edit1:
to get all elements you can do:
for(Element e : doc.getAllElements){
System.out.println(e.getTagName());//prints the tag name
}
I am currently scraping pollen data from wunderground since their API accessor doesn't offer pollen data, specifically the values attributed to each day.
I've navigated the HTML using Chrome Dev Tools and found the specific line that I want. Using the documentation offered by JSoup, I tried putting in my own custom CSS Selectors, but I am quite lost.
I was wondering if anyone would give me some insight on how to access that particular element.
For example, below is an example of what I have so far.
doc = Jsoup.connect("http://www.wunderground.com/DisplayPollen.asp?Zipcode=19104").get();
Element title = doc.getElementById("td");
Element tagName = doc.tagName("id");
System.out.println(tagName);
You don't want to use doc.getElementById("td") because <td> is not id attribute, but tag (also getElementById doesn't support CSS query).
What you want is to select first <td> with class levels. You can do it via
Element tag = doc.select("td.levels").first();
Also to get only text which will be generated with this tag (and not entire HTML) use text() method like
System.out.println(tag.text());
Document doc = Jsoup.connect("http://www.wunderground.com/DisplayPollen.asp?Zipcode=19104").get();
Elements days = doc.select("table.pollen-table").first().select("td.even-four");
for (Element day : days) {
System.out.println(day.text());
}
Elements levels = doc.select("td.levels");
for (Element level : levels) {
System.out.println(level.text());
}
I’m working with JSoup to parse the html website.
I want to get the article from (for example) Wikipedia.
I would like to get the text from the main page (http://en.wikipedia.org/wiki/Main_Page) from the table “From today’s featured article”.
Here’s the code:
Document doc = Jsoup.connect("http://en.wikipedia.org/wiki/Main_Page”);
Elements el = doc.select("div.mp-tfa”);
System.out.println(el);
The problem is that it doesn’t work properly - it prints out just a blank line.
The “From today’s featured article” table is inserted in div class=“mp-tfa”.
How to get this text in my java program?
Thanks in advance.
Change:
doc.select("div.mp-tfa");
To:
doc.select("div#mp-tfa");
The better way would to iterate over the Elements thus retrieved for the tag, class or Element of your choice, simply put:
Document doc = Jsoup.connect("http://en.wikipedia.org/wiki/Main_Page").get();
Elements el = doc.select("div#mp-tfa");
for (Element e : el) {
System.out.println(e.text());
}
Would give:
The Boulonnais is a heavy draft horse breed from Fr....
I think it's supposed to be:
Document doc = Jsoup.connect("http://en.wikipedia.org/wiki/Main_Page").get();
Elements el = doc.select("div#mp-tfa");
System.out.println(el);