Perhaps I'm doing something wrong, but I'm trying to parse this page using jsoup, it for some reason it doesn't find me the div I'm looking for
doc = Jsoup.connect(params[0]).get();
content = doc.select("div.itemcontent").first().text();
Where am I going wrong here?
Thanks
The problem: you get a different website using jsoup than using a browser. I set another useragent in Jsoup, but no luck. Possible the content is changed through JavaScript?!
However, you can change the selector according to the webseite you get.
It's always a good idea to take a look into document as it's parsed - a simple System.out.println(doc) is enough.
Here are some steps you can try:
Print your Document doc (eg. using System.out)
Search for the required value(s) in there
Select those tags instead
I just played around a bit, but maybe you can use this snipped:
content = doc.select("description").first().text();
It seems to me, <description>...</description> is what you're looking for.
Related
Basically, I need a table with all the possible books that exist, and I don't wanna do that, because I'm a very lazy person xD. So, my question is.. can I use a site, that I have in mind, and just like cut off the rest this site(that I don't need) and leave only the search part(maybe do some kind of changes in layout)... then, make the search, find the book and store in my database only the data that make sense for me. Is that possible? I heard that JSOUP could help.
So, I just want some tips. (thx for reading).
the site: http://www.isbn.bn.br/website/consulta/cadastro
Yes, you can do that using Jsoup, the main problem is that the URL you shared uses JavaScript so you'll need to use Selenium to force the JS execution or you can also get the book URL and parse it.
The way to parse a web using Jsoup is:
Document document = Jsoup.connect("YOUR-URL-GOES-HERE")
.userAgent("Mozilla/5.0")
.get();
The you retrieve the whole HTML in a Document so you can get any Element contained in the Element using CSS Selectors, for example, if in the HTML you want to retrieve the title of the web, you can use:
Elements elements = document.select("title");
And that for every HTML tag that you want to retrieve information from. You can check the Jsoup Doc an check some of the examples explained: Jsoup
I hope it helps you!
I have written a Jsoup class file to scrape a page and grab the hrefs for every element on the page. What I would like to do from there is to extract the Xpath for each of the elements from their hrefs.
Is there a way to do this in JSoup? If not is what is the best way to do this in Java (and are there any resources on this)?
Update
I want to clarify my question.
I want to scan a page for all the href identifiers and grab the links (that part is done). For my script, I need to get the xpath of all the elements I have identified and scraped from the (scanned) page.
The problem is that I assumed I could easily translate the href links to Xpath.
The comment from #Rishal dev Singh ended up being the right answer.
Check his link here:
http://stackoverflow.com/questions/7085539/does-jsoup-support-xpath
I was having some practices with programming, and I got stuck (also because of my lacking knowledge of web programming) in this part: I was to get some information from this page: http://db.fowtcg.us/index.php?p=card&code=VS01-003+R , but only the card properties, and I'm struggling a little with JSoup, I was able to fetch the data with:
Document doc = Jsoup.connect("http://db.fowtcg.us/?p=card&code=TTW-080+SR").get();
Elements newsHeadlines = doc.select("div.card-props");
System.out.println(newsHeadlines);
But I couldn't get the data back from the Element object (but i could see it was there debugging).
How can i proceed in order to fetch this information?
Here, use this instead:
Elements property = doc.select("div.col-xs-12.col-sm-7.box.card-props");
You need to make sure the selector you use match the original html document exactly.
You can use contains/ends-with selector also
//contains
Elements property = doc.select("div[class*=card-props]");
//ends-with
Elements property = doc.select("div[class$=card-props]");
Go through below link to know more about css selectors.
http://jsoup.org/cookbook/extracting-data/selector-syntax
I am trying to read an element with ID main_LbStatsOpenProfit from http://www.zulutrade.com/trader/104769 but it is always empty. I have tested it on try Jsoup http://try.jsoup.org/ and it works fine.
I have read many other values with ID and those ones worked fine.
Any Ideas why this isolated problem might occur?
or any other parser example that can read this?
You might be using html() on the selected Element? If so it won't work as the Element in the html of the given url is empty.
Try the below code :
doc.select("#main_LbStatsOpenProfit").first().toString()
Please deal with this trivial question. It is available in bits and pieces on stackoverflow.
I have HTML dump of a website in the form of String. I want to extract text from the specific tags of it.
In other way, I want to mimic
Document doc = Jsoup.connect(url).userAgent("Mozilla").get();
Elements links = doc.getElementsByTag("cite");
I am not using Jsoup because I don't want it to connect to the website (I have another service for that which returns html dump in the form of text). I found HTMLEditorKit for converting text to HTMLDocument but it doesn't seem to be very easy to use(like Jsoup or HTMLParser) or I am unable to get it.
Any help would be useful.
Thanks.
If you have used Jsoup and it worked yet, you should continue using it.
Document doc = Jsoup.parse("<html>...");
should do.
see: The API