Develop Reference

JSoup not able to get links from html

I'm trying to get links from html of a site but unable to do so using Jsoup.
This is the HTML:
<div class="anime_muti_link">
<ul>
<li><div class="doamin">Domain</div><div class="link">Link</div></li>
<li class="anime">
Server m1</div><span>Watch This Link</span>
</li>
<li class="anime">
Server m2</div><span>Watch This Link</span>
</li>
<li class="xstreamcdn">
Xstreamcdn</div><span>Watch This Link</span>
</li>
<li class="mixdrop">
<div class="server mixdrop">Mixdrop</div><span>Watch This Link</span>
</li>
<li class="streamsb">
StreamSB</div><span>Watch This Link</span>
</li>
<li class="doodstream">
Doodstream</div><span>Watch This Link</span>
</li>
</ul>
</div>
This is the android code that I wrote which doesn't seem to work:
try {
Document doc = Jsoup.connect(URL).get();
Elements content = doc.getElementsByClass("anime_muti_link");
Elements links = content.select("a");
String[] urls = new String[links.size()];
for (int i = 0; i < links.size(); i++) {
urls[i] = links.get(i).attr("data-video");
if (!urls[i].startsWith("https://")) {
urls[i] = "https:" + urls[i];
}
}
arrayList.addAll(Arrays.asList(urls));
Log.d("CALLING_URL", "Links: " + Arrays.toString(urls));
} catch (IOException e) {
e.getMessage();
}
Can someone please help me with this? Thanks
Edit: Basically I'm trying to get those 6 links and add them to my list to use it within the app.
Edit 2:
So I found another HTML that can seems better:
<div class="heading-servers">
<span><i class="fa fa-signal"></i> Servers</span>
<ul class="servers">
<li data-vs="https://example.com" class="server server-active" style="display: block;" onclick="return loadIframe('ifrm', this.getAttribute('data-vs'));">Netu</li>
<li data-vs="https://example.com" class="server" style="display: block;" onclick="return loadIframe('ifrm', this.getAttribute('data-vs'));">VideoVard</li>
<li data-vs="https://example.com" class="server" style="display: block;" onclick="return loadIframe('ifrm', this.getAttribute('data-vs'));">Doodstream</li>
<li data-vs="https://example.com" class="server" style="display: block;" onclick="return loadIframe('ifrm', this.getAttribute('data-vs'));">Okstream</li>
</ul>
</div>

As you can see, in this li definition you are including a nested div:
<li class="xstreamcdn">
Xstreamcdn</div><span>Watch This Link</span>
</li>
This is causing that the variable content, the HTML fragment with class anime_muti_link, to look like:
<div class="anime_muti_link">
<ul>
<li>
<div class="doamin">
Domain
</div>
<div class="link">
Link
</div></li>
<li class="anime"> <a href="#" class="active" rel="1" data-video="example.com">
<div class="server m1">
Server m1
</div><span>Watch This Link</span></a> </li>
<li class="anime"> <a href="#" rel="1" data-video="example.com">
<div class="server m1">
Server m2
</div><span>Watch This Link</span></a> </li>
<li class="xstreamcdn"> Xstreamcdn</li>
</ul>
</div>
A similar result will be obtained even if you tidy your HTML. I used this code from one of my previous answers:
Tidy tidy = new Tidy();
tidy.setXHTML(true);
tidy.setIndentContent(true);
tidy.setPrintBodyOnly(true);
tidy.setInputEncoding("UTF-8");
tidy.setOutputEncoding("UTF-8");
tidy.setSmartIndent(true);
tidy.setShowWarnings(false);
tidy.setQuiet(true);
tidy.setTidyMark(false);
org.w3c.dom.Document htmlDOM = tidy.parseDOM(new ByteArrayInputStream(html.getBytes()), null);
OutputStream out = new ByteArrayOutputStream();
tidy.pprint(htmlDOM, out);
String tidiedHtml = out.toString();
// System.out.println(tidiedHtml);
Document document = Jsoup.parse(tidiedHtml);
Elements content = document.getElementsByClass("anime_muti_link");
System.out.println(content);
And this is why you are finding only three anchors.
Please, try correcting your HTML or selecting the anchor tag as the document level instead:
Document document = Jsoup.parse(html);
// Elements content = document.getElementsByClass("anime_muti_link");
// System.out.println(content);
Elements links = document.select("a");
String[] urls = new String[links.size()];
for (int i = 0; i < links.size(); i++) {
urls[i] = links.get(i).attr("data-video");
if (!urls[i].startsWith("https://")) {
urls[i] = "https://" + urls[i];
}
}
System.out.println(Arrays.asList(urls));
If the result obtained contains undesired links, perhaps you can try narrowing the selector used, something like:
document.select(".anime_muti_link a")
If this doesn't work, another possible alternative could be selecting the anchor elements with a data-video attribute, a[data-video]:
Document document = Jsoup.parse(html);
Elements videoLinks = document.select("a[data-video]");
String[] urls = new String[videoLinks.size()];
for (int i = 0; i < videoLinks.size(); i++) {
urls[i] = videoLinks.get(i).attr("data-video");
if (!urls[i].startsWith("https://")) {
urls[i] = "https://" + urls[i];
}
}
System.out.println(Arrays.asList(urls));
With your new test case, you can obtain the desired information with a very similar code:
String html = "<div class=\"heading-servers\">\n" +
" <span><i class=\"fa fa-signal\"></i> Servers</span>\n" +
" <ul class=\"servers\">\n" +
" <li data-vs=\"https://example.com\" class=\"server server-active\" style=\"display: block;\" onclick=\"return loadIframe('ifrm', this.getAttribute('data-vs'));\">Netu</li>\n" +
" <li data-vs=\"https://example.com\" class=\"server\" style=\"display: block;\" onclick=\"return loadIframe('ifrm', this.getAttribute('data-vs'));\">VideoVard</li>\n" +
" <li data-vs=\"https://example.com\" class=\"server\" style=\"display: block;\" onclick=\"return loadIframe('ifrm', this.getAttribute('data-vs'));\">Doodstream</li>\n" +
" <li data-vs=\"https://example.com\" class=\"server\" style=\"display: block;\" onclick=\"return loadIframe('ifrm', this.getAttribute('data-vs'));\">Okstream</li>\n" +
" </ul>\n" +
" </div>";
Document document = Jsoup.parse(html);
Elements videoLinks = document.select("div.heading-servers ul.servers li.server");
String[] urls = new String[videoLinks.size()];
for (int i = 0; i < videoLinks.size(); i++) {
urls[i] = videoLinks.get(i).attr("data-vs");
if (!urls[i].startsWith("https://")) {
urls[i] = "https://" + urls[i];
}
}
System.out.println(Arrays.asList(urls));
The most important part is the definition of the selector that should be applied to the parsed document, div.heading-servers ul.servers li.server in our case.
I provided a selector with many fragments, but depending on the actual use HTML it could be simplified with ul.servers li.server or even li.server.

How to get the text attribute of a input element? - webdriver

<ul class="list-group opened-list d-none" xpath="1">
<li class="list-group-item col-12" xpath="1">My team</li>
<li class="list-group-item col-12" xpath="1">My name</li>
<li class="list-group-item col-12" xpath="1">My film</li>
<li class="list-group-item col-12" xpath="1">My football teammate</li>
</ul>
Dropdown list without select tag

To get all li elements use .list-group.opened-list .list-group-item css selector.
Code below wait visibility of li elements and then print text for each:
WebDriverWait wait = new WebDriverWait(driver, 10);
List<WebElement> options = wait.until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.cssSelector(".list-group.opened-list .list-group-item")));
options.forEach(element -> System.out.println(element.getText()));
If you want to select one of the element by text, check example here.

Jsoup select li from ul

I want to select from following ul all children with Jsoup.
<ul class="breadcrumbs xlarge-12 columns hide-for-small-only">
<li>
<a href="http://www.thalia.de/shop/home/show/">
Home
</a>
</li>
<li>
<a href="http://www.thalia.de/shop/buecher/show/">
Bücher
</a>
</li>
<li>
<a href="http://www.thalia.de/shop/fachbuecher-115/show/">
Fachbücher
</a>
</li>
<li>
<a href="http://www.thalia.de/shop/chemie-143/show/">
Chemie
</a>
</li>
</ul>
Here is my code, but this gives me only the first two elements. What am I doing wrong?
Elements category = doc.select("div.ncMain.productMainView");
Elements category2 = category.select("ul.breadcrumbs.xlarge-12.columns.hide-for-small-only");
Elements category3 = category2.select("ul li a");
String categoryString = category3.text();

Iterate through a list?

I have the following markup.
Edit: Added full markup
<div id="SelectList">
<div class="select-area-left"></div>
<div class="select-area-right"></div>
<div id="SelectedOption">Option0</div>
<ul id="ShowOptions">
<li id="ShowOption0">Option0</li>
<li id="ShowOption1">Option1</li>
<li id="ShowOption2">Option2</li>
<li id="ShowOption3">Option3</li>
<li id="ShowOption4">Option4</li>
<li id="ShowOption5">Option5</li>
<li id="ShowOption6">Option6</li>
<li id="ShowOption7">Option7</li>
<li id="ShowOption8">Option8</li>
<li id="ShowOption9">Option9</li>
<li id="ShowOption10">Option10</li>
<li id="ShowOption11">Option11</li>
<li id="ShowOption12">Option12</li></ul></div>
And i'm trying to use the following code to print out each list elements text.
List<WebElement> allElements = driver.findElements(By.xpath("//div[#id='SelectList']/ul"));
for (WebElement element: allElements) {
System.out.println(element.getText());
}
But its just given a blank output. Is there something i'm missing?

Try following code :
List<WebElement> allElements = driver.findElements(By.xpath("//div[#id='SelectList']/ul/li"));
for (WebElement element: allElements) {
System.out.println(element.getText());
}
Basically you missed to point li element in the xpath used. Above code should work for you.

Pagination in webDriver displays Stale Element Reference Exception

Environment : Eclipse, Chrome, Java
I am dealing with test case for pagination in the application. I have tried with some code, but it moves only upto 2nd page.
Code :
List<WebElement> allpages = driver.findElements(By.xpath("//div[#class='pagination']//a"));
System.out.println(allpages.size());
if(allpages.size()>0)
{
System.out.println("Pagination exists");
for(int i=0; i<allpages.size(); i++)
{
Thread.sleep(3000);
allpages.get(i).click();
driver.manage().timeouts().pageLoadTimeout(5, TimeUnit.SECONDS);
//System.out.println(i);
}
}
else
{
System.out.println("Pagination doesn't exists");
}
}
Size displayed is 12. The issue is it moves upto 2nd page only and then displays error of StaleElementReference
Here's the HTML Code for the same pagination.
HTML Code :
<div id="page-navigation" class="pull-right">
<div id="303b171e-5a26-e456" class="flex-view">
<div class="pagination">
<ul>
<li class="">
«
</li>
<li class="" data-value="0">
1
</li>
<li class="" data-value="1">
2
</li>
<li class="active" data-value="2">
3
</li>
.....

When the new page is loaded after clicking one of the pagination links, the allpages WebElements are no longer valid and need to be found again. You will have to put another call to allpages = driver.findElements(By.xpath("//div[#class='pagination']//a")); in your for loop so that you can get new references for each new page.