Develop Reference

how to Web Scraping a dynamic page in android with JSOUP

I am trying to web scrape the website savevideo.tube using JSOUP.
When we put a link in the search bar and click the search button, the website dynamically loads and shows some download links that I want to scrape. My problem is how to load link in JSOUP with the link search without clicking the search button and showing the results (scraping the results).
Is there any way to search for a link and load it without clicking any button and get results?
I tried this code but I'm not getting the required result.
val result:Document = Jsoup.connect(Constants.BASE_URL)
.data("url", Constants.YOUTUBE_LINK)
.data("sid", "9823478982349872384789273489238904790234")
.userAgent("Mozilla").post()

JSOUP is a Static HTML parser. You cannot parse the content that is loaded by javascript dynamically. For that, you have to use a web drive.
The best web drives that you can use are
HTML Unit
JBrowserDriver
You can also use selenium but it may not be ideal for android
:-

As mentioned by ʀᴀʜɪʟ, JSOUP is a static HTML parser only. If you want to scrape a website that uses JS generated content you should probably take a look at skrape.it library
fun getDocumentByUrl(urlToScrape: String) = skrape(BrowserFetcher) { // <--- pass BrowserFetcher to include rendered JS
request { url = urlToScrape }
response { htmlDocument { this } }
}
fun main() {
// do stuff with the document
}

Jsoup with a plugin

I'm using Jsoup to scrape some online data from different stores, but I'm having trouble figuring out how to programmatically replicate what I do as a user. To get the data manually (after logging in), a user must select a store from a tree that pops up.
As best I can tell, the tree is not hard-coded into the site but is built interactively when your computer interacts with the server. When you look for the table in "view page source," there are no entries. When I inspect the tree, I do see the HTML and it seems to come from the "FancyTree" plugin.
As best as I can tell from tracking my activity on Developer Tools -- Network, the next step is a "GET" request which doesn't change the URL, so I'm not sure how my store selection is being transferred.
Any advice on how to get Jsoup or Java generally to programmatically interact with this table would be extremely helpful, thank you!

Jsoup can only parse the original source file, not the DOM. In order to parse the DOM, you'll need to render the page with something like HtmlUnit. Then you can parse the html content with Jsoup.
// load page using HTML Unit and fire scripts
WebClient webClient = new WebClient();
HtmlPage myPage = webClient.getPage(myURL);
// convert page to generated HTML and convert to document
doc = Jsoup.parse(myPage.asXml());
// do something with html content
System.out.println(doc.html());
// clean up resources
webClient.close();
See Parsing Javascript Generated Page with Jsoup.

Parsing shopping websites usign jsoup

I have the following code:
doc = Jsoup.connect("http://www.amazon.com/gp/goldbox").userAgent("Mozilla").timeout(5000).get();
Elements hrefs = doc.select("div.a-row.layer");
System.out.println("Results:"+ hrefs); //I am trying to print out contents but not able to see the output.
Problem: Want to display all image src within the div with class name "a-row layer". But, i am unable to see the output.
What is the mistake with my query?

I have taken a look at the website and tested it myself. The issue seems to be that the piece of html code you want to extract (div.a-row.layer) is generated by JavaScript.
Jsoup does not support JavaScript and cannot parse those generated by it. You would need a headless web browser to deal with this, such as HTMLUnit.

How can I reload only a part of code - Java

I have a JSP page that at the end comes a pop-up window from a javascript, I want after the users click to reload only a specific part of the JSP page more specific one if - loop want to be reloaded one more time ...can this happened or the idea is totally wrong ?

What you are looking for is called AJAX.
AJAX is the art of exchanging data with a server, and updating parts
of a web page - without reloading the whole page.
jQuery has a very nice ajax api.
To load html pages, you could use jQuery.load like this:
$("#result").load( "ajax/test.html" );
You can also generate dynamic html by calling a REST function of a php page using a different url. Example:
var data = { limit: 25, otherProp: "val" }
$("#result").load( "getHtmldata.php", data);

jQuery dilemma in loading different pages on one page

Hello i need some help to figure out what to do .
I am trying to create a page that has a list of events and each time I click one of the list's elements a different photo gallery should load. I did this by loading each gallery in a different iframe.
The problem is that right now the only thing it dose is loading the first galery and the other ones don't seem to manage loading any pictures (if I refresh each frame than they work fine)
What should i do?
This is the script I used in the webpage
You can find the page source here http://www.avantsolutions.ro/exp.txt

You can try with jQuery UI Plugin instead of iFrame.
On clicking of list item(West pane), you can load center pane div with corresponding images.
Check the examples here.

You can use jQuery's load function to load a gallery via ajax.
$(selectorForYourGalleryDisplayDiv).load( url, [ data ], [ complete(responseText, textStatus, XMLHttpRequest) ] )
Load will take the result from url, if you need to pass it data it will use a HTTP POST, and replace the inner html of the wrapped set it is called on.
I don't know the internals of your architecture, but the main process is giving your galleries Ids, pass them to the server using the JSON format, your code will use that id to get what's needed for the gallery, render the html and return it.
load will drop that html into the elements that match selectorForYourGalleryDisplayDiv.