Develop Reference

Jsoup connect(url) get encoding

I'm having some issues when connecting to a URL with Jsoup, I am unable to set the encoding of HTML, the text in the tags are only displayed as "?". I've searched exhaustedly here in the forum and in the documentation but I can't make any solution that is proposed to work.
This is one of the HTML parts that gives me the issue when running the Jsoup connect
The result when running the connection is this:
(source: i.ibb.co)
If I try to use the parser, I have the following message: "Please enable JavaScript to view the page content"
As described in some threads here in stackoverflow, I've changed the output encoding to check if the problem was that, but the result was the same. I tried saving the content to a file in the correct iso and it didn't work as well, same output with the question marks.
The snippet that I am using is yet very simple since I am just trying to get the HTML:
Document doc = Jsoup.connect(a)
.header("Content-Type", "application/x-www-form-urlencoded")
.postDataCharset("ISO-8859-1") // tried other encodings but no success as well, same output
.get();
System.out.println(doc);
Have anyone had this problem before using the connect().get() from Jsoup?
Update
Using another site the issue is not presented:
String a = "https://flatschart.com/html5/descricao.html";
Document doc = Jsoup.connect(a)
.header("Content-Type", "application/x-www-form-urlencoded")
.postDataCharset("ISO-8859-1")
.get();
System.out.println(doc);

Extract the list of URLs obtained during a HTML page render in Java

I want to be able to get the list of all URLs that a browser will do a GET request for when we try to open a page. For example, if we try to open cnn.com, there are multiple URLs within the first HTTP response which the browser recursively requests for.
I'm not trying to render a page, but I'm trying to obtain a list of all the URLs that are requested when a page is rendered. Doing a simple scan of the HTTP response content wouldn't be sufficient, as there could potentially be images in the CSS which are downloaded. Is there any way I can do this in Java?
My question is similar to this question, but I want to write this in Java.

You can use Jsoup library to extract all the links from a webpage, e.g.:
Document document = Jsoup.connect("http://google.com").get();
Elements links = document.select("a[href]");
for(Element link : links) {
System.out.println(link.attr("href"));
}
Here's the documentation.

web scraping jsoup java unable to scrape full information

I have an information to be scraped from a website. I could scrape it. But not all the information is being scraped. There is so much of data loss. The following images helps you further to understand :
I used Jsoup, connected it to URL and then extracted this particular data using the following code :
Document doc = Jsoup.connect("https://www.awattar.com/tariffs/hourly#").userAgent("Mozilla/17.0").get();
Elements durationCycle = doc.select("g.x.axis g.tick text");
But in the result, I couldn't find any of that related information at all. So I printed the whole document from the URL and it shows the following :
I could see the information when I download the page and read it as an input file but not when I connect directly to URL. But I want to connect it to URL. Is there any suggestion?
I hope my question is understandable. Let me know in case if it is not explanatory.

There is a request body limitation in Jsoup. you should use the maxBodySize parameter:
Document doc = Jsoup.connect("https://www.awattar.com/tariffs/hourly#").userAgent("Mozilla/17.0").maxBodySize(0).get();
"0" is no limit.

Log in https aspx web page with jsoup

Is it possible to log in to a https aspx web page using jsoup ?
the page where i try to log in is: https://by.vulog.com/communauto-labs/login.aspx
what i'm tryng to do at the end is to access https://by.vulog.com/communauto-labs/index.aspx in order to parse the html to get some information, but when u try to access this page, i still redirecting me to the login page (I can see that by looking at the html of homePage variable)
Or should I use some other tools ?
Here is the my code wich does not seem to work:
Connection.Response response = Jsoup.connect("https://by.vulog.com/communauto-labs/login.aspx")
.method(Connection.Method.GET)
.execute();
response = Jsoup.connect("https://by.vulog.com/communauto-labs/login.aspx")
.data("ctl00$ContentPlaceHolder1$LoginForm$UserName", "my_login")
.data("ctl00$ContentPlaceHolder1$LoginForm$Password", "my_password")
.cookies(response.cookies())
.method(Connection.Method.POST)
.execute();
Document homePage = Jsoup.connect("https://by.vulog.com/communauto-labs/index.aspx")
.cookies(response.cookies())
.get();

Struggling with the problem, I used a brutal solution :
Connecting through my naviagtor (chrome), using developers tools to get the authetification cookies, and pass them directly to my program before launching it.
I don't like this solution but it's for a single use programm.

How to log in to pages using Jsoup

After a couple hours of searching, I'm still a bit stumped as to how to access an html page after I log in. Looking at the various other posts on here as well as the Jsoup API, I understand that accessing the page after the log-in page will require some code like this:
Connection.Response loginForm = Jsoup.connect("https://parentviewer.pisd.edu/")
.method(Connection.Method.GET)
.execute();
Document document = Jsoup.connect("https://parentviewer.pisd.edu/")
.data("username", "testUser")
.data("password", "testPass")
.data("LoginButton", "Login")
.cookies(loginForm.cookies())
.post();
However, I think my understanding may be a little skewed, as I still don't quite undestand exactly what I should put for each value.
For example, on the website of , would I be using input name="ctl00$ContentPlaceHolder1$portalLogin$UserName" as the key and "testUser" as the value?
Is my method of approaching this task even correct?
Any help is greatly appreciated.

Yes, this code will look like yours.
Connection.Response loginForm = Jsoup.connect("https://parentviewer.pisd.edu/")
.method(Connection.Method.GET)
.execute();
Document document = Jsoup.connect("https://parentviewer.pisd.edu/")
.data("ctl00$ContentPlaceHolder1$portalLogin$UserName", "testUser")
.data("ctl00$ContentPlaceHolder1$portalLogin$Password", "testPass")
.cookies(loginForm.cookies())
.post();
System.out.println(document.body().html());
How to make this working? Best way is to enable Web Developer Console in your browser and login this page. After this check what is sended from broswer to server and send this data with JSoup.
In your example request data look like this:
Request URL:https://parentviewer.pisd.edu/
Request Method:POST
Status Code:200 OK
FormData:
__LASTFOCUS:
__EVENTTARGET:
__EVENTARGUMENT:
__VIEWSTATE:/wEPDwULLTEwNjY5NzA4NTBkZMM/uYdqyffE27bFnREF10B/RqD4
__SCROLLPOSITIONX:0
__SCROLLPOSITIONY:106
__EVENTVALIDATION:/wEdAASCW34hepkNwIXSnvGxEUTlqcZt0XO7QUOibAd3ocrpayqHxD2e5zCnWBj9+m7TCi0S+C76MEjhL0ie/PsBbOp+Shjkt2W533uAqvBQcWZNXoh672M=
ctl00$ContentPlaceHolder1$portalLogin$UserName:testUser#gmail.com
ctl00$ContentPlaceHolder1$portalLogin$Password:testPass
ctl00$ContentPlaceHolder1$portalLogin$LoginButton:Login
Not all data are required, try with minimal request and check if this works.