Is it possible to log in to a https aspx web page using jsoup ?
the page where i try to log in is: https://by.vulog.com/communauto-labs/login.aspx
what i'm tryng to do at the end is to access https://by.vulog.com/communauto-labs/index.aspx in order to parse the html to get some information, but when u try to access this page, i still redirecting me to the login page (I can see that by looking at the html of homePage variable)
Or should I use some other tools ?
Here is the my code wich does not seem to work:
Connection.Response response = Jsoup.connect("https://by.vulog.com/communauto-labs/login.aspx")
.method(Connection.Method.GET)
.execute();
response = Jsoup.connect("https://by.vulog.com/communauto-labs/login.aspx")
.data("ctl00$ContentPlaceHolder1$LoginForm$UserName", "my_login")
.data("ctl00$ContentPlaceHolder1$LoginForm$Password", "my_password")
.cookies(response.cookies())
.method(Connection.Method.POST)
.execute();
Document homePage = Jsoup.connect("https://by.vulog.com/communauto-labs/index.aspx")
.cookies(response.cookies())
.get();
Struggling with the problem, I used a brutal solution :
Connecting through my naviagtor (chrome), using developers tools to get the authetification cookies, and pass them directly to my program before launching it.
I don't like this solution but it's for a single use programm.
Related
When I open https://www.instagram.com/metallica/ in browser and view its source code, I see javascript variable window._sharedData containing "graphql" field
When I get this page by HtmlUnit, variable window._sharedData is not the same
What's the problem? How can I get the same js field as in browser using HtmlUnit?
BrowserVersion my = new BrowserVersionBuilder(BrowserVersion.FIREFOX_52)
.setUserAgent("Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)").build();
WebClient webClient = new WebClient(my);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setJavaScriptEnabled(false);
webClient.getOptions().setCssEnabled(false);
HtmlPage htmlPage = webClient.getPage("https://www.instagram.com/metallica/");
String pageContent = htmlPage.getWebResponse().getContentAsString();
UPD
window._sharedData in Browser: {"config":{"csrf_token":"zkBxaROkhJqxHV7QAYKvHYNU8QCP15vm","viewer":null,"viewerId":null},"country_code":"RU","language_code":"ru","locale":"ru_RU","entry_data":{"ProfilePage":
window._sharedData in response: {"config":{"csrf_token":"Rpm5P3Ok3ZUh7wVklBLPkMzw9k3u1tbz","viewer":null,"viewerId":null},"country_code":"RU","language_code":"en","locale":"en_US","entry_data":{"LoginAndSignupPage":
so the difference in LoginAndSignupPage instead of ProfilePage
UPD2
On my server instagram for unknown reason redirects any address to /accouts/login that's why content is different. So now the question is how can I prevent this redirect?
getWebResponse returns the response you got from the server. If you like to get the current state of the page you have to wait for the js in the page to finish and then use something like page.getEnclosingWindow().getEnclosedPage().asXML();
If you compare with a real browser please make sure
there are no cookies stored by the browser because HtmlUnit always starts with an empty cookie store
enable JavaScript for HtmlUnit
I am trying to create an App that logs into my schoool's grade servers and displays data. The website I am trying to log into is: "https://portal.mcpsmd.org/public/"
I have written my code according to this stackoverflow question. Here is the relevant code for my specific situation:
Connection.Response loginForm = Jsoup.connect("https://portal.mcpsmd.org/public/")
.method(Connection.Method.GET)
.execute();
Document document = Jsoup.connect("https://portal.mcpsmd.org/guardian/home.html/")
.data("account", "#######")
.data("pw","*******")
.cookies(loginForm.cookies())
.post();
System.out.println(document.title());
The reason I have written my code this way is because when I did "inspect element" on my school's page I saw this:
screenshot of school's inspect element
I can see that my school uses the method "post" and the login request is at "/guardian/home.html"
Many of the Stack Overflow questions I have looked at has told me to use a method like this, where I connect to the login request and send it my username and password as data. Am I going about this correctly at all? I am trying to use JSoup to login to my school's website, so in my mind if I do "document.title()" it should print the name of the page I get once already signing in. This program is only printing the name of the log-in screen page.
I have been working on this project for almost two days now, and initially started with 0 knowledge of JSoup. I would greatly appreciate any explanation on the best way to go about logging into a website using JSoup.
Guys I am trying send post method to https://www.servientrega.com/wps/portal/Colombia/transacciones-personas/rastreo-envios and get results of tracke and trace. I need to send this number for example : 2003159943. This is my code:
Connection.Response Form = Jsoup
.connect("https://www.servientrega.com/wps/portal/Colombia/transacciones-personas/rastreo-envios")
.validateTLSCertificates(false)
.method(Connection.Method.GET)
.execute();
Document document = Jsoup
.connect("https://www.servientrega.com/wps/portal/Colombia/transacciones-personas/rastreo-envios")
.validateTLSCertificates(false)
.data("txtNumGuia", "2003159943")
.cookies(Form.cookies())
.post();
I need to get this history:
Image with the data what I want
but I get this when I tried println(document):
Image with the result what I got
enter image description here
The data you want to obtain are set by javascript after page is downloaded. Jsoup does not execute javascript, it only downloads initial html.
If you examine what connections are made, for example with browser debugging tools you will find out, that the data are downloaded with request to the api: https://web.servientrega.com/PortalServientrega/WebServicePortal/tracking/api/envio/2003159943/1/es
The data you are looking for should be in response.
Document document = Jsoup.connect("https://web.servientrega.com/PortalServientrega/WebServicePortal/tracking/api/envio/2003159943/1/es")
.validateTLSCertificates(false)
.ignoreContentType(true)
.get();
System.out.println(document.text());
I'm testing my website and what I do is moving inside of it using Htmlunit library and Java. Like this for example:
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_45);
HtmlPage page1 = webClient.getPage(mypage);
// sent using POST
HtmlForm form = page1.getForms().get(0);
HtmlSubmitInput button = form.getInputByName("myButton");
HtmlPage page2 = button.click();
// I want to open page2 on a web browser and continue there using a function like
// continueOnBrowser(page2);
I filled a form programmatically using Htmlunit then I sent the form which uses a POST method. But I'd want to see the content of the response inside a web browser page. The fact is that if I use the URL to see the response it doesn't work since it's the response to a POST method.
It seems like it's the wrong approach to me, it's obvious that if you do anything programmatically you could not expect to open the browser and continue there... I can't figure out what could solve my problem.
Do you have any suggestions?
After a couple hours of searching, I'm still a bit stumped as to how to access an html page after I log in. Looking at the various other posts on here as well as the Jsoup API, I understand that accessing the page after the log-in page will require some code like this:
Connection.Response loginForm = Jsoup.connect("https://parentviewer.pisd.edu/")
.method(Connection.Method.GET)
.execute();
Document document = Jsoup.connect("https://parentviewer.pisd.edu/")
.data("username", "testUser")
.data("password", "testPass")
.data("LoginButton", "Login")
.cookies(loginForm.cookies())
.post();
However, I think my understanding may be a little skewed, as I still don't quite undestand exactly what I should put for each value.
For example, on the website of , would I be using input name="ctl00$ContentPlaceHolder1$portalLogin$UserName" as the key and "testUser" as the value?
Is my method of approaching this task even correct?
Any help is greatly appreciated.
Yes, this code will look like yours.
Connection.Response loginForm = Jsoup.connect("https://parentviewer.pisd.edu/")
.method(Connection.Method.GET)
.execute();
Document document = Jsoup.connect("https://parentviewer.pisd.edu/")
.data("ctl00$ContentPlaceHolder1$portalLogin$UserName", "testUser")
.data("ctl00$ContentPlaceHolder1$portalLogin$Password", "testPass")
.cookies(loginForm.cookies())
.post();
System.out.println(document.body().html());
How to make this working? Best way is to enable Web Developer Console in your browser and login this page. After this check what is sended from broswer to server and send this data with JSoup.
In your example request data look like this:
Request URL:https://parentviewer.pisd.edu/
Request Method:POST
Status Code:200 OK
FormData:
__LASTFOCUS:
__EVENTTARGET:
__EVENTARGUMENT:
__VIEWSTATE:/wEPDwULLTEwNjY5NzA4NTBkZMM/uYdqyffE27bFnREF10B/RqD4
__SCROLLPOSITIONX:0
__SCROLLPOSITIONY:106
__EVENTVALIDATION:/wEdAASCW34hepkNwIXSnvGxEUTlqcZt0XO7QUOibAd3ocrpayqHxD2e5zCnWBj9+m7TCi0S+C76MEjhL0ie/PsBbOp+Shjkt2W533uAqvBQcWZNXoh672M=
ctl00$ContentPlaceHolder1$portalLogin$UserName:testUser#gmail.com
ctl00$ContentPlaceHolder1$portalLogin$Password:testPass
ctl00$ContentPlaceHolder1$portalLogin$LoginButton:Login
Not all data are required, try with minimal request and check if this works.