I am working on an app that will help me log in the website and view data that I need. While I have no trouble with making sure that I parse that data and work with it properly, I did face an issue with logging into the website. I tried sending POST request, yet that didn't really work for some reason so I started looking more closely into how POST request to that website is sent in the browser and here is what I got:
Picture
I also asked a guy who developed that website and he said that I should use two cookies with "ulogin" and "upassword" for my log in. I tried using JSOUP as shown right here: https://jsoup.org/cookbook/input/load-document-from-url
I used .cookies("upassword", "10101010"), yet it didn't work so it makes me think that there is a bit more to it than just writing a simple line a post request.
Please, can someone explain to me how do I use cookies to log into website or at least point me in the direction where I can learn that, because I am so close to making that app happen and I will be able to proceed further with it's development, but it's just this one step that I am really being stuck with.
Here is an additional picture with Response and Request Headers from the Firefox. Picture
I managed to get it working a long time, yet didn't post an answer. So, here we go.
Cookies are just simple Headers, therefore you should treat them as such. In my case, with the use of HttpURLConnection, here is a piece of working code:
Note: My original request is for Java, however, I have since moved to Kotlin, so this solution uses Kotlin and this function is a "suspend" function which means that it is designed to be used with Kotlin Couroutines.
suspend fun httpRequest(): String {
val conn: HttpURLConnection = url_profile.openConnection() as HttpURLConnection
conn.requestMethod = "POST"
conn.doOutput = true
conn.doInput = true
conn.setRequestProperty(
"Cookie",
"YOUR COOKIE DATA"
)
val input: BufferedReader = BufferedReader(InputStreamReader(conn.inputStream))
return input.readText()
}
Related
I've been familiarizing myself with Play Framework in Java. I took on a fairly simple change request for an application, but I don't think I fully understand how to go about fixing it. The request is to repopulate the email field after a failed login attempt. I realize it's probably a pretty simple fix but I wasn't able to find a solution for this particular situation. Here's what I've found
Routes file:
#Sessions
GET /logout #femr.ui.controllers.SessionsController.delete()
POST /login/reset #femr.ui.controllers.SessionsController.editPasswordPost()
POST /login #femr.ui.controllers.SessionsController.createPost()
GET /login #femr.ui.controllers.SessionsController.createGet()
Code Snippit within SessionsController.java (Commented lines are my attempted fix):
public Result createPost() {
final Form<CreateViewModel> createViewModelForm = formFactory.form(CreateViewModel.class);
CreateViewModel viewModel = createViewModelForm.bindFromRequest().get();
ServiceResponse<CurrentUser> response = sessionsService.createSession(viewModel.getEmail(), viewModel.getPassword(), request().remoteAddress());
if (response.hasErrors()) {
//CreateViewModel emailModel = new CreateViewModel();
//emailModel.setEmail(viewModel.getEmail());
//createViewModelForm.fill(emailModel);
return ok(create.render(createViewModelForm));
}
Here I'm attempting to fill in the email field before rendering the form, but it didn't quite do the trick. I looked online and saw something about doing an explicit render but that didn't quite work for me either.
The form I found is create.scala.html and I believe the solution may involve using javascript or something in this, but I'd like to handle it just in the SessionController in Java if possible.
Suggestions welcome!
I'm start learning java programming, and I want make a simple server application. I read about com.sun.net.httpserver.HttpServer and find a good example on this link: https://github.com/imetaxas/score-board-httpserver-corejava.
I understand how to do Get-request in url, but I don't know how POST works. I think it must be sent a form or data on the server.
I attach the link of project, which I'm learning, in readme the author wrote http://localhost:8081/2/score?sessionkey=UICSNDK - it's not working...
I wrote in url and get sessionkey: "localhost:8081/4711/login --> UICSNDK"
I wrote in url this for Post request: "localhost:8081/2/score?sessionkey=UICSNDK" - not working and in chrome return 404 bad request
3.wrote in url this:"localhost:8081/2/highscorelist"
Please help me, I am beginner.
The difference between GET and POST is that with a GET request the data you wish to pass to the endpoint is done by modifying the url itself by adding parameters to it.
With a POST any data you wish to send to the endpoint must be in the body of the request.
The body of a request is arbitrary data that comes after a blank line in the header The reqiest has the request line, following by any number of header attributes, then a blank line.
The server would need to know what the format of the body of the request was and parse it as appropriate.
Of course 'modern' frameworks like jax-rs allow you to automatically convert request data to objects, so that it is much simpler.
I'm developing an app which I plan on implementing Piwik Analytics into. Although there seems to be no real documentation (unless I'm blind) on how to use the Java Tracking API (this one), I think I managed to figure it out, but when I try it, I get this error
WARNING: Warning:400 Bad Request
org.piwik.PiwikException: error:400 Bad Request
at org.piwik.SimplePiwikTracker.sendRequest(SimplePiwikTracker.java:878)
at test.Main.main(Main.java:32)
Here is my code:
SimplePiwikTracker tracker = new SimplePiwikTracker("http://example.com/piwik/piwik.php");
tracker.setIdSite(2);
tracker.setPageUrl("http://example.com/javatest/test");
tracker.setPageCustomVariable("test", "10");
//This is line 32
tracker.sendRequest(tracker.getLinkTrackURL("http://example.com/piwik/piwik.php"));
I also tried this:
String datURL = tracker.getLinkTrackURL("http://example.com/piwik/piwik.php").toString();
URL url = new URL(datURL);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
System.out.println(connection.getResponseCode());
connection.disconnect();
But this just prints out response code 404 which doesn't make any sense because if I paste the link generated by tracker.getLinkTrackURL into my browser it works perfectly and logs correctly in Piwik (I have debug mode on).
So my question is, why is Piwik returning a bad request and why can't Java find the URL (404) when it works perfectly fine in my browser?
I figured it out. Instead of having the URL point to the http://example.com/piwik/piwik.php, it should just be going to http://example.com/piwik
I am looking for a clean/simple way in HtmlUnit to request a webpage from a server in a specific language.
To do this i have been trying to request "bankofamerica.com" for their homepage in spanish instead of english.
This is what i have done so far:
I tried to set "Accept-Language" header to "es" in the Http request. I did this using:
myWebClient.addRequestHeader("Accept-Language" , "es");
It did not work. I then created a web request with the following code:
URL myUrl = new URL("https://www.bankofamerica.com/");
WebRequest myRequest = new WebRequest(myUrl);
myRequest.setAdditionalHeader("Accept-Language", "es");
HtmlPage aPage = myWebClient.getPage(myRequest);
Since this failed too i printed out the request object for this url , to check if these headers are being set.
[<url="https://www.bankofamerica.com/", GET, EncodingType[name=application/x-www-form-urlencoded], [], {Accept-Language=es, Accept-Encoding=gzip, deflate, Accept=*/*}, null>]
So the server is being requested for a spanish page but in response its sending the homepage in english (the response header has the value of Content-Language set to en-US)
I did find a hack to retrieve the BOA page in spanish. I visited this page and used the chrome developer tool to get the cookie value from the request
header. I used this value to do the following:
myRequest.setAdditionalHeader("Cookie", "TLTSID= ........._LOCALE_COOKIE=es-US; CONTEXT=es_US; INTL_LANG=es_US; LANG_COOKIE=es_US; hp_pf_anon=anon=((ct=+||st=+||fn=+||zc=+||lang=es_US));..........1870903; throttle_value=43");
I am guessing the answer lies somewhere here.
Here lies my next question. If i am writing a script to retrieve 100 different websites in Spanish (ie Assuming they all have their pages in the spanish) . Is there a clean way in HtmlUnit to accomplish this.
(If cookies is indeed a solution then to create them in htmlunit you need to specify the domain name. One would have to then create cookies for each of the 100 sites. As far as i know there is no way in HtmlUnit to do something like:
Cookie langCookie = new Cookie("All Domains","LANG_COOKIE","es_US");
myWebClient.getCookieManager().addCookie(langCookie);)
NOTE: I am using HtmlUnit 2.12 and setting BrowserVersion.CHROME in the webclient
Thanks.
Regarding your first concern the clear/simple(/only?) way of requesting a webpage in a particular language is, as you said, to set the HTTP Accept-Language request header to the locale(s) you want. That is it.
Now the fact that you request a page in a particular language doesn't mean that you will actually get a page in that language. The server has to be set up to process that HTTP header and respond accordingly. Even if a site has a whole section in spanish it doesn't mean that the site is responding to the HTTP header.
A clear example of this is the page you provided. I performed a quick test on it and found that it is clearly not responding accordingly to the Accept-Language I've set (which was es). Hitting the home page using es resulted in getting results in english. However, the page has a link that states En Español which means In Spanish the page does switch to spanish and you get redirected to https://www.bankofamerica.com?request_locale=es_US.
So you might be tempted to think that the page handles the locale by a request parameter. However, that is not (only) the case. Because if you then open the home page again (without the locale parameter) you will see the Spanish version again. That is clearly a proof that they are being stored somewhere else, most likely in the session, which will most likely be handled by cookies.
That can easily be confirmed by opening a private session or clearing the cookies and confirming this behaviour (I've just done that).
I think that explains the mystery of the webpage existing in Spanish but being fetched in English. (Note how most bank webpages do not conform to basic standards such as responding to simple HTTP requests... and they are handling our money!)
Regarding your second question, it would be like asking What is the recipe to not get ill ever?. It just doesn't depend on you. Also note that your first concerned used the word request while your second concern used the word retrieve. I think it should be clear by now that you can only be 100% sure of what you request but not of what you retrieve.
Regarding setting a value in a cookie manually, that is technically possible. However, that is just like adding another parameter in a get request: http://domain.com?login=yes. The parameter will only be processed by the server if it is expecting it. Otherwise, it will be ignored. That is what will happen to the value in your cookie.
Summary: There are standards to follow. You can try to use them but if the one in the other side doesn't then you won't get the results you expect. Your best choice: do your best and follow the standards.
I am having a Java applet developed on my behalf. The applet makes AJAX requests to a server. I have not written the server code yet.
Is there a design pattern I can use to mock the server responses, whilst the server is not yet ready, so that the applet can be developed and tested against this "mock server"?
Some sample code on how to implement the mockup server would be very useful
Although I'm not entirely sure if this suits your needs, but you can checkout a simple class I made for this purposes here. Here is some sample code:
//let's say that when you GET to /users?id=2 the server should return the user with id 2
//first start the server on your favourite port
MockHttpServer server = new MockHttpServer(portNum);
server.start();
//We will add a mock response for every request we will do, so in this case, just one mock response
server.enqueueResponse(Status.OK, "application/json,"{\"user_id\":15,\"name\":\"paul\"}");
//now we will use curl or whatever to make the GET
curl http://0.0.0.0:3000/users?id=2
// we get the request object
Request req = server.getRequest();
assertEquals(req.getMethod(),"GET");
assertEquals(req.getParams().get("id"),"2")
Its still a work in progress but you can get an idea by reading the code on github. Hope it helps :)