I access a webpage by passing the session id and url and output is a HTML response.
I want to use jSoup to parse this response and get the tag elements.
I see the examples in Jsoup takes a String for establishing connection. How do i proceed.
pseudo code:
I tried the above method and got this exception
java.io.IOException: 401 error loading URL http://www.abc.com/index
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:387)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:364)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:143)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:132)
Basically the entity.getContent() has the HTML response which has to be passed as a String to the connect method. But it doesn't work.
Apache Commons HttpClient and Jsoup do not share the same cookie store. You basically need to pass the very same cookies as HttpClient has retrieved back through Jsoup's Connection. You can find some concrete examples here:
Sending POST request with username and password and save session cookie
how to maintain variable cookies and sessions with jsoup?
Alternatively, you can also just continue using HttpClient for firing HTTP requests and maintaining the cookies and instead feeds its HttpResponse as String through Jsoup#parse().
So this should do:
HttpResponse httpResponse = httpclient1.execute(httpget, httpContext);
String html = EntityUtils.toString(httpResponse.getEntity());
Document doc = Jsoup.parse(html, testUrl);
// ...
By the way, you do not necessarily need to create a whole new HttpClient for a subsequent request. Just reuse httpclient which you already created. Also your way of obtaining the response as String is clumsy. The second line in the above example shows how to do it at simplest.
It shows an http error 401 which means
Similar to 403 Forbidden, but specifically for use when authentication is possible but has failed or not yet been provided.
Therefore, i think you need to login into the website using your java code or identify yourself by sending cookies through your code.
Related
i want to send a SAML request to my IDP (Azure AD) but ia m not sure how to send the request at all.
First i used OpenSAML to build an AuthRequest. Which i encoded as a String.
Now i wanted to use ApacheHttpClient to send the request and read the response and i am not sure if OpenSAML provides http sending methods at all so my idea was to use Apaches HttpClient for this for now.
String encodedAuthRequest = generateAuthRequest();
String url = "http://myidp/samlendpoint";
CloseableHttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(url);
// add request header
request.addHeader("User-Agent", USER_AGENT);
// what is to add else?
HttpResponse response = client.execute(request);
I am stuck now since i am not sure how to setup the request, does it need to be a query parameter like ?saml=.... in GET or do i have to put the encoded saml response in the body as POST..
Can someone help or clarify these issue?
Update from Guillaumes answer:
I have this from the IDPs MetaData:
<IDPSSODescriptor>
<SingleSignOnService
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect"
Location="https://myidp/saml2" />
<SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
Location="https://myidp/saml2" />
Depends on which binding you are supposed to use. The IdP documentation or metadata should mention that. There are several:
Redirect Binding (using a GET), by far the most common for Requests
POST Binding
Artifact Binding (more complex, but I have never seen it used for Requests)
...
I suppose that Redirect Binding will be used in your case (EDIT: you added the metadata from your IdP, it mentions that you can use both Redirect and POST bindings). It is described here: https://docs.oasis-open.org/security/saml/v2.0/saml-bindings-2.0-os.pdf page 15.
Short version: your must first use the DEFLATE algorithm to compress your XML Request, encode it using base64, encode it using URL encoding, then pass it as a query parameter named SAMLRequest
?SAMLRequest=<your url-encoded base64-encoded deflated authnrequest>
https://en.wikipedia.org/wiki/SAML_2.0#SP_Redirect_Request.3B_IdP_POST_Response
is it possible to pass some data in HTTP Header, while redirecting a request from one server to another.
Here is my scenario,
I have one generic filter, via which every request is passing.
Now, based on some condition, I'm redirecting the request to some different server using the API objHttpServletResponse.sendRedirect(strURL).
But, the issue is, when I'm setting some data in response header like objHttpServletResponse.setHeader("Key", "Value"); That's not available in the redirected server.
So, my questions are,
1. Is there any way to pass some data in header while redirecting a request?
2. If not, what are the other possible ways to send some data while redirecting a request?
Please Note: few other ways, like
using URL parameters:
objHttpServletResponse.sendRedirect(strURL+"?param="+ strParamValue);
or
using session:
HttpSession session = httpRequest.getSession();
session.setAttribute("Key", "Value");
is not what I'm expecting.
The headers you set are written to the response that gets sent to the client, along with a Location header and a status code. See Redirecting a request using servlets and the "setHeader" method not working
The client is then supposed to send an identical request to the URL you specified in the Location header. Identical to the request it sent to you.
You want the browser to send a header you specify along with the redirected request. Have you considered adding a (domain) Cookie header? Some googling leads me to believe that cookies set in a redirect response will get picked up by most browsers. See http://blog.dubbelboer.com/2012/11/25/302-cookie.html
Please have a look at Apache HttpClient.
This example adds several parameters to the post request :
String url = "https://selfsolve.apple.com/wcResults.do";
HttpClient client = HttpClientBuilder.create().build();
HttpPost post = new HttpPost(url);
// add header
post.setHeader("User-Agent", USER_AGENT);
List<NameValuePair> urlParameters = new ArrayList<NameValuePair>();
urlParameters.add(new BasicNameValuePair("sn", "C02G8416DRJM"));
urlParameters.add(new BasicNameValuePair("cn", ""));
urlParameters.add(new BasicNameValuePair("locale", ""));
urlParameters.add(new BasicNameValuePair("caller", ""));
urlParameters.add(new BasicNameValuePair("num", "12345"));
post.setEntity(new UrlEncodedFormEntity(urlParameters));
HttpResponse response = client.execute(post);
System.out.println("Response Code : "
+ response.getStatusLine().getStatusCode());
The problem is that the redirect() method of the response initiates a new request altogether, thereby loosing the attributes that were set before redirecting. Luckily there is a fluent way of solving the problem still.
response.setHeader("Key", "Value");
request.getRequestDispatcher("redirecturl").forward(request, response);
Then in your destination you can do
response.getHeaders("key")
You can use JS redirect, i.e. instead of calling sendRedirect return HTML page with embedded javascript that will do redirect setting headers you need.
However, using GET parameters is really the best solution. If you have concerns about users altering parameters manually - use MAC code to protect parameters.See
Message authentication code
In simplest form, ?p1=1&p2=2&mac={mac value}, where {mac value} = md5('MY_SECRET_KEY' + 'p1=1&p2=2').
Receiving side can recalculate MAC and compare it with provided one. Since external users can not know 'MY_SECRET_KEY', they will not be able to make valid MAC.
Have you checked the HTTP request/response from/to server? You can use a number of plugins on chrome or firefox to check that. You would be able to see if value is being passed from your server to another server or not
Also retrieve the header using httpResponse.getHeader("Key"); not using request.getHeader("key"). One of my colleague was facing same issue some days back, he was using request to fetch header values
This might be a trivial question but I'm trying to send web request to USPS to get a http post response (or email response depending on my request) containing the tracking information based on the tracking number that I send in. The documentation says the xml needs to appended as part of the url like below
http://secure.shippingapis.com/ShippingAPITest.dll?API=TrackV2&XML=<PTSEmailRequest USERID="xxxxx"><TrackId>xxxxx</TrackId><RequestType>EN</RequestType></PTSEmailRequest>
I saw there were 2 ways to make an xml request, one using HttpPost and the other URLConnection. I'm a bit thrown by how I go about this and I'm failing to appreciate what's the difference between appending xml in the url and a normal http request. Can someone please clear things up for me?
USPS documentation for tracking =>
https://www.usps.com/business/web-tools-apis/track-and-confirm.pdf
I read these related Stackoverflow posts
Java: How to send a XML request?
posting XML request in java
HttpClient httpclient = HttpClients.createDefault();
HttpPost httppost = new HttpPost("http://secure.shippingapis.com/ShippingAPITest.dll");
List<String> params = new ArrayList<String>(2);
params.add(new BasicNameValuePair("API", "TrackV2"));
params.add(new BasicNameValuePair("XML", FuncTOGenerateXML()));
httppost.setEntity(new UrlEncodedFormEntity(params, "UTF-8"));
HttpResponse response = httpclient.execute(httppost);
HttpEntity entity = response.getEntity();
if (entity != null) {
InputStream instream = entity.getContent();
//.....
// .....
instream.close();
}
An HTTP request can use one of several methods, like POST, GET, DELETE, PUT... Here we talk about POST and GET
Technical differences
With GET, the data is retrieved from the parameters in the URL.
With POST, the data is retrieved from the data transmitted inside the HTTP message.
Intended use differences
GET is intended to be used when the request does not cause a change (v.g., searching in Google). Since you can repeat the request without side effects, the data is in the URL and can be stored in the browser history, favorites, etc.
POST is intended to use when you are performing a change (v.g. sending an e-mail, doing a on-line purchase). The data related is not stored with the URL (it is then that, if you go back to a page that was obtained using POST, the browser many times will show you a pop-up asking for permission to send the data again.
In real usage, the distinction is not so clear cut, in particular POST is sometimes used when the data is too large (URLs have limited length). Also, sometimes GET is used with the meaning of POST so the data can be presented as an HTML link.
Finally, URLConnection is the basic API for opening a connection (which you can use as a POST or GET request, based in how you pass the data, or something else) and HttpPost is just a higher level API for creating a POST request. If you go the basic way, use HttpURLConnection better.
I'm trying to write a Java client (with Apache HttpClient) for the Gengo API which makes use of HTTP GET, POST, PUT and DELETE. However for every RESTful API "method" that they expose, you must pass your API key and signature as "parameters".
Would this mean query string parameters, POST variables, key-value pair headers, or something else?
I guess I'm just confused by what is meant by the word "parameters" in the context of all these different HTTP request methods. In other words, how would I pass the API key as a "parameter" to their API when I could be using GET, POST, PUT or DELETE? My understanding was that only HTTP GET can handle query string params, and that HTTP POST can only handle POST variables. And I have never used PUT or DELETE before so I'm not sure what they require.
So I ask: what mechanism can I use to send the API key/signature via all 4 types of request methods, or do they all support the processing of query string parameters? Thanks in advance.
You can try this. It works for my HttpClient application with POST request.
DefaultHttpClient httpClient = new DefaultHttpClient();
httpClient.getParams().setParameter(name, value);
......
For Example, I set the connection timeout:
httpClient.getParams().setIntParameter(HttpConnectionParams.CONNECTION_TIMEOUT, httpTimeout);
Then later, to send(execute) the request:
HttpResponse response = httpClient.execute([My HttpPost instance was here, but I think you can use HttpGet, HttpPut, and HttpDelete here as well]);
All verbs can use request parameters (also known as query parameters) and they will be available to the server in the same way regardless of if you also send a body.
In your example (Gengo) there is a good example on there page about authentication.
I have a problem with redirection, in my work I have to make a connection to a URL that automatically go to another URL at this point it takes the credential (username and password) and redirect to a URL that contains a parameter which need me. How can I take this parameter?
To be precise, I have to do this:
Embed a Web browser with the URL https://graph.facebook.com/oauth/authorize with your client_id and the type set to user_agent. Since your application is embedding a small window, you can trigger a compact "popup" version of the authorization dialog with the display parameter:
graph.facebook.com/oauth/authorize?
client_id=...&
redirect_uri=www.facebook.com/connect/login_success.html&
type=user_agent&
display=popup
After the user authorizes your application, we redirect the user back to www.facebook.com/connect/login_success.html with the access token in the URL fragment: www.facebook.com/connect/login_success.html#access_token=...&expires_in=... Intercept the redirect above and read the access token out of the URL. Use the access token to fetch data from the Graph API on behalf of the user:
graph.facebook.com/me?access_token=....
I need access_token.
you can use apache http components httpclient to do this,
it will automatically follow redirects
HttpClient client=new HttpClient();
GetMethod get=new GetMethod("graph.facebook.com/oauth/authorize?client_id=...& redirect_uri=www.facebook.com/connect/login_success.html&type=user_agent&display=popup");
int status=client.exectueMethod(get);
Then you will have the information you need in the Location header of the response which you can access by using:
String location=get.getResponseHeader("location").getValue();
and parse the location-header for the url fragment you want.
hope that helped