This might be a trivial question but I'm trying to send web request to USPS to get a http post response (or email response depending on my request) containing the tracking information based on the tracking number that I send in. The documentation says the xml needs to appended as part of the url like below
http://secure.shippingapis.com/ShippingAPITest.dll?API=TrackV2&XML=<PTSEmailRequest USERID="xxxxx"><TrackId>xxxxx</TrackId><RequestType>EN</RequestType></PTSEmailRequest>
I saw there were 2 ways to make an xml request, one using HttpPost and the other URLConnection. I'm a bit thrown by how I go about this and I'm failing to appreciate what's the difference between appending xml in the url and a normal http request. Can someone please clear things up for me?
USPS documentation for tracking =>
https://www.usps.com/business/web-tools-apis/track-and-confirm.pdf
I read these related Stackoverflow posts
Java: How to send a XML request?
posting XML request in java
HttpClient httpclient = HttpClients.createDefault();
HttpPost httppost = new HttpPost("http://secure.shippingapis.com/ShippingAPITest.dll");
List<String> params = new ArrayList<String>(2);
params.add(new BasicNameValuePair("API", "TrackV2"));
params.add(new BasicNameValuePair("XML", FuncTOGenerateXML()));
httppost.setEntity(new UrlEncodedFormEntity(params, "UTF-8"));
HttpResponse response = httpclient.execute(httppost);
HttpEntity entity = response.getEntity();
if (entity != null) {
InputStream instream = entity.getContent();
//.....
// .....
instream.close();
}
An HTTP request can use one of several methods, like POST, GET, DELETE, PUT... Here we talk about POST and GET
Technical differences
With GET, the data is retrieved from the parameters in the URL.
With POST, the data is retrieved from the data transmitted inside the HTTP message.
Intended use differences
GET is intended to be used when the request does not cause a change (v.g., searching in Google). Since you can repeat the request without side effects, the data is in the URL and can be stored in the browser history, favorites, etc.
POST is intended to use when you are performing a change (v.g. sending an e-mail, doing a on-line purchase). The data related is not stored with the URL (it is then that, if you go back to a page that was obtained using POST, the browser many times will show you a pop-up asking for permission to send the data again.
In real usage, the distinction is not so clear cut, in particular POST is sometimes used when the data is too large (URLs have limited length). Also, sometimes GET is used with the meaning of POST so the data can be presented as an HTML link.
Finally, URLConnection is the basic API for opening a connection (which you can use as a POST or GET request, based in how you pass the data, or something else) and HttpPost is just a higher level API for creating a POST request. If you go the basic way, use HttpURLConnection better.
Related
We have some old java code that POSTs some fields and values to a dotnet5 web api - The api is having problems dealing with the body of the POST as it includes the url/uri as the first part of the body.
The Java sends: http://127.0.0.1:5555?producerRef=GREEN&systemId=78&status=false
But the api is expecting something like: producerRef=GREEN&systemId=78&status=false
as per https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST#example. If we send a test message via Postman then the api has no problems.
This is the Java code:
List<NameValuePair> params = new ArrayList<NameValuePair>(queryParams.size());
for (Map.Entry<String, String> entry : queryParams.entrySet()) {
params.add(new BasicNameValuePair(entry.getKey(), entry.getValue()));
}
// the address is just that, there's NO parameters
HttpPost post = new HttpPost(this.cmAddress.toURI());
post.setEntity(new UrlEncodedFormEntity(params, "UTF-8"));
post.setHeader(HttpHeaders.CONTENT_TYPE, "application/x-www-form-urlencoded");
CloseableHttpResponse response = httpClient.execute(post);
It's quite simple, but always adds the url to the start of the body of the request. If this is the only way to produce this, what could I do to produce something that looks like this: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST#example
Many Thanks.
This request seems like a GET request rather than a POST since the request params are in the URL. i don't know about the specifications of the Api you're using, but you can try OKHTTP, you can easily copy the code directly from postman
Postman Get example:
Your issue seems to be at below line
HttpPost post = new HttpPost(this.cmAddress.toURI());
This is the only place which will set the POST url ( another way is to use setURI which is not called anywhere in the code sample you have shared).
If you can use a debugger try checking the value of cmAdress variable
i want to send a SAML request to my IDP (Azure AD) but ia m not sure how to send the request at all.
First i used OpenSAML to build an AuthRequest. Which i encoded as a String.
Now i wanted to use ApacheHttpClient to send the request and read the response and i am not sure if OpenSAML provides http sending methods at all so my idea was to use Apaches HttpClient for this for now.
String encodedAuthRequest = generateAuthRequest();
String url = "http://myidp/samlendpoint";
CloseableHttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(url);
// add request header
request.addHeader("User-Agent", USER_AGENT);
// what is to add else?
HttpResponse response = client.execute(request);
I am stuck now since i am not sure how to setup the request, does it need to be a query parameter like ?saml=.... in GET or do i have to put the encoded saml response in the body as POST..
Can someone help or clarify these issue?
Update from Guillaumes answer:
I have this from the IDPs MetaData:
<IDPSSODescriptor>
<SingleSignOnService
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect"
Location="https://myidp/saml2" />
<SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
Location="https://myidp/saml2" />
Depends on which binding you are supposed to use. The IdP documentation or metadata should mention that. There are several:
Redirect Binding (using a GET), by far the most common for Requests
POST Binding
Artifact Binding (more complex, but I have never seen it used for Requests)
...
I suppose that Redirect Binding will be used in your case (EDIT: you added the metadata from your IdP, it mentions that you can use both Redirect and POST bindings). It is described here: https://docs.oasis-open.org/security/saml/v2.0/saml-bindings-2.0-os.pdf page 15.
Short version: your must first use the DEFLATE algorithm to compress your XML Request, encode it using base64, encode it using URL encoding, then pass it as a query parameter named SAMLRequest
?SAMLRequest=<your url-encoded base64-encoded deflated authnrequest>
https://en.wikipedia.org/wiki/SAML_2.0#SP_Redirect_Request.3B_IdP_POST_Response
is it possible to pass some data in HTTP Header, while redirecting a request from one server to another.
Here is my scenario,
I have one generic filter, via which every request is passing.
Now, based on some condition, I'm redirecting the request to some different server using the API objHttpServletResponse.sendRedirect(strURL).
But, the issue is, when I'm setting some data in response header like objHttpServletResponse.setHeader("Key", "Value"); That's not available in the redirected server.
So, my questions are,
1. Is there any way to pass some data in header while redirecting a request?
2. If not, what are the other possible ways to send some data while redirecting a request?
Please Note: few other ways, like
using URL parameters:
objHttpServletResponse.sendRedirect(strURL+"?param="+ strParamValue);
or
using session:
HttpSession session = httpRequest.getSession();
session.setAttribute("Key", "Value");
is not what I'm expecting.
The headers you set are written to the response that gets sent to the client, along with a Location header and a status code. See Redirecting a request using servlets and the "setHeader" method not working
The client is then supposed to send an identical request to the URL you specified in the Location header. Identical to the request it sent to you.
You want the browser to send a header you specify along with the redirected request. Have you considered adding a (domain) Cookie header? Some googling leads me to believe that cookies set in a redirect response will get picked up by most browsers. See http://blog.dubbelboer.com/2012/11/25/302-cookie.html
Please have a look at Apache HttpClient.
This example adds several parameters to the post request :
String url = "https://selfsolve.apple.com/wcResults.do";
HttpClient client = HttpClientBuilder.create().build();
HttpPost post = new HttpPost(url);
// add header
post.setHeader("User-Agent", USER_AGENT);
List<NameValuePair> urlParameters = new ArrayList<NameValuePair>();
urlParameters.add(new BasicNameValuePair("sn", "C02G8416DRJM"));
urlParameters.add(new BasicNameValuePair("cn", ""));
urlParameters.add(new BasicNameValuePair("locale", ""));
urlParameters.add(new BasicNameValuePair("caller", ""));
urlParameters.add(new BasicNameValuePair("num", "12345"));
post.setEntity(new UrlEncodedFormEntity(urlParameters));
HttpResponse response = client.execute(post);
System.out.println("Response Code : "
+ response.getStatusLine().getStatusCode());
The problem is that the redirect() method of the response initiates a new request altogether, thereby loosing the attributes that were set before redirecting. Luckily there is a fluent way of solving the problem still.
response.setHeader("Key", "Value");
request.getRequestDispatcher("redirecturl").forward(request, response);
Then in your destination you can do
response.getHeaders("key")
You can use JS redirect, i.e. instead of calling sendRedirect return HTML page with embedded javascript that will do redirect setting headers you need.
However, using GET parameters is really the best solution. If you have concerns about users altering parameters manually - use MAC code to protect parameters.See
Message authentication code
In simplest form, ?p1=1&p2=2&mac={mac value}, where {mac value} = md5('MY_SECRET_KEY' + 'p1=1&p2=2').
Receiving side can recalculate MAC and compare it with provided one. Since external users can not know 'MY_SECRET_KEY', they will not be able to make valid MAC.
Have you checked the HTTP request/response from/to server? You can use a number of plugins on chrome or firefox to check that. You would be able to see if value is being passed from your server to another server or not
Also retrieve the header using httpResponse.getHeader("Key"); not using request.getHeader("key"). One of my colleague was facing same issue some days back, he was using request to fetch header values
I am working on a program that will download a facebook page so I have the html. However, when I download it, I get the facebook page that isn't logged in.
Is there a way to somehow send my facebook login so facebook thinks the program is logged in, so I get the correct html?
If you want to get information from Facebook, you should be using the API available to you at https://developers.facebook.com/docs/
. The API provides access to most data that you would want, and it does so with OAuth authentication and JSON responses, which both have a lot of support in the development community (you will be able to find several libraries to handle these types of data without having to code it yourself). There are also samples, SDKs for a number of different programming language, and lots of good information.
Otherwise, if you want the HTML specifically, you can use the components at http://hc.apache.org/httpcomponents-client-ga/index.html
. This is some code that you can use to get a HTTP page...
HttpGet request = new HttpGet("http://www.facebook.com");
// Send the request
DefaultHttpClient client = new DefaultHttpClient();
HttpResponse response = client.execute(request);
// Get the response code
if (response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
// Success
String responseMessage = EntityUtils.toString(response.getEntity());
// do something with the response here
}
If you need to send data, such as sending the login information to Facebook, you can use HttpPost instead of HttpGet, and attach your login information.
You should be warned, however, that Facebook is pretty strict at banning people that use HTTP requests rather than their API, so I would discourage you away from doing this. I direct you to the Facebook Platform Policy here... https://developers.facebook.com/policy/
You should look at the Facebook API: https://developers.facebook.com/
I access a webpage by passing the session id and url and output is a HTML response.
I want to use jSoup to parse this response and get the tag elements.
I see the examples in Jsoup takes a String for establishing connection. How do i proceed.
pseudo code:
I tried the above method and got this exception
java.io.IOException: 401 error loading URL http://www.abc.com/index
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:387)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:364)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:143)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:132)
Basically the entity.getContent() has the HTML response which has to be passed as a String to the connect method. But it doesn't work.
Apache Commons HttpClient and Jsoup do not share the same cookie store. You basically need to pass the very same cookies as HttpClient has retrieved back through Jsoup's Connection. You can find some concrete examples here:
Sending POST request with username and password and save session cookie
how to maintain variable cookies and sessions with jsoup?
Alternatively, you can also just continue using HttpClient for firing HTTP requests and maintaining the cookies and instead feeds its HttpResponse as String through Jsoup#parse().
So this should do:
HttpResponse httpResponse = httpclient1.execute(httpget, httpContext);
String html = EntityUtils.toString(httpResponse.getEntity());
Document doc = Jsoup.parse(html, testUrl);
// ...
By the way, you do not necessarily need to create a whole new HttpClient for a subsequent request. Just reuse httpclient which you already created. Also your way of obtaining the response as String is clumsy. The second line in the above example shows how to do it at simplest.
It shows an http error 401 which means
Similar to 403 Forbidden, but specifically for use when authentication is possible but has failed or not yet been provided.
Therefore, i think you need to login into the website using your java code or identify yourself by sending cookies through your code.