I am trying to "spoof" a Firefox HTTP POST request in Java using java.net.HttpURLConnection.
I use Wireshark to check the HTTP headers being sent, so I have (hopefully) reliable source of information, why the Java result doesn't match the ideal situation (using Firefox).
I have set all header fields exactly to the values that Firefox sends via HTTP and noticed, that the sequence of the header fields is not the same.
The output for Firefox is like:
POST ...
**Host**
User-Agent
Accept
Accept-Language
Accept-Encoding
Referer
Connection
Content-Type
Content-Length
When I let wireshark tap off my implementation in Java, it gives me a slightly different sequence of fields:
POST...
**User-Agent**
Accept
Accept-Language
Accept-Encoding
Referer
Content-Type
Host
Connection
Content-Length
So basically, I have all the fields, just in a different order.
I have also noticed that the Host field is sent with a different value:
www.thewebsite.com (Firefox) <---> thewebsite.com (Java HttpURLConnection), although I pass on the String to httpUrlConnection.setRequestProperty with the "www."
I have not yet analyzed the byte output of Wireshark, but I know that the server is not returning the same Location in the header fields of my response.
My questions are:
(1) Is is possible to control the sequence the header fields in the request, and if yes is it possible to do using HttpURLConnection? If not, is it possible to directly control the bytes in the HTTP header using Java? [I don't own the server, so my only hope to get the POST method working is through my application pretending to be Firefox, the server is not really verbose, my only info are: Apache with PHP]
(2) Is there a way to fix the setRequestProperty() problem ("www") as described above?
(3) What else could matter? (Do I need to concern the underlying layers, TCP....?)
Thanks for any comments.
PS. I am trying to model a situation without cookies being sent, so that I can ignore the effect.
First, the order of the headers is irrelevant.
Second, in order to manually override the host header you need to set sun.net.http.allowRestrictedHeaders=true either in code
System.setProperty("sun.net.http.allowRestrictedHeaders", "true")
or at JVM start
-Dsun.net.http.allowRestrictedHeaders=true
This is a security precaution introduced by Oracle a while ago. That's because according to RFC
The Host request-header field specifies the Internet host and port
number of the resource being requested, as obtained from the original
URI given by the user or referring resource (generally an HTTP URL).
the headers order is not important. the headers got by server are also out-of-order. And you can not control httpUrlConnection header order. But if you write your own TCP client, you can control your header order. like:
clientSocket = new Socket(serverHost, serverPort);
OutputStream os = clientSocket.getOutputStream();
String send = "GET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\nGET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\n";
os.write(send.getBytes());
The Second question is answered by Marcel Stör in the first answer.
a
I got lucky with Apache Http Components, my guess is that the "Host" header's missing "www." made the difference, which can be set exactly as intended using Apache's HttpPost:
httpPost.setHeader("Host", "www.thewebsite.com");
The Wireshark output confirmed my suspicion. Also this time the TCP communication prior to my HTTP post looks different (client ---> server, server ---> client, client ---> server) instead of (client ---> server, server ---> client, client ---> server, client---> server).
Now I get the desired Location header value and the server is also setting the cookies. :)
For the most part, this question is resolved.
Actually I wanted to use the lightweihgt HttpUrlConnection because that's what the Android Developers blog suggesting. The System.setProperty("sun.net.http.allowRestrictedHeaders", "true") might work as well, if it allows to "www." in the Host value.
Related
In order to send a string data to the server once, I do as below:
Make “HttpURLConnection” to my URL address and open it
Set the required headers
for the connection I Set setDoOutput to True
new a DataOutputStream from my connection and finally write my string data to it.
HttpURLConnection myConn = (HttpURLConnection);
myUrl.openConnection();
myConn.setRequestProperty("Accept", "application/json, text/plain, */*");
myConn.setDoOutput(true);
DataOutputStream my_output = new DataOutputStream(myConn.getOutputStream());
my_output.write(myData.getBytes("UTF-8"));
But what to do if I want to send exactly the same data with same URl and headers multiple times?
Can I write to it multiple times?(I mean that is it possible to use the last line of code multiple times?) Or should I repeat the above steps and try it with a new connection?
And if yes should I wait for some second or millisecond before sending the next one?
I also searched for some other alternatives such as “HttpClient” Http API and making synchronous Http request which as far as I got can help me setting the headers only once.
At the end, I appreciate your help and support and any other alternatives would be welcome.
Thanks a million.
I understand that the question has be answered in the comments, but I am leaving this here so that future viewers can see it.
An HTTP request contains 3 main parts:
Request Line: Method, Path, Protocol
Headers: Key-Pairs
Body: Data
Running my_output.write() will just add bytes to the body until my_output.flush() has been executed. Flushing the stream will write the data to the server.
Because HTTP requests are usually closed by the server once all data has been sent/received, whether or not you create a new connection or just add on to the body depends on your intentions. Typically, clients will create a new connection for each request because each response should be handled individually, rather than sending a repetitive body. This will vary though because some servers choose to hold a connection (such as WebSockets).
If you are open to external libraries, you may find this chart insightful:
AsyncHttpClient would be a good fit for your intentions.
Alternatively, you can use cURL by running a terminal command with Runtime.getRuntime().exec(). More information about using cURL with POST Requests can be found here. While cURL is efficient, you have to depend on the fact that your OS supports the command (though usually all devices that can run Java have this command).
The application I am working on works in a way where the request is sent from the browser to server A, server A makes some modifications to the request and forwards it to server B. Server B returns the response to server A which in turn returns it to the browser(Browser <=> Server A <=> Server B).
At server A I need to modify/remove a particular header "access-control-allow-origin" from the response given by server B. I can't do that at B where the response is created since I don't have access to the code. I need to do this at A itself.
I tried using the HttpServletResponseWrapper( How do delete a HTTP response header? ) but I think it is useful only when the headers are not set already. In my case the headers are set and need to be modified/removed.
Try this:
((org.apache.catalina.connector.Response)response).getCoyoteResponse()
.getMimeHeaders()
.removeHeader("access-control-allow-origin");
getCoyoteResponse() gets the underlying Response object which is where the headers are actually stored in a MimeHeaders object. MimeHeaders has a convenient removeHeader method, among others.
Note that I didn't actually test this, but it looks like it should work. Also, I found this in Tomcat 8.0.26 source, YMMV with a different version.
I'm in the process of learning how to use HP Quality Center's REST api to query and manipulate data. Unlike REST standard, this API is not completely stateless. It uses cookies to store authentication sessions.
I've tried to implement a very simple test, using the Jersey Client library. I can successfully authenticate my user, by sending my credentials. The API reference claims that this will set a cookie, and I am good to go with further calling the REST api. However, a simple "is-authenticated" call returns a 401, Authentication failed.
I have a feeling that the cookie writing or reading is not working properly, as everything else seems to work as it should. But I haven't been able to find out if or how cookies are set and read, when no browser is involved. So How does cookies work, when calling cookie-setting REST services from java VM? Does it work at all? Where are they stored?
I am using Eclipse Kepler as my IDE, if that matters at all, and a 32-bit java 1.6 JDK and JRE.
Code, and response strings below:
1. Logging in:
Client client = ClientBuilder.newClient();
Response response = client
.target("http://[host]:[port]").path("qcbin/authentication-
point/alm-authenticate")
.request().post(Entity.entity("<alm-authentication>
<user>username</user>
<password>secret</password></alm-authentication>",
MediaType.TEXT_XML_TYPE));
System.out.println(response.toString());
Output:
InboundJaxrsResponse{ClientResponse{method=POST,
uri=http://[host]:[port]/qcbin/authentication-point/alm-authenticate,
status=200, reason=OK}}
API Return description:
One of:
HTTP code 200 and sets the LWSSO cookie (LWSSO_COOKIE_KEY).
HTTP code 401 for non-authenticated request. Sends header
WWW-Authenticate: ALMAUTH
2. Verifying Logged in:
response = client.target("http://[host]:[port]")
.path("qcbin/rest/is-authenticated")
.request().get();
System.out.println(response.toString());
Output:
InboundJaxrsResponse{ClientResponse{method=GET,
uri=http://[host]:[port]/rest/is-authenticated, status=401,
reason=Authentication failed. Browser based integrations - to login append
'?login-form-required=y to the url you tried to access.}}
PS: adding the ?login-form-required=y to the URL, will bring up a log-in window when called in a browser, but not here. Appending the line to the URL actually still gives the same error message, and suggestion to append it again. Also, when called in a browser, the is-authenticated returns a 200, success, even without the login-form.
When you log in, you're getting a cookie which is a name plus a value.
The REST server expects you to pass this in the request header with every request you make.
Look into the object which you get for client.request(); there should be a way to specify additional headers to send to the server. The header name must be Cookie and the header value must be name=value.
So if the server responds with a cookie called sessionID with the value 1234, then you need something like:
client.request().header("Cookie", "sessionID=1234")
Related:
http://en.wikipedia.org/wiki/HTTP_cookie
I am talking to a file upload service that accepts post data, not form data. By default, java's HttpURLConnection sets the Content-Type header to application/x-www-form-urlencoded. this is obviously wrong if i'm posting pure data.
I (the client) don't know the content type. I don't want the Content-Type header set at all. the service has a feature where it will guess at the content type (based on the file name, reading some data from the file, etc).
How do I unset a header? There's no remove header, and setting it to null doesn't change the value and setting it to the empty string results in the header being set with no value.
I haven't tested this approach but you can try this:
Extend HttpURLConnection and try by overriding its getContentHandler() and setContentHandler(...) methods. Most probably this should work as, you will take a look at code of getContentHandler().
Use Apache HttpClient instead of URLConnection
Use fluent Request to generate your request
use removeHeader()
What do you mean "i don't want the Content-Type header to set at all"?
The browser (or other http client) sends your post request to the server, so it has to inform the server which way it encoded the parameters.
If the Content-Type header is not set, on the server side you (= your server) won't be able to understand how to parse the received data.
If you didn't set Content-Type, the default value will be used.
You browser (or other http client) MUST do two things:
Send key/value pairs.
Inform the server how the key/value pairs were encoded.
So, it is impossible to completely get rid of this header.
I just accomplished this by setting the header to null.
connection.setRequestProperty(MY_HEADER, null);
Is there a way to use addRequestProperty if you have already connected to the URLConnection?
I need to do this cause I first have to connect to get the headerfields for the first Cookie, but later I have to send that first Cookie in order to get headerfields where a second set-cookie field is defined.
That won't work. The HTTP protocol conveys those properties as header items at the beginning of the request data. You'll need multiple requests.