Develop Reference

JAVA POST request and then redirect to it?

What I need to do is send POST request to specific URL with two parameters and when the request is sent, I need to redirect user to that link so that he would be able to access functionality.
So far, what I have managed to do from various examples is this:
private void postRemoteAdvisoryLink() throws IOException {
URL obj = new URL(KdrmApplicationContext.getRemoteAdvisoryUrlPath());
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setConnectTimeout(60000);
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", "Mozilla/5.0");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
// For post only - start
con.setDoOutput(true);
OutputStream os = con.getOutputStream();
os.write(("?auth=ssor&TransportKey=" + ssorTransportKey).getBytes());
os.flush();
os.close();
int responseCode = con.getResponseCode();
}
The problem is that now I get connection time out when trying to execute OutputStream os = con.getOutputStream(); line. Also, I still have no idea how to redirect user when request is completed.
Any ideas?

Using the basic Java URL classes would require you to manually handle the details of HTTP protocol - it's better to use libraries like Apache Http Components, as they deal with the underlying protocols for you. Some examples including POST requests can be found on their website.
Given the original question, the Timeout is likely related to host not responding or your Java application being unable to connect to given URL (due to no proxy configuration for example).
If you want to redirect a request based on the answer, you need to check the response headers and http status - if the status is 302, then there should be a header called Location, which will contain the URL you should make another request to.
Before getting an OutputStream, also make sure to set the Content-Length header (and ideally the Content-Type header as well).

Program can not login into website

This is my code:
URL url = new URL("http://superchillin.com/login2.php");
HttpURLConnection urlConnection = (HttpURLConnection)url.openConnection();
urlConnection.setUseCaches(false);
urlConnection.setRequestMethod("POST");
String data = "email="+URLEncoder.encode(name, "UTF-8")+"&password="+URLEncoder.encode(pass, "UTF-8");
urlConnection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
urlConnection.setRequestProperty("Accept-Encoding", "gzip,deflate");
urlConnection.setRequestProperty("Accept-Language", "en-US,en;q=0.8,lt;q=0.6");
urlConnection.setRequestProperty("Cache-Control", "max-age=0");
urlConnection.setRequestProperty("Connection", "keep-alive");
urlConnection.setRequestProperty("Content-Length", Integer.toString(data.getBytes().length));
urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
urlConnection.addRequestProperty("Cookie", "place=1");
urlConnection.addRequestProperty("Cookie", "lvca_unique_user=1");
urlConnection.setRequestProperty("Host", "superchillin.com");
urlConnection.setRequestProperty("Origin", "http://superchillin.com");
urlConnection.setRequestProperty("Referer", "http://superchillin.com/login.php");
urlConnection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36");
urlConnection.setDoOutput(true);
urlConnection.setDoInput(true);
urlConnection.setInstanceFollowRedirects(true);
DataOutputStream wr = new DataOutputStream(urlConnection.getOutputStream());
wr.writeBytes(data);
wr.flush();
wr.close();
After that code I only read the response. It redirects me to "login.php" and is trying to set cookie "place=1"...
Connecting via browser works great. The reason for so many headers is I thought they may be the problem so I copied all headers from which I see when using a browser.
The response code is 200.
I also noticed that if password or email is incorrect, there's a message saying that in HTML which i retrieve.
When I use a browser I get redirected to index.php and cookie "auth" is set. So that's what I'm expecting from my program aswell. Curently I get redirected back to "login.php".

There is no universal answer to this question, I'm afraid. What you're asking is "why does the remote server not return an auth cookie when I send this exact request?" And that depends entirely on what the server's documentation says about those requests, whether it has any bugs in its implementation, etc.
If you don't have access to the server's own source and logs, then you'll likely have to get by with experimentation. Use something like Firebug or Chrome's Developer Tools to capture the exact requests sent by the browser with the login works successfully. Since these text strings are the only thing the remote server sees, if you replicate them exactly with your Java program you will(/should) get exactly the same responses.
If you think you're sending the same requests from Java and find that you're still not getting the expected responses, there must be some difference. Try recording the network traffic with something like Wireshark in order to see exactly what your app is sending - and then address the differences.
And if you get to the point where e.g. a redirect isn't being followed, and you're not sure how to do that with a URLConnection - then that's a good concrete question to ask.

Java HttpURLConnection | POST method | Sequence of HTTP header fields

I am trying to "spoof" a Firefox HTTP POST request in Java using java.net.HttpURLConnection.
I use Wireshark to check the HTTP headers being sent, so I have (hopefully) reliable source of information, why the Java result doesn't match the ideal situation (using Firefox).
I have set all header fields exactly to the values that Firefox sends via HTTP and noticed, that the sequence of the header fields is not the same.
The output for Firefox is like:
POST ...
**Host**
User-Agent
Accept
Accept-Language
Accept-Encoding
Referer
Connection
Content-Type
Content-Length
When I let wireshark tap off my implementation in Java, it gives me a slightly different sequence of fields:
POST...
**User-Agent**
Accept
Accept-Language
Accept-Encoding
Referer
Content-Type
Host
Connection
Content-Length
So basically, I have all the fields, just in a different order.
I have also noticed that the Host field is sent with a different value:
www.thewebsite.com (Firefox) <---> thewebsite.com (Java HttpURLConnection), although I pass on the String to httpUrlConnection.setRequestProperty with the "www."
I have not yet analyzed the byte output of Wireshark, but I know that the server is not returning the same Location in the header fields of my response.
My questions are:
(1) Is is possible to control the sequence the header fields in the request, and if yes is it possible to do using HttpURLConnection? If not, is it possible to directly control the bytes in the HTTP header using Java? [I don't own the server, so my only hope to get the POST method working is through my application pretending to be Firefox, the server is not really verbose, my only info are: Apache with PHP]
(2) Is there a way to fix the setRequestProperty() problem ("www") as described above?
(3) What else could matter? (Do I need to concern the underlying layers, TCP....?)
Thanks for any comments.
PS. I am trying to model a situation without cookies being sent, so that I can ignore the effect.

First, the order of the headers is irrelevant.
Second, in order to manually override the host header you need to set sun.net.http.allowRestrictedHeaders=true either in code
System.setProperty("sun.net.http.allowRestrictedHeaders", "true")
or at JVM start
-Dsun.net.http.allowRestrictedHeaders=true
This is a security precaution introduced by Oracle a while ago. That's because according to RFC
The Host request-header field specifies the Internet host and port
number of the resource being requested, as obtained from the original
URI given by the user or referring resource (generally an HTTP URL).

the headers order is not important. the headers got by server are also out-of-order. And you can not control httpUrlConnection header order. But if you write your own TCP client, you can control your header order. like:
clientSocket = new Socket(serverHost, serverPort);
OutputStream os = clientSocket.getOutputStream();
String send = "GET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\nGET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\n";
os.write(send.getBytes());
The Second question is answered by Marcel Stör in the first answer.
a

I got lucky with Apache Http Components, my guess is that the "Host" header's missing "www." made the difference, which can be set exactly as intended using Apache's HttpPost:
httpPost.setHeader("Host", "www.thewebsite.com");
The Wireshark output confirmed my suspicion. Also this time the TCP communication prior to my HTTP post looks different (client ---> server, server ---> client, client ---> server) instead of (client ---> server, server ---> client, client ---> server, client---> server).
Now I get the desired Location header value and the server is also setting the cookies. :)
For the most part, this question is resolved.
Actually I wanted to use the lightweihgt HttpUrlConnection because that's what the Android Developers blog suggesting. The System.setProperty("sun.net.http.allowRestrictedHeaders", "true") might work as well, if it allows to "www." in the Host value.

send back compressed JSON in Restlet/GAE

I am writing a Restlet application on GAE similar as described here:
First Application
I am sending back a JSON represntation of an entity, and this works. But I am so far unsuccessful in sending the response compressed.
I tried to add to request an accept-encoding header with "gzip". but that didn't help. Here is how i tested it:
URL url = new URL(address);
URLConnection urlConn = url.openConnection();
urlConn.setRequestProperty("Accept-Encoding", "gzip");
InputStream openStream = urlConn.getInputStream();
Any ideas would be very much appreciated!

I believe you also need to specify the User-Agent header to force the compression. From the docs:
https://developers.google.com/appengine/docs/python/runtime#Responses
If the client sends HTTP headers with the request indicating that the
client can accept compressed (gzipped) content, App Engine compresses
the response data automatically and attaches the appropriate response
headers. It uses both the Accept-Encoding and User-Agent request
headers to determine if the client can reliably receive compressed
responses. Custom clients can force content to be compressed by
specifying both Accept-Encoding and User-Agent headers with a value of
"gzip".

java.io.IOException: Server returned HTTP response code: 403 for URL [duplicate]

This question already has answers here:
403 Forbidden with Java but not web browser?
(4 answers)
Closed 4 years ago.
My code goes like this:
URL url;
URLConnection uc;
StringBuilder parsedContentFromUrl = new StringBuilder();
String urlString="http://www.example.com/content/w2e4dhy3kxya1v0d/";
System.out.println("Getting content for URl : " + urlString);
url = new URL(urlString);
uc = url.openConnection();
uc.connect();
uc.getInputStream();
BufferedInputStream in = new BufferedInputStream(uc.getInputStream());
int ch;
while ((ch = in.read()) != -1) {
parsedContentFromUrl.append((char) ch);
}
System.out.println(parsedContentFromUrl);
However when I am trying to access the URL through browser there is no problem , but when I try to access it through a java program, it throws expection:
java.io.IOException: Server returned HTTP response code: 403 for URL
What is the solution?

Add the code below in between uc.connect(); and uc.getInputStream();:
uc = url.openConnection();
uc.addRequestProperty("User-Agent",
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");
However, it a nice idea to just allow certain types of user agents. This will keep your website safe and bandwidth usage low.
Some possible bad 'User Agents' you might want to block from your server depending if you don't want people leeching your content and bandwidth. But, user agent can be spoofed as you can see in my example above.

403 means forbidden. From here:-
10.4.4 403 Forbidden
The server understood the request, but
is refusing to fulfill it.
Authorization will not help and the
request SHOULD NOT be repeated. If the
request method was not HEAD and the
server wishes to make public why the
request has not been fulfilled, it
SHOULD describe the reason for the
refusal in the entity. If the server
does not wish to make this information
available to the client, the status
code 404 (Not Found) can be used
instead.
You need to contact the owner of the site to make sure the permissions are set properly.
EDIT I see your problem. I ran the URL through Fiddler. I noticed that I am getting a 407 which means below. This should help you go in the right direction.
10.4.8 407 Proxy Authentication Required
This code is similar to 401
(Unauthorized), but indicates that the
client must first authenticate itself
with the proxy. The proxy MUST return
a Proxy-Authenticate header field
(section 14.33) containing a challenge
applicable to the proxy for the
requested resource. The client MAY
repeat the request with a suitable
Proxy-Authorization header field
(section 14.34). HTTP access
authentication is explained in "HTTP
Authentication: Basic and Digest
Access Authentication"
Also see this relevant question.
java.io.IOException: Server returned HTTP response code: 403 for URL

IF the browser can access the page, and your code cannot, then there's something different between the browser request and your request. You can look at the browser request, using, say, Firebug, to see what the differences are. Some things I can think of are:
The site sets a
cookie (maybe during login). You may be able to handle
this in code, you will have to
explicitly add support for passing
the cookie. This is most likely.
The site filters based on user agents. You can set the user agent. This is not as likely.