I want to develop an android app to get post.xml with HttpClient. But it failed to get content with 80 port.
If I start the web server(WEBrick here) with 3000 port, the URI is http://192.168.1.103:3000/posts.xml;
Android App can get response with correct length, like 568;
The same web files, I started them with another server (Nignx here) with 80 port, the uri is
"http://192.168.1.103/posts.xml; The Android App can NOT get content with length, it's -1 here.
This URI can be opened with browser(both PC and android emulator) correctly. Furthermore, the response is "HTTP/1.1 200 OK" with responsep.getStatusLine().
is it related with "Socket ports below 1024 can NOT access in linux like system", which is on
http://groups.google.com/group/android-developers/browse_thread/thread/660123ca64ba1229#
Any Ninja can tell me what should I do if I can to get content with 80 port?
The following is my code.
public class AndroidWorldActivity extends Activity {
/** Called when the activity is first created. */
TextView tv;
#Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
retreiveProjects();
}
private void retreiveProjects()
{
HttpClient httpClient = new DefaultHttpClient();
try
{
String url3000 = "http://192.168.1.103:3000/posts.xml";
String url = "http://192.168.1.103/posts.xml";
Log.d( "posts", "performing get " + url3000);
HttpGet httpGet=new HttpGet(url3000);
HttpResponse responsep=httpClient.execute(httpGet);
System.out.println(responsep.getStatusLine());
HttpEntity httpEntity = responsep.getEntity();
int length = ( int ) httpEntity.getContentLength();
// print the lenght of content
System.out.println("The content length is: "+length);
Log.d( "posts", "The content length is: " + length );
From your description, I understand that you are trying to connect from adroid to an external HTTP server attached to port 80? If so, restriction about port lower than 1024 on android has nothing to do (you are not trying to listen on port 80 on android device). I think, that you have a problem with Nginx.
Try to execute GET request from an external machine to Nginx and investigate response content (headers, payload). I would recommend to do it with some more low-level tool instead of web browser (almost all web browser nowadays are able to "repair" illegal server responses), for example curl:
curl -D - http://192.168.1.103/posts.xml
You seem to have two separate problems.
The problem on WeBrick seems to be that the UNIX / Linux won't allow your web server to bind to port 80. There are two obvious things to check:
Is the web server running with root privilege when it attempts to bind to the port? See Is there a way for non-root processes to bind to "privileged" ports on Linux? for a variety of ways to work around this problem.
Is some other application already bound to port 80? Is so, kill it and try to run your web server again.
The problem with Nignx is different. Here, the server is up and running and giving your client responses, but the client is seeing -1 as the content length.
This is normal behaviour. The getContentLength() method returns -1 if the response doesn't have a "Content-length" header, and it is perfectly legitimate (according to the HTTP specification) for a response to not have this header. You have two choices:
Change your client-side application to deal with the case where the content length is unspecified; e.g. just read the body into a buffer and count how many bytes you got.
Change the server to set the relevant header.
FOLLOWUP
I see. Your original question was hard to understand and I misinterpreted. You seemed to be saying that WEBrick wasn't working at all.
The difference between WEBrick and Nginx is that they simply implement the response differently. Both are legitimate (valid) implementation. The real problem is that your application is assuming that a web server will always set the "Content-length" header. That is an incorrect assumption.
To repeat, the problem / fault is in your client code, not in Nginx.
Related
In my web-app I need to register http clients accessing from a local network behind a router.
I started with remoteHost : remotePort combination, but soon enough it became clear, that the port numer gets regenereated upon each connection.
I need to be able to identify the clients on something similar to MAC address, some property that doesn't change. I wanted to use headers[ "X-Forwarded-For" ], but it's not present at all:
[Pragma=no-cache, Cache-Control=no-cache, Host=somhost.com:8822, Upgrade=websocket, Connection=Upgrade, Sec-WebSocket-Key=scnlM7hzjjy3cklJhJciA==, Sec-WebSocket-Extensions=x-webkit-deflate-frame,deflate-frame, Sec-WebSocket-Version=13]
What are the other options to identify clients?
You could use an API key, that is, a unique identifier that the clients send along with each request to identify themselves. Depending on the authentication method you are using, you could consider the standard HTTP Authorization header to send this value:
Authorization: API-Key <value goes here>
Or create a custom HTTP header for this purpose. But be careful with custom headers: proxies might strip them out.
One option is using cookies. As the client accesses the webapp for the first time we could set a cookie on the client side that has a very long expiry date.
During the subsequent user re-logins we can rely on this cookie as cookies get sent to the server.
You can try this bit of PHP to see what the server knows about an incoming http request:
$keys = array_keys($_SERVER);
echo "<table bgcolor='black' cellpadding='1' cellspacing='1'>\n";
echo " <tr bgcolor='yellow'><td><b>Key</b></td><td><b>Value</b></td></tr>\n";
foreach ($keys as $key) {
echo " <tr bgcolor='white'><td>" . $key . "</td><td>" . $_SERVER[$key] . "</td></tr>\n";
}
echo "</table>\n";
Are you identifying the user at the keyboard or the device making the request? Do you need to track these long term or only for the duration of a use session? Do your users connect from multiple devices?
Client side id certificates could work, depending on how the local machines are managed. If they are accessing your app from someplace they've already authenticated, then setting up a single sign on solution could work. Prompting for authentication always works too.
I have a 404 status error (page not found). I only want to send a request from my Android app to Mean.io web app through
the following url:
http://192.168.0.103:3000/auth/register
I have also tried:
http://10.0.2.2:3000/auth/register
I had already googled but both of the solutions above didn't worked for me. However the url: http://192.168.0.103:3000/auth/register does work
on my Chrome browser on my pc.
Here is the code:
public class AppConfig {
// Server user register url
//public static String URL_REGISTER = "http://10.0.2.2:3000/auth/register";
public static String URL_REGISTER = "http://192.168.0.103:3000/auth/register";
}
If you want to know where the variable URL_REGISTER gets used. It's getting used in the registerUser() method.
I'm posting the method through a link, because the method is too big to post it here. In the link below you can see that the URL_REGISTER gets used on line 10.
Link: http://pastebin.com/ttH6upnb
1 be sure you connect to the server
192.168 and 10.0 are local addresses (not going to internet)
beware, if you get 404, perhaps another server like proxy responds to you
2 read this: Using java.net.URLConnection to fire and handle HTTP requests
3 begin by getting page "/" and check the headers (good server, etc.)
4 then verify your code, step by step
5 check if GET or POST, and authentication is not easy (check the headers)
The question is self explanatory, I hope. I am setting up a Spring Security enviroment with a CAS-server. Because the exact same application is deployed on the same server, but the server is accessible via different host names (.de domain, .com domain, possibly more than that) and we want to deploy the same application on test systems and the local one as well, I built a dynamic service, where the service URL is derived from request URL.
public static String makeDynamicUrlFromRequest(ServiceProperties serviceProperties, HttpServletRequest request) {
String serviceUrl = "https://backup-url.de/login";
URI uri = null;
try {
uri = new URI(request.getRequestURL().toString());
} catch (URISyntaxException e) {
logger.error("Someone tried accessing a disallowed service!", e);
}
if(uri != null){
serviceUrl = uri.getScheme() + "://" + uri.getHost() + "/login";
}
return serviceUrl;
}
Is it possible to spoof this? If it is, does an additional regex-check provide me with the necessary security against this?
#developerwjk
"If they modified the request url how would you have gotten the request?"
An HTTP server is just a program that listens on a TCP port, waits for some incoming text and writes out some text as a response. (A trivial web server can be written in like 20 lines of code.) It only sees the IP address and port of whatever connected to it. That could even be a proxy, or some other sort of middle-ware. If you don't tell the program "by the way, I reached you through the URL http://my.com/myapp/servlet" then it just doesn't know e.g. how a browser will reach it.
#Schaka
I don't know about your particular setup, but for jetty9, the result of getRequestURL is determined from the request URL in the request header, and - if the former is missing - the URL in the Host parameter. That is, if you connect to my.com and send the following request:
POST http://spoofed1.tld/myapp/servlet HTTP/1.1
Host: spoofed2.tld
(Keep in mind that the Host parameter is mandatory.)
Then getRequestURL will return http://spoofed1.tld/myapp/servlet
And if you send this:
POST /myapp/servlet HTTP/1.1
Host: spoofed2.tld
Then jetty itself will respond with
HTTP/1.1 302 Found
Location: http://spoofed2.tld/myapp/servlet
Content-Length: 0
Server: Jetty(<some version number>)
So the answer is yes, HttpServletRequest.getRequestURL() is spoofable! by modifying the request URL and/or the Host request header.
I am trying to "spoof" a Firefox HTTP POST request in Java using java.net.HttpURLConnection.
I use Wireshark to check the HTTP headers being sent, so I have (hopefully) reliable source of information, why the Java result doesn't match the ideal situation (using Firefox).
I have set all header fields exactly to the values that Firefox sends via HTTP and noticed, that the sequence of the header fields is not the same.
The output for Firefox is like:
POST ...
**Host**
User-Agent
Accept
Accept-Language
Accept-Encoding
Referer
Connection
Content-Type
Content-Length
When I let wireshark tap off my implementation in Java, it gives me a slightly different sequence of fields:
POST...
**User-Agent**
Accept
Accept-Language
Accept-Encoding
Referer
Content-Type
Host
Connection
Content-Length
So basically, I have all the fields, just in a different order.
I have also noticed that the Host field is sent with a different value:
www.thewebsite.com (Firefox) <---> thewebsite.com (Java HttpURLConnection), although I pass on the String to httpUrlConnection.setRequestProperty with the "www."
I have not yet analyzed the byte output of Wireshark, but I know that the server is not returning the same Location in the header fields of my response.
My questions are:
(1) Is is possible to control the sequence the header fields in the request, and if yes is it possible to do using HttpURLConnection? If not, is it possible to directly control the bytes in the HTTP header using Java? [I don't own the server, so my only hope to get the POST method working is through my application pretending to be Firefox, the server is not really verbose, my only info are: Apache with PHP]
(2) Is there a way to fix the setRequestProperty() problem ("www") as described above?
(3) What else could matter? (Do I need to concern the underlying layers, TCP....?)
Thanks for any comments.
PS. I am trying to model a situation without cookies being sent, so that I can ignore the effect.
First, the order of the headers is irrelevant.
Second, in order to manually override the host header you need to set sun.net.http.allowRestrictedHeaders=true either in code
System.setProperty("sun.net.http.allowRestrictedHeaders", "true")
or at JVM start
-Dsun.net.http.allowRestrictedHeaders=true
This is a security precaution introduced by Oracle a while ago. That's because according to RFC
The Host request-header field specifies the Internet host and port
number of the resource being requested, as obtained from the original
URI given by the user or referring resource (generally an HTTP URL).
the headers order is not important. the headers got by server are also out-of-order. And you can not control httpUrlConnection header order. But if you write your own TCP client, you can control your header order. like:
clientSocket = new Socket(serverHost, serverPort);
OutputStream os = clientSocket.getOutputStream();
String send = "GET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\nGET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\n";
os.write(send.getBytes());
The Second question is answered by Marcel Stör in the first answer.
a
I got lucky with Apache Http Components, my guess is that the "Host" header's missing "www." made the difference, which can be set exactly as intended using Apache's HttpPost:
httpPost.setHeader("Host", "www.thewebsite.com");
The Wireshark output confirmed my suspicion. Also this time the TCP communication prior to my HTTP post looks different (client ---> server, server ---> client, client ---> server) instead of (client ---> server, server ---> client, client ---> server, client---> server).
Now I get the desired Location header value and the server is also setting the cookies. :)
For the most part, this question is resolved.
Actually I wanted to use the lightweihgt HttpUrlConnection because that's what the Android Developers blog suggesting. The System.setProperty("sun.net.http.allowRestrictedHeaders", "true") might work as well, if it allows to "www." in the Host value.
I am trying to read a website using the java.net package classes. The site has content, and i see it manually in html source utilities in the browser. When I get its response code and try to view the site using java, it connects successfully but interprets the site as one without content(204 code). What is going on and is it possible to get around this to view the html automatically.
thanks for your responses:
Do you need the URL?
here is the code:
URL hef=new URL(the website);
BufferedReader kj=null;
int kjkj=((HttpURLConnection)hef.openConnection()).getResponseCode();
System.out.println(kjkj);
String j=((HttpURLConnection)hef.openConnection()).getResponseMessage();
System.out.println(j);
URLConnection g=hef.openConnection();
g.connect();
try{
kj=new BufferedReader(new InputStreamReader(g.getInputStream()));
while(kj.readLine()!=null)
{
String y=kj.readLine();
System.out.println(y);
}
}
finally
{
if(kj!=null)
{
kj.close();
}
}
}
Suggestions:
Assert than when manually accessing the site (with a web browser client) you are effectively getting a 200 return code
Make sure that the HTTP request issued from the automated (java-based) logic is similar/identical to that of what is sent by an interactive web browser client. In particular, make sure the User-Agent is identical (some sites purposely alter their responses depending on the agent).
You can use a packet sniffer, maybe something like Fiddler2 to see exactly what is being sent and received to/from the server
I'm not sure that the java.net package is robot-aware, but that could be a factor as well (can you check if the underlying site has robot.txt files).
Edit:
assuming you are using the java.net package's HttpURLConnection class, the "robot" hypothesis doesn't apply.
On the other hand you'll probably want to use the connection's setRequestProperty() method to prepare the desired HTTP header for the request (so they match these from the web browser client)
Maybe you can post the relevant portions of your code.