I have an Android client making HTTPS request to a server. The firewall logs have entries for CONNECT request methods which are not desired.
When would the CONNECT request get sent out and how can I prevent it from being sent? I expect only a GET request. My understanding is that the call to openConnection() does not actually make a request and that the GET request would go on the call to getResponseMessage().
How can I disable the http client from attempting to establish a proxy tunnel?
Here is how I sent up my connection and make the call:
URL url = new URL("https://some.server.com");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setConnectTimeout(CONNECT_TIMEOUT);
connection.setReadTimeout(REAT_TIMEOUT);
connection.setRequestProperty(ACCEPT_CHARSET, UTF8_CHARSET);
connection.setRequestProperty(CONTENT_TYPE_HEADER, contentType);
setCustomRequestProperties(connection);
connection.setRequestMethod("GET");
//create response
connection.getResponseMessage();
connection.getResponseCode();
connection.getContentType();
connection.getContent();
EDIT:
Here is the Firewall Log entry that I am trying to prevent:
CONNECT someURL.domain.com:443 HTTP/1.1
Host: someURL.domain.com
User-Agent: CustomUserAgent;1.0.0(Android 4.3;Nexus 4;T-Mobile)
Proxy-Connection: Keep-Alive
X-SSL-Secure: true
X-CS-Source-IP: 10.3.3.3
X-Forwarded-For: 10.3.3.3
I thought this was proxy related because of the "Proxy-Connection" header
The default for URLConnection is to pool the TCP (socket) connections. It seems that the "Proxy-Connection" header was misleading as that header can be misappropriated when "Connection" is really intended.
When I set the system property
System.setProperty("http.keepAlive", "false");
to disable the connection pool, the CONNECT method request entries went away.
There is a slight performance hit since Sockets need to be created for each request but the app does not make a significant number of requests so this is acceptable.
This only happens when you have an HTTP proxy configured. Possibly you have set the system properties http.proxyHost and http.proxyPort, or maybe Android has another way of configuring this.
Related
I want to send 2 kinds of request via proxy: http and https. Here's how I'm doing an HTTP request:
HttpURLConnection conn = (HttpURLConnection) url.openConnection(proxy);
and HTTPS:
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection(proxy);
The rest of the code is identical:
//..................
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("ip", 1234));
//..................
conn.setRequestProperty("Proxy-Connection", "Keep-Alive");
conn.setRequestProperty("Proxy-Authorization", auth);
conn.setDoInput(true);
System.out.println("Response Code : " + conn.getResponseCode());
System.out.println("\n");
In case of HTTP it returns 200 and the body of the response which means everything is working fine. However, for HTTPS requests it returns:
Unable to tunnel through proxy. Proxy returns "HTTP/1.0 407 Proxy Authentication Required
How come?
I don't consider using the class Authenticator or System.setProperty. I want to figure out why my code isn't working correctly for https and is for http.
You use Basic authentication schema.
According Oracle update # 8u111
In some environments, certain authentication schemes may be
undesirable when proxying HTTPS. Accordingly, the Basic authentication
scheme has been deactivated, by default, in the Oracle Java Runtime,
by adding Basic to the jdk.http.auth.tunneling.disabledSchemes
networking property.
Thus you need clear list of disabledSchemes by
- or invoke at main static method System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");
- or add JVM parameter -Djdk.http.auth.tunneling.disabledSchemes=
- or use Apache Common HttpClient
I want to use HttpURLConnection to connect to my webservice, POST an XML and get a result. I am using the following code:
URL url = new URL(urlString);
connection = (HttpURLConnection)url.openConnection();
connection.setRequestProperty("Content-Type", "text/xml; charset=utf-8");
Problem is, when I call the setRequestProperty method, it fails with an IllegalStateException saying I am "already connected". Apparently, openConnection in fact opens the connection to the URL (in my debugger, I can see the connected boolean set to true). According to the URL documentation of Oracle though, it shouldn't. The Android docs are unclear about it.
How do I prevent openConnection to connect, so I can set extra properties?
Update it looks like the connection is in some pool and doesn't get disconnected, not even after calling connection.disconnect(), or even killing the server.
I do not think this is a duplicate of this question as it gives no real answer. Also, the documentation seems to be unclear.
openConnection() does not connect, and the code you have posted does not behave as yu have described. What opens the TCP connection is any of the following:
getInputStream()
getErrorStream()
getResponseCode()
Ergo you must have called one of those before trying to set a request property.
I have HttpURLConnection instance created from URL and also I set query parameters and called some setters on this HttpURLConnection instance. I use this instance to get response from web service.
Is there some way to get the HTTP request string that will be sent over the network when using the given HttpURLConnection instance ? (just for debugging purposes). Can we do this programatically using HttpURLConnection or if it's not possible how can I monitor the outgoing HTTP traffic ?
The reason I need this that in some cases it can be easier to detect what is wrong with your configuration of HttpURLConnection by looking directly at the request that is defined by this instance than trying to figure out what is wrong with particular configuration of HttpURLConnection by checking what setters was called.
Thank you for any suggestion.
You can monitor your http Traffic by using Fiddler
Here is the download link
I'm getting a too many redirects redirect error from URLConnection when trying to fetch www.palringo.com
URL url = new URL("http://www.palringo.com/");
HttpURLConnection.setFollowRedirects(true);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
System.out.println("Response code = " + connection.getResponseCode());
outputs the dreaded:
Exception in thread "main" java.net.ProtocolException: Server redirected too many times (20)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
According to wget there is just one redirect, from www.palringo.com to www.palringo.com/en/gb/
Any ideas why my request using URLConnection for /en/gb results in another 302 response for the same resource ?
The problem is exemplified by:
URL url = new URL("http://www.palringo.com/en/gb/");
HttpURLConnection.setFollowRedirects(false);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
// Just for testing, use Chrome header, to eliminate "anti-crawler" response!
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.30 (KHTML, like Gecko) Ubuntu/11.04 Chromium/12.0.742.112 Chrome/12.0.742.112 Safari/534.30");
System.out.println("Response code = " + connection.getResponseCode());
This outputs:
Response code = 302
Redirected to /en/gb/
hence an infinite redirect loop.
Interestingly although browsers and wget handle it, curl does not:
joel#bohr:/tmp$ curl http://www.palringo.com/en/gb/
curl: (7) couldn't connect to host
A request for /en/gb/ is redirected to /en/gb/ precisely once.
The problem is that your HttpURLConnection (or whatever code you use -- sorry, I'm NOT familiar with Java) does not use cookies.
Disable cookies in browser and observe exactly the same behaviour -- infinite redirect.
The reason: Server checks if cookie is set. If not set -- it sets it and redirects. Because cookies are not supported/disabled, script on server side redirects over and over again.
Solution: Enable/add cookie support to your code and try again.
I think that redirect is defined with pattern like /* -> /en/gb
So, when you arrive to /en/gb the redirect rule works again.
Check your redirect rules. Where are they defined? In apache web server or in other place? Check all. Verify that this is (or is not) a case and fix the rules accordingly.
The problem is on the server side. It might be a broken Apache httpd rewrite rule that is sending redirects that loop back to the same place. It might be something else. Whatever it is, you are unlikely to be able to fix it on the client side.
I'm basically running a crawler and just noticed this issue.
Ah.
It is possible that it is an anti-crawler defence measure. "Hmmm ... looks like one of those pesky crawlers who ignore my robots.txt file, waste all of my bandwidth and steal my precious content. Lets cause him some pain with a redirect loop!!".
Check that your crawler is obeying the "robots.txt" protocol. Check the ToS for the site you are crawling to see if what you are doing is allowed.
You could be right, but if so how come wget and browsers handle this with just the one redirect?
Maybe because the server is looking at the request headers, or at your pattern of requests.
The Terms of Service (that I see) say this:
"You agree to not use the Service to: ... xiii - Run any automated systems, processes, scripts or bots for any purpose without the express written permission of Palringo."
Arguably, crawling their site is in violation of that.
You will also get this error if you're trying to connect to a service that requires authentication and you provide wrong username and password.
My code is like the following:
URLConnection cnx = address.openConnection();
cnx.setAllowUserInteraction(false);
cnx.setDoOutput(true);
cnx.addRequestProperty("User-Agent",
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");
InputStream is = cnx.getInputStream();
Is it ok if I set the headers before I get the InputStream? Will my header be sent, or will the server see the default URLConnection's user-agent ( if any ) ?
The headers must be set prior to getting the InputStream to have any affect - an IllegalStateException will be thrown if the connection is already open.
As far as the User-Agent header specifically, it should be sent if it has been set.
See the URLConnection JavaDoc.
To answer the question, the code is correct. The moment getInputStream(), an HTTP get is sent to the target server.
A side-note on user-agent, if you don't set it, URLConnection will send the default one anyway, which is:
User-Agent: Java/1.6.0_24 (varies depending on your java version)
I'd advise against using low-level constructs such as URLConnection. There are plenty of libraries for sending HTTP requests, with the most prominent being Apache HTTP Client.