I would like to migrate from Commons HttpClient (3.x) to HttpComponents Client (4.x) but having difficulty how to handle redirects. The code works properly under Commons HttpClient but breaks when migrated to HttpComponents Client. Some of the links get undesirable redirects but when I set "http.protocol.handle-redirects" to 'false' a large number links stop working altogether.
Commons HttpClient 3.x:
private static HttpClient httpClient = null;
private static MultiThreadedHttpConnectionManager connectionManager = null;
private static final long MAX_CONNECTION_IDLE_TIME = 60000; // milliseconds
static {
//HttpURLConnection.setFollowRedirects(true);
CookieManager manager = new CookieManager();
manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(manager);
connectionManager = new MultiThreadedHttpConnectionManager();
connectionManager.getParams().setDefaultMaxConnectionsPerHost(1000); // will need to set from properties file
connectionManager.getParams().setMaxTotalConnections(1000);
httpClient = new HttpClient(connectionManager);
}
/*
* Retrieve HTML
*/
public String fetchURL(String url) throws IOException{
if ( StringUtils.isEmpty(url) )
return null;
GetMethod getMethod = new GetMethod(url);
HttpClient httpClient = new HttpClient();
//configureMethod(getMethod);
//ObjectInputStream oin = null;
InputStream in = null;
int code = -1;
String html = "";
String lastModified = null;
try {
code = httpClient.executeMethod(getMethod);
in = getMethod.getResponseBodyAsStream();
//oin = new ObjectInputStream(in);
//html = getMethod.getResponseBodyAsString();
html = CharStreams.toString(new InputStreamReader(in));
}
catch (Exception except) {
}
finally {
try {
//oin.close();
in.close();
}
catch (Exception except) {}
getMethod.releaseConnection();
connectionManager.closeIdleConnections(MAX_CONNECTION_IDLE_TIME);
}
if (code <= 400){
return html.replaceAll("\\s+", " ");
} else {
throw new Exception("URL: " + url + " returned response code " + code);
}
}
HttpComponents Client 4.x :
private static HttpClient httpClient = null;
private static HttpParams params = null;
//private static MultiThreadedHttpConnectionManager connectionManager = null;
private static ThreadSafeClientConnManager connectionManager = null;
private static final int MAX_CONNECTION_IDLE_TIME = 60000; // milliseconds
static {
//HttpURLConnection.setFollowRedirects(true);
CookieManager manager = new CookieManager();
manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(manager);
connectionManager = new ThreadSafeClientConnManager();
connectionManager.setDefaultMaxPerRoute(1000); // will need to set from properties file
connectionManager.setMaxTotal(1000);
httpClient = new DefaultHttpClient(connectionManager);
// HTTP parameters stores header etc.
params = new BasicHttpParams();
params.setParameter("http.protocol.handle-redirects",false);
}
/*
* Retrieve HTML
*/
public String fetchURL(String url) throws IOException{
if ( StringUtils.isEmpty(url) )
return null;
InputStream in = null;
//int code = -1;
String html = "";
// Prepare a request object
HttpGet httpget = new HttpGet(url);
httpget.setParams(params);
// Execute the request
HttpResponse response = httpClient.execute(httpget);
// The response status
//System.out.println(response.getStatusLine());
int code = response.getStatusLine().getStatusCode();
// Get hold of the response entity
HttpEntity entity = response.getEntity();
// If the response does not enclose an entity, there is no need
// to worry about connection release
if (entity != null) {
try {
//code = httpClient.executeMethod(getMethod);
//in = getMethod.getResponseBodyAsStream();
in = entity.getContent();
html = CharStreams.toString(new InputStreamReader(in));
}
catch (Exception except) {
throw new Exception("URL: " + url + " returned response code " + code);
}
finally {
try {
//oin.close();
in.close();
}
catch (Exception except) {}
//getMethod.releaseConnection();
connectionManager.closeIdleConnections(MAX_CONNECTION_IDLE_TIME, TimeUnit.MILLISECONDS);
connectionManager.closeExpiredConnections();
}
}
if (code <= 400){
return html;
} else {
throw new Exception("URL: " + url + " returned response code " + code);
}
}
I won't want redirects but under HttpClient 4.x if I enable redirects then I get some that are undesirable, e.g. http://www.walmart.com/ => http://mobile.walmart.com/. Under HttpClient 3.x no such redirects happens.
What do I need to do to migrate HttpClient 3.x to HttpClient 4.x without breaking the code?
It is not the issue with HttpClient 4.x, might be the way target server handle the request, since the user agent is httpclient, it may be handled as mobile (target server may consider other than available browsers like, i.e, chrome, mozilla etc as mobile.)
Please use below code to set it manually
httpclient.getParams().setParameter(
org.apache.http.params.HttpProtocolParams.USER_AGENT,
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2"
);
I'm getting an IOException from my code using AndroidHttpClient. Is there something wrong with the code? The code is thrown whether I use port 80 or port 443, and with Http Schema http or https.
The exception is thrown at client.execute...it is a UnknownHostException exception..I'm not sure why. I can hit the service from the browser.
// declare locals
JSONObject joResponse = new JSONObject();
String response = "";
try
{
// prepare to execute
HttpHost target = new HttpHost(hostname, Const.HTTP_PORT, Const.HTTP_SCHEME);
HttpGet get = new HttpGet();
ResponseHandler<String> responseHandler = new BasicResponseHandler();
AndroidHttpClient client = AndroidHttpClient.newInstance(null);
// execute
response = client.execute(target, get, responseHandler); // this is where the exception is thrown
joResponse = new JSONObject(response);
}
catch (ClientProtocolException e)
{
}
catch (IOException e)
{
}
catch(Exception e)
{
}
Do you have the internet permission declared in your app?
You need to add the following line to your AndroidManifest.xml
<uses-permission android:name="android.permission.INTERNET" />
I am trying to implement request retries after response isn't received.
I was reading about it in the Httpclient tutorial. the HttpRequestRetryHandler gets invoked only once and then it throw exception . what do I do wrong here?
updated
I added one if condition in the exception handling for SocketTimeoutException.
but what to do from there? how can I retry?
private void setRetry(int executionCount)
{
myRetryHandler = new HttpRequestRetryHandler() {
public boolean retryRequest(IOException exception,int executionCount,HttpContext context) {
if (executionCount >= 4) {
// Do not retry if over max retry count
System.out.println("retry count");
return false;
}
if (exception instanceof NoHttpResponseException) {
System.out.println("NoHttpResponseException exception");// Retry if the server dropped connection on us
return true;
}
if (exception instanceof SSLHandshakeException) {
// Do not retry on SSL handshake exception
System.out.println("SSLHandshakeException exception");
return false;
}
if (exception instanceof java.net.SocketTimeoutException) {
// Do not retry on SSL handshake exception
System.out.println("java.net.SocketTimeoutException exception");
return false;
}
HttpRequest request = (HttpRequest) context.getAttribute( ExecutionContext.HTTP_REQUEST);
boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);
if (idempotent) {
System.out.println("idempotent exception");
// Retry if the request is considered idempotent
return true;
}
return false;
}
};
}
public String postHttpReqest(int retries,
int socketTimeoutMillis,
int isSSL,
String target,
String url,
String base_url,
int port,
LinkedHashMap<String, String> lHashMapParams) throws Exception
{
HttpParams httpParams = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(httpParams, 50000);
HttpConnectionParams.setSoTimeout(httpParams, socketTimeoutMillis);
defaulthttpclient = new DefaultHttpClient(httpParams);
setRetry(retries); // here i set the handler
defaulthttpclient.setHttpRequestRetryHandler(myRetryHandler);
String line = "";
List<BasicNameValuePair> params = new ArrayList<BasicNameValuePair>();
for (String key : lHashMapParams.keySet()) {
String val = lHashMapParams.get(key);
params.add(new BasicNameValuePair(key,val));
}
UrlEncodedFormEntity query = new UrlEncodedFormEntity(params);
HttpPost httppost = new HttpPost(url+":"+Integer.toString(port)+"//"+base_url);
httppost.setEntity(query);
HttpResponse response_ = defaulthttpclient.execute(httppost);
HttpEntity entity = response_.getEntity();
if (entity != null) {
line = EntityUtils.toString(entity);
System.out.println(line);
}
return line;
}
In the server, I set break point so it will hold the connection and will not return response.
The error I am getting in the Httpclient :
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:149)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:110)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:260)
at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:298)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:633)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:454)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at com.fts.lb.connector.ut.HttpClientImpl.postHttpReqest(HttpClientImpl.java:183)
Um .. your RetryHandler is specifically returning false for a SocketTimeoutException.
So I would expect it to throw an Exception 100% of the time, which it is.
If you want the Request to be retried when the client times out before receiving a response, then you need to have your RetyrHandler return true for SocketTimeOutExceptions. You might also want to increase your SocketTimeOut before retrying.
Googlein a little bit I found this bug in the HttpClient 4.1 client (only on Windows).
For my problem I already use the latest 4.1.1, but maybe I have to implement the retryHandler.
I hope it will be helpful.
I'm trying to open a connection:
HttpPost hp = new HttpPost();
HttpParams rp = new BasicHttpParams();
HttpClientConnection hc = new DefaultHttpClientConnection();
HttpContext hctx = new BasicHttpContext();
hp.setURI(new URI(baseUrl + "/login.jsp"));
rp.setParameter("os_username", username);
rp.setParameter("os_password", password);
rp.setParameter("os_destination", "/secure");
hp.setParams(rp);
HttpResponse response = httpexecutor.execute(hp, hc, hctx);
But, I'm getting this stack trace:
java.lang.IllegalStateException: Connection is not open
at org.apache.http.impl.SocketHttpClientConnection.assertOpen(SocketHttpClientConnection.java:76)
at org.apache.http.impl.AbstractHttpClientConnection.sendRequestHeader(AbstractHttpClientConnection.java:239)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:213)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
what could be causing a "Connection is not open"? I don't see any specific methods that I can use to force open the connection.
As I remember this needs to be surrounded by:
try{
...code...
} catch (Exception e) {
out.println(e.getLocalizedMessage()); //or System.out.println
}
Hope this helps.
I have created the following function for checking the connection status:
private void checkConnectionStatus() {
HttpClient httpClient = new DefaultHttpClient();
try {
String url = "http://xxx.xxx.xxx.xxx:8000/GaitLink/"
+ strSessionString + "/ConnectionStatus";
Log.d("phobos", "performing get " + url);
HttpGet method = new HttpGet(new URI(url));
HttpResponse response = httpClient.execute(method);
if (response != null) {
String result = getResponse(response.getEntity());
...
When I shut down the server for testing the execution waits a long time at line
HttpResponse response = httpClient.execute(method);
Does anyone know how to set the timeout in order to avoid waiting too long?
Thanks!
In my example, two timeouts are set. The connection timeout throws java.net.SocketTimeoutException: Socket is not connected and the socket timeout java.net.SocketTimeoutException: The operation timed out.
HttpGet httpGet = new HttpGet(url);
HttpParams httpParameters = new BasicHttpParams();
// Set the timeout in milliseconds until a connection is established.
// The default value is zero, that means the timeout is not used.
int timeoutConnection = 3000;
HttpConnectionParams.setConnectionTimeout(httpParameters, timeoutConnection);
// Set the default socket timeout (SO_TIMEOUT)
// in milliseconds which is the timeout for waiting for data.
int timeoutSocket = 5000;
HttpConnectionParams.setSoTimeout(httpParameters, timeoutSocket);
DefaultHttpClient httpClient = new DefaultHttpClient(httpParameters);
HttpResponse response = httpClient.execute(httpGet);
If you want to set the Parameters of any existing HTTPClient (e.g. DefaultHttpClient or AndroidHttpClient) you can use the function setParams().
httpClient.setParams(httpParameters);
To set settings on the client:
AndroidHttpClient client = AndroidHttpClient.newInstance("Awesome User Agent V/1.0");
HttpConnectionParams.setConnectionTimeout(client.getParams(), 3000);
HttpConnectionParams.setSoTimeout(client.getParams(), 5000);
I've used this successfully on JellyBean, but should also work for older platforms ....
HTH
If your are using Jakarta's http client library then you can do something like:
HttpClient client = new HttpClient();
client.getParams().setParameter(HttpClientParams.CONNECTION_MANAGER_TIMEOUT, new Long(5000));
client.getParams().setParameter(HttpClientParams.SO_TIMEOUT, new Integer(5000));
GetMethod method = new GetMethod("http://www.yoururl.com");
method.getParams().setParameter(HttpMethodParams.SO_TIMEOUT, new Integer(5000));
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
int statuscode = client.executeMethod(method);
If you're using the default http client, here's how to do it using the default http params:
HttpClient client = new DefaultHttpClient();
HttpParams params = client.getParams();
HttpConnectionParams.setConnectionTimeout(params, 3000);
HttpConnectionParams.setSoTimeout(params, 3000);
Original credit goes to http://www.jayway.com/2009/03/17/configuring-timeout-with-apache-httpclient-40/
For those saying that the answer of #kuester2000 does not work, please be aware that HTTP requests, first try to find the host IP with a DNS request and then makes the actual HTTP request to the server, so you may also need to set a timeout for the DNS request.
If your code worked without the timeout for the DNS request it's because you are able to reach a DNS server or you are hitting the Android DNS cache. By the way you can clear this cache by restarting the device.
This code extends the original answer to include a manual DNS lookup with a custom timeout:
//Our objective
String sURL = "http://www.google.com/";
int DNSTimeout = 1000;
int HTTPTimeout = 2000;
//Get the IP of the Host
URL url= null;
try {
url = ResolveHostIP(sURL,DNSTimeout);
} catch (MalformedURLException e) {
Log.d("INFO",e.getMessage());
}
if(url==null){
//the DNS lookup timed out or failed.
}
//Build the request parameters
HttpParams params = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(params, HTTPTimeout);
HttpConnectionParams.setSoTimeout(params, HTTPTimeout);
DefaultHttpClient client = new DefaultHttpClient(params);
HttpResponse httpResponse;
String text;
try {
//Execute the request (here it blocks the execution until finished or a timeout)
httpResponse = client.execute(new HttpGet(url.toString()));
} catch (IOException e) {
//If you hit this probably the connection timed out
Log.d("INFO",e.getMessage());
}
//If you get here everything went OK so check response code, body or whatever
Used method:
//Run the DNS lookup manually to be able to time it out.
public static URL ResolveHostIP (String sURL, int timeout) throws MalformedURLException {
URL url= new URL(sURL);
//Resolve the host IP on a new thread
DNSResolver dnsRes = new DNSResolver(url.getHost());
Thread t = new Thread(dnsRes);
t.start();
//Join the thread for some time
try {
t.join(timeout);
} catch (InterruptedException e) {
Log.d("DEBUG", "DNS lookup interrupted");
return null;
}
//get the IP of the host
InetAddress inetAddr = dnsRes.get();
if(inetAddr==null) {
Log.d("DEBUG", "DNS timed out.");
return null;
}
//rebuild the URL with the IP and return it
Log.d("DEBUG", "DNS solved.");
return new URL(url.getProtocol(),inetAddr.getHostAddress(),url.getPort(),url.getFile());
}
This class is from this blog post. Go and check the remarks if you will use it.
public static class DNSResolver implements Runnable {
private String domain;
private InetAddress inetAddr;
public DNSResolver(String domain) {
this.domain = domain;
}
public void run() {
try {
InetAddress addr = InetAddress.getByName(domain);
set(addr);
} catch (UnknownHostException e) {
}
}
public synchronized void set(InetAddress inetAddr) {
this.inetAddr = inetAddr;
}
public synchronized InetAddress get() {
return inetAddr;
}
}
An option is to use the OkHttp client, from Square.
Add the library dependency
In the build.gradle, include this line:
compile 'com.squareup.okhttp:okhttp:x.x.x'
Where x.x.x is the desired library version.
Set the client
For example, if you want to set a timeout of 60 seconds, do this way:
final OkHttpClient okHttpClient = new OkHttpClient();
okHttpClient.setReadTimeout(60, TimeUnit.SECONDS);
okHttpClient.setConnectTimeout(60, TimeUnit.SECONDS);
ps: If your minSdkVersion is greater than 8, you can use TimeUnit.MINUTES. So, you can simply use:
okHttpClient.setReadTimeout(1, TimeUnit.MINUTES);
okHttpClient.setConnectTimeout(1, TimeUnit.MINUTES);
For more details about the units, see TimeUnit.
HttpParams httpParameters = new BasicHttpParams();
HttpProtocolParams.setVersion(httpParameters, HttpVersion.HTTP_1_1);
HttpProtocolParams.setContentCharset(httpParameters,
HTTP.DEFAULT_CONTENT_CHARSET);
HttpProtocolParams.setUseExpectContinue(httpParameters, true);
// Set the timeout in milliseconds until a connection is
// established.
// The default value is zero, that means the timeout is not used.
int timeoutConnection = 35 * 1000;
HttpConnectionParams.setConnectionTimeout(httpParameters,
timeoutConnection);
// Set the default socket timeout (SO_TIMEOUT)
// in milliseconds which is the timeout for waiting for data.
int timeoutSocket = 30 * 1000;
HttpConnectionParams.setSoTimeout(httpParameters, timeoutSocket);
you can creat HttpClient instance by the way with Httpclient-android-4.3.5,it can work well.
SSLContext sslContext = SSLContexts.createSystemDefault();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(
sslContext,
SSLConnectionSocketFactory.STRICT_HOSTNAME_VERIFIER);
RequestConfig.Builder requestConfigBuilder = RequestConfig.custom().setCircularRedirectsAllowed(false).setConnectionRequestTimeout(30*1000).setConnectTimeout(30 * 1000).setMaxRedirects(10).setSocketTimeout(60 * 1000);
CloseableHttpClient hc = HttpClients.custom().setSSLSocketFactory(sslsf).setDefaultRequestConfig(requestConfigBuilder.build()).build();
If you are using the HttpURLConnection, call setConnectTimeout() as described here:
URL url = new URL(myurl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setConnectTimeout(CONNECT_TIMEOUT);
public boolean isInternetWorking(){
try {
int timeOut = 5000;
Socket socket = new Socket();
SocketAddress socketAddress = new InetSocketAddress("8.8.8.8",53);
socket.connect(socketAddress,timeOut);
socket.close();
return true;
} catch (IOException e) {
//silent
}
return false;
}