Develop Reference

Apache Async HTTP send requests with new Proxy on each Request

I have to check thousands of proxy servers continuously.
To speed it up, I am thinking to create a batch of size N(say 50) and send requests to them concurrently. Each proxy server has a unique IP/Port and username/password authentication.
Since I am checking proxies, I will configure the request to use the given Proxy and send a request to the target site and measure the response.
Here is an example to use proxy with auth from the Apache client docs:
public static void main(String[] args)throws Exception {
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
new AuthScope("localhost", 8889),
new UsernamePasswordCredentials("squid", "nopassword"));
CloseableHttpAsyncClient httpclient = HttpAsyncClients.custom()
.setDefaultCredentialsProvider(credsProvider)
.build();
try {
httpclient.start();
HttpHost proxy = new HttpHost("localhost", 8889);
RequestConfig config = RequestConfig.custom()
.setProxy(proxy)
.build();
HttpGet httpget = new HttpGet("https://httpbin.org/");
httpget.setConfig(config);
Future<HttpResponse> future = httpclient.execute(httpget, null);
HttpResponse response = future.get();
System.out.println("Response: " + response.getStatusLine());
System.out.println("Shutting down");
} finally {
httpclient.close();
}
}
As you can see, if you are using an authenticated proxy, you need to provide the credentials in the Client itself.
This means that if I am checking 50 proxy servers concurrently then I have to create a new client for each of them. Which means that the requests will not be concurrent and better if I just use a multi-threaded solution.
The issue is that if I use multithreading then I will put excessive loads on the server as most of the threads will block on I/O. A concurrent non-blocking I/O is much better for this type of challenge.
How can I check multiple authenticated proxy servers concurrently if I have to create a client for each of them?

How to get persistent HttpConnection with Apache HttpClient?

In my test application I execute consecutive HttpGet requests to the same host with Apache HttpClient but upon each next request it turns out that the previous HttpConnection is closed and the new HttpConnection is created.
I use the same instance of HttpClient and don't close responses. From each entity I get InputStream, read from it with Scanner and then close the Scanner. I have tested KeepAliveStrategy, it returns true. The time between requests doesn't exceed keepAlive or connectionTimeToLive durations.
Can anyone tell me what could be the reason for such behavior?
Updated
I have found the solution. In order to keep the HttpConnecton alive it is necessary to set HttpClientConnectionManager when building HttpClient. I have used BasicHttpClientConnectionManager.
ConnectionKeepAliveStrategy keepAliveStrat = new DefaultConnectionKeepAliveStrategy() {
#Override
public long getKeepAliveDuration(HttpResponse response, HttpContext context)
{
long keepAlive = super.getKeepAliveDuration(response, context);
if (keepAlive == -1)
keepAlive = 120000;
return keepAlive;
}
};
HttpClientConnectionManager connectionManager = new BasicHttpClientConnectionManager();
try (CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager) // without this setting connection is not kept alive
.setDefaultCookieStore(store)
.setKeepAliveStrategy(keepAliveStrat)
.setConnectionTimeToLive(120, TimeUnit.SECONDS)
.setUserAgent(USER_AGENT)
.build())
{
HttpClientContext context = new HttpClientContext();
RequestConfig config = RequestConfig.custom()
.setCookieSpec(CookieSpecs.DEFAULT)
.setSocketTimeout(10000)
.setConnectTimeout(10000)
.build();
context.setRequestConfig(config);
HttpGet httpGet = new HttpGet(uri);
CloseableHttpResponse response = httpClient.execute(httpGet, context);
HttpConnection conn = context.getConnection();
HttpEntity entity = response.getEntity();
try (Scanner in = new Scanner(entity.getContent(), ENC))
{
// do something
}
System.out.println("open=" + conn.isOpen()); // now open=true
HttpGet httpGet2 = new HttpGet(uri2); // on the same host with other path
// and so on
}
Updated 2
In general checking connections with conn.isOpen() is not proper way to check the connections state because: "Internally HTTP connection managers work with instances of ManagedHttpClientConnection acting as a proxy for a real connection that manages connection state and controls execution of I/O operations. If a managed connection is released or get explicitly closed by its consumer the underlying connection gets detached from its proxy and is returned back to the manager. Even though the service consumer still holds a reference to the proxy instance, it is no longer able to execute any I/O operations or change the state of the real connection either intentionally or unintentionally." (HttpClent Tutorial)
As have pointed #oleg the proper way to trace connections is using the logger.

First of all you need to make sure remote server you're working with does support keep-alive connections. Just simply check whether remote server does return header Connection: Keep-Alive or Connection: Closed in each and every response. For Close case there is nothing you can do with that. You can use this online tool to perform such check.
Next, you need to implement the ConnectionKeepAliveStrategy as defined in paragraph #2.6 of this manual. Note that you can use existent DefaultConnectionKeepAliveStrategy since HttpClient version 4.0, so that your HttpClient will be constructed as following:
HttpClient client = HttpClients.custom()
.setKeepAliveStrategy(DefaultConnectionKeepAliveStrategy.INSTANCE)
.build();
That will ensure you HttpClient instance will reuse the same connection via keep-alive mechanism if it is being supported by server.

Your application must be closing response objects in order to ensure proper resource de-allocation of the underlying connections. Upon response closure HttpClient keeps valid connections alive and returns them back to the connection manager (connection pool).
I suspect your code simply leaks connections and every request ens up with a newly created connection while all previous connections keep on piling up in memory.

From the example at HttpClient website:
// In order to ensure correct deallocation of system resources
// the user MUST call CloseableHttpResponse#close() from a finally clause.
// Please note that if response content is not fully consumed the underlying
// connection cannot be safely re-used and will be shut down and discarded
// by the connection manager.
So as #oleg said you need to close the HttpResponse before checking the connection status.

Validating client credentials on a server using Java SimpleFramework

I am developing a SSL/TLS enabled server using the Java SimpleFramework. I am wondering how to validate client authentications on the server.
On the server side, I am extending org.simpleframework.http.core.ContainerServer and overriding the process() method as follows:
#Override
public void process(Socket socket) throws IOException {
// Ensures client authentication is required
socket.getEngine().setNeedClientAuth(true);
super.process(socket);
}
This is to make sure that clients authenticate. Note that if I remove the call to setNeedClientAuth(), my program works perfectly.
On the client side, the following code is used:
HttpClient client = new HttpClient();
Credentials defaultcreds = new UsernamePasswordCredentials("username", "password");
client.getState().setCredentials(AuthScope.ANY, defaultcreds);
GetMethod get = new GetMethod("https://url.to.server");
get.setDoAuthentication(true);
client.executeMethod(get);
When enabling authentication requirement, I get the following exception:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
I am guessing this relates to the fact that the passed credentials is never validated.
To summarize my question, how should I proceed to validate clients on the server?

Disable or delay timeout in Apache Httpclient request

I have a REST webservice with some methods.
I'm sending requests to the rest with Apache HttpClient 4.
When I make a connection to this rest, in a method that is bigger and slower, it throws a NoHttpResponseException.
After googling, I discovered that the server is cutting down the connection with my client app.
So, I tried to disable the timeout this way :
DefaultHttpClient httpclient = null;
HttpParams params = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(params, 0);
HttpConnectionParams.setSoTimeout(params, 0);
HttpConnectionParams.setStaleCheckingEnabled(params, true);
httpclient = new DefaultHttpClient(params);
httpclient.execute(httpRequest, httpContext);
But it failed. The request dies in 15 seconds (possible default timeout?)
Does anyone know the best way to do this?

I would suggest that you return data to the client before the timeout can occur. This may just be some bytes that says "working" to the client. By trickling the data out, you should be able to keep the client alive.

Can't authenticate with DefaultHttpClient

I want to add authentication header to my request. I'm using DefaultHttpClient from Apache httpclient 4.0.
I found that's done this way:
URI uri = new URI("http://www.bla.bla/folder/");
String host = uri.getHost();
int port = uri.getPort();
httpClient.getCredentialsProvider().setCredentials(
new AuthScope(host, port, AuthScope.ANY_SCHEME),
new UsernamePasswordCredentials("myuser", "mypassword")
);
This is executed and even with the debugger I see some credentials variable of the httpClient are set at the moment of doing the request. But I inspect web traffic with Charles and there's no authentication header.
Content of vars:
host: www.bla.bla
port: -1
Btw. I enabled Charles as a proxy to see the headers of the request, with:
HttpHost proxy = new HttpHost("127.0.0.1", 8888, "http");
httpParameters.setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
I think that should not be altering my headers, would make no sense for webproxy... anyways if I disable the proxy stuff it also doesn't work (although I can't see the content of the header but I suppose it's the same reason).
Also tried using a request interceptor like described in Softhinker.com's post here: How can I send HTTP Basic Authentication headers in Android?
And I get exactly the same request, without authentification header.
What am I doing wrong?
Thanks in advance.

I got it working setting the header "manually" in the request.
request.setHeader(new BasicHeader("Authorization", authstring));