Develop Reference

HttpClient : hostname didn't match - accessible from browser but not from code

I am trying to access a website from my code using HttpClient :
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet("https://www.datamed.org/search.php?query=gene&searchtype=data");
ResponseHandler<String> responseHandler = new BasicResponseHandler();
String responseBody = httpclient.execute(httpget, responseHandler);
This is the error i am getting :
Exception in thread "main" javax.net.ssl.SSLException: hostname in certificate didn't match: <www.datamed.org> != <ucrexdc.ucsd.edu> OR <ucrexdc.ucsd.edu>
I checked the certificate from browser, it seems correct, with correct names.
Not sure from where it is picking up ucrexdc.ucsd.edu .
The code does work if I use a proxy.
Gone through a lot of similar issues on StackOverflow, but in most cases the server was under user's control. In my case, this is an already existing website. and i have this problem only for this website.
Can it be a problem with my environment?
UPDATE:
I found out that both the websites (datamed.org and ucrexdc.ucsd.edu) have the same IP , 169.228.51.21 . Can it be a problem, why doesn't the browser have issues with this?
UPDATE 2:
I was using apache http-client 4.3.1,
When i updated to 4.4.1, it was resolved. the issue was most possibly related to SNI.

HttpClient provides two implementations for Hostname verification.
DefaultHostnameVerifier
NoopHostnameVerifier
by default HttpClient uses DefaultHostnameVerifier implementation. You can try the different hostname verifier implementation.
SSLContext sslContext = SSLContexts.createSystemDefault();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext, NoopHostnameVerifier.INSTANCE);
HttpClient httpClient = HttpClientBuilder.create().setSSLSocketFactory(sslsf).build();

How to handle Cookies with Apache HttpClient 4.3

I need to implement a series of HTTP requests in Java and decided to use Apaches HttpClient in version 4.3 (the most current one).
The problem is all these requests use a cookie for session management and I seem to be unable to find a way of accessing that cookie and passing it from request to request. My commands in using curl look something like:
# Login
curl -c cookies -d "UserName=username&Password=password" "https://example.com/Login"
# Upload a file
curl -b cookies -F fileUpload=#IMG_0013.JPG "https://example.com/File"
# Get results of server processing file
curl -b cookies "https://example.com/File/1234/Content"
They work perfectly. However with HttpClient it seems not to work. What I tried was:
URI serverAddress = new URI("https://example.com/");
URI loginUri = UriBuilder.fromUri(serverAddress).segment("Login").queryParam("UserName", "username")
.queryParam("Password", "password").build();
RequestConfig globalConfig = RequestConfig.custom().setCookieSpec(CookieSpecs.BEST_MATCH).build();
CookieStore cookieStore = new BasicCookieStore();
HttpClientContext context = HttpClientContext.create();
context.setCookieStore(cookieStore);
CloseableHttpClient httpClient = HttpClients.custom().setDefaultRequestConfig(globalConfig).setDefaultCookieStore(cookieStore).build();
HttpGet httpGet = new HttpGet(loginUri);
CloseableHttpResponse loginResponse = httpClient.execute(httpGet,context);
System.out.println(context.getCookieStore().getCookies());
The output of the last line is always an empty list. I think it should contain my Cookie, am I right?
Can someone give me a small example on how to handle the cookie using Apache HttpClient 4.3?
Thanks

Your code looks OK to me (other than not releasing resources, but I presume exception handling was omitted for brevity). The reason for cookie store being empty may be violation of the actual cookie policy (which is BEST_MATCH in your case) by the target server. So, cookies sent by the server get rejected as invalid. You can find out if that is the case (and other useful contextual details) by turning on context / wire logging as described here

Can't authenticate with DefaultHttpClient

I want to add authentication header to my request. I'm using DefaultHttpClient from Apache httpclient 4.0.
I found that's done this way:
URI uri = new URI("http://www.bla.bla/folder/");
String host = uri.getHost();
int port = uri.getPort();
httpClient.getCredentialsProvider().setCredentials(
new AuthScope(host, port, AuthScope.ANY_SCHEME),
new UsernamePasswordCredentials("myuser", "mypassword")
);
This is executed and even with the debugger I see some credentials variable of the httpClient are set at the moment of doing the request. But I inspect web traffic with Charles and there's no authentication header.
Content of vars:
host: www.bla.bla
port: -1
Btw. I enabled Charles as a proxy to see the headers of the request, with:
HttpHost proxy = new HttpHost("127.0.0.1", 8888, "http");
httpParameters.setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
I think that should not be altering my headers, would make no sense for webproxy... anyways if I disable the proxy stuff it also doesn't work (although I can't see the content of the header but I suppose it's the same reason).
Also tried using a request interceptor like described in Softhinker.com's post here: How can I send HTTP Basic Authentication headers in Android?
And I get exactly the same request, without authentification header.
What am I doing wrong?
Thanks in advance.

I got it working setting the header "manually" in the request.
request.setHeader(new BasicHeader("Authorization", authstring));

HTTPClient sends out two requests when using Basic Auth?

I have been using HTTPClient version 4.1.2 to try to access a REST over HTTP API that requires Basic Authentication. Here is client code:
DefaultHttpClient httpClient = new DefaultHttpClient(new ThreadSafeClientConnManager());
// Enable HTTP Basic Auth
httpClient.getCredentialsProvider().setCredentials(
new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT),
new UsernamePasswordCredentials(this.username, this.password));
HttpHost proxy = new HttpHost(this.proxyURI.getHost(), this.proxyURI.getPort());
httpClient.getParams().setParameter(ConnRouteParams.DEFAULT_PROXY, proxy);
When I construct a POST request, like this:
HttpPost request = new HttpPost("http://my/url");
request.addHeader(new BasicHeader("Content-type", "application/atom+xml; type=entry")); // required by vendor
request.setEntity(new StringEntity("My content"));
HttpResponse response = client.execute(request);
I see in Charles Proxy that there are two requests being sent. One without the Authorization: Basic ... header and one with it. The first one fails with a 401, as you would expect, but the second goes through just fine with a 201.
Does anyone know why this happens? Thanks!
EDIT:
I should make clear that I have already looked at this question, but as you can see I set the AuthScope the same way and it didn't solve my problem. Also, I am creating a new HttpClient every time I made a request (though I use the same ConnectionManager), but even if I use the same HttpClient for multiple requests, the problem still persists.
EDIT 2:
So it looks like what #LastCoder was suggesting is the way to do. See this answer to another question. The problem stems from my lack of knowledge around the HTTP spec. What I'm looking to do is called "preemptive authentication" and the HttpClient docs mention it here. Thankfully, the answer linked to above is a much shorter and cleaner way to do it.

Rather than using .setCredentials() why don't you just encode USERNAME:PASSWORD and add the authentication header with .addHeader()

This means that your server/target endpoint is creating a new session for every client request. This forces every request of yours to go through a hand-shake, which means the clients first makes the call and realizes that it needs authorization, then it follows with the authorization. What you need to do is send the authorization preemptively as follows:
httpClient.getParams().setAuthenticationPreemptive(true);
Just to understand the process you may log your client request headers, to give you an idea of what your client is sending and receiving:
See if this works.

Does Apache HttpClient add the Cookies set by the java.net.CookieHandler to Request?

My simple Apache HttpClient (4.0.1) client application makes an HttpGet request to a server URL in the main() method and prints the response. On startup, the application registers an implementation of java.net.CookieHandler in a static block.
On checking the cookies received on the server side, I found that the cookies are not being received by the Server when the HttpClient makes the GET request.
On the other hand, when I replaced the Apache HttpClient with a plain java.net.URL(HTTP_URL).openStream(), the cookies were set by the CookieHandler on the Request and were received by the Server.
Is it that CookieHandler does not work with Apache HttpClient?
Code:
Client.java
static {
CookieHandler.setDefault(new CookieHandler() {
public Map get(URI u, List r) {
return Collections.singletonMap("Cookie",
Collections.singletonList(COOKIE_STRING));
}
});
}
Using HttpClient (does not put cookies on request)
HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet(HTTP_URL);
client.execute(get);
Using java.net.URL (sets the cookies on request)
URL url = new URL(HTTP_URL);
InputStream is = url.openStream();

Is it that CookieHandler does not work with Apache HttpClient?
That is correct.
The Apache HttpClient codebase uses its own cookie and cookie store representations / mechanisms. Here is a link to the relevant section of the HttpClient tutorial. (It is pretty sketchy, but if you look at the javadocs for the relevant classes, you should be able to figure out how to use it.)
(If you are using an older version of Apache HttpClient, beware that the APIs have changed significantly.)