Develop Reference

Java 11 gives Unsupported handshake message: server_hello_done on HTTPS connection with client certificate

We have a Java applet (running under tomcat) which makes calls to third parties. One of them uses a client certificate to authenticate. This was working under Java 8, but we recently upgraded the system to Java 11 and it doesn't any more. The error is
Unsupported handshake message: server_hello_done
(which is odd, since I thought server_hello_done was a valid part of the handshake)
We did have an issue with the java keystore after upgrade. The service failed saying it wasn't a valid PCKS12 stream. Listing the contents with keytool worked, but with warning
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /path/to/keystore -destkeystore /path/to/keystore -deststoretype pkcs12"
We used the suggested command and it now opens the keystore OK, but we get the handshake error.
The stack trace back to our code is:
Unsupported handshake message: server_hello_done
javax.net.ssl.SSLProtocolException: Unsupported handshake message: server_hello_done
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:126)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:446)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
We are using the apache http client (org.apache.http.impl.client.CloseableHttpClient). The next line in the stack trace is simply a call to client.execute() in our code.
It also includes
Caused by: java.lang.UnsupportedOperationException: Not supported yet.
at java.base/sun.security.ssl.HandshakeHash$CloneableHash.archived(HandshakeHash.java:616)
at java.base/sun.security.ssl.HandshakeHash$T12HandshakeHash.archived(HandshakeHash.java:546)
at java.base/sun.security.ssl.HandshakeHash.archived(HandshakeHash.java:188)
at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:581)
at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:740)
at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:173)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
The third party confirms they see "no shared cipher" in their logs, but say they support a broad range of ciphers ("SSL3, TLS 1.0, 1.1, or 1.2. Ciphers are marked as HIGH:MEDIUM:!aNULL:!eNULL:#STRENGTH"). I think we support most of that except SSL3. As far as I know we have default settings in java 11 for such. Tried activating SSLv3 temporarily but couldn't connect (although since trying that i can't connect from that machine at all (times out) even after reverting, so that may not say much - I tried from a test machine not the production one for that).
Any ideas? Am I on the right lines to keep looking at ciphers or is there something I'm missing?

It turned out that all that was needed was a restart of tomcat! But, since this was a production server doing many things I didn't want to do that as step one. But it fixed it.
My best guess as to what happened was:
The keystore was in the wrong format for the updated java version
The attempt to use the invalid keystore put tomcat/java into some weird state
Updating the keystore allowed java to use it, but it was still in some weird state
Restart fixed it.
What is really weird is that it behaved the same way even against a different IP (I tried against the third party's Test server) and with a different keystore (I made a copy of the updated keystore and did a request which used that). The initial error was "Stream is not a valid PKCS12 keystore" and the stack trace went back to the line that tried to open the keystore in our code. Having fixed the keystore format, it got further in our code (stack trace now went back to client.execute()) but it failed with _Unsupported handshake message: server_hello_done_.
I created a copy of the whole tomcat folder (including jre), with initially the original (invalid) keystore, changed it to run on another port and started it up next to this tomcat on the same machine. It behaved the same but then worked after a restart. This morning i restarted the main service and now it works

LDAPS : Simple bind failed

I'm facing issue connecting to LDAPS from my application. I have imported all necessary certificates on JRE keystore.
I'm able to make calls to LDAPs when I put the following string in java.security
jdk.tls.disabledAlgorithms=MD5, SSLv3, DSA, RSA keySize < 2048
When I change this line to
jdk.tls.disabledAlgorithms=MD5, DSA, DESede, DES_CBC, DHE, RC4, SSLv3, ECDH_anon, DH_anon, NULL, DH keySize < 768, RSA keySize < 2048
my connection fails with following error:
Caused by: javax.naming.CommunicationException: simple bind failed: testxxxxl.xxxx.com:636 [Root exception is java.net.SocketException: Socket closed]
at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:218)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2740)
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:316)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:193)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:211)
at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:154)
at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:84)
at org.jboss.as.naming.InitialContext.getDefaultInitCtx(InitialContext.java:122)
... 72 more
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:709)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:431)
at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:404)
at com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:358)
In wireshark it complains about certificate invalid.My question is if my certificate is invalid, it shouldn't be working with this line as well.
jdk.tls.disabledAlgorithms=MD5, SSLv3, DSA, RSA keySize < 2048
Please help me to solve the issue.

If you use a secure connection to the LDAP server and you see an error like the following when trying to connect to Active Directory:
simple bind failed: ad.hostname.com:636
Import the LDAP server public certificate directly into the Klocwork keystore (which should be_jvm\lib\security\cacerts).
This causes the certificate validation process at the Klocwork end to be bypassed, since you have decided to trust the LDAP server certificate by importing it into your list of trusted certificates.
Ask your LDAP administrator to set this extension of your LDAP server certificate to non-critical.

In recent versions of Java the TLSv1 has been disabled.
In my case this happened when upgrading from Java 8 to Java 11.
Re-enabling TLSv1 helped. It is as easy as defining property:
-Djdk.tls.client.protocols=TLSv1
Of course you can need also other protocols, so you can specify full list:
-Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2,TLSv1.3

SSL Handshake issue on cloudera 5.9

I am trying to use sales force wave api library (https://github.com/springml/salesforce-wave-api) in cloudera cluster 5.9 to get data, I have to use proxy because from our cluster its the only way to communicate outside world.
So I made changes to the library to take proxy host and port to communicate below are the place where I made changes.
Change 1:-
config.setProxy("myenterpiseproxy server",port);
https://github.com/springml/salesforce-wave-api/blob/0ac76aeb2221d9e7038229fd352a8694e8cde7e9/src/main/java/com/springml/salesforce/wave/util/SFConfig.java#L101
Change 2:-
HttpHost proxy = new HttpHost("myenterpiseproxy server", port, "http");
https://github.com/springml/salesforce-wave-api/blob/0ac76aeb2221d9e7038229fd352a8694e8cde7e9/src/main/java/com/springml/salesforce/wave/util/HTTPHelper.java#L127
Change 3:-
RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(timeout)
.setConnectTimeout(timeout).setConnectionRequestTimeout(timeout).setProxy(proxy).build();
https://github.com/springml/salesforce-wave-api/blob/0ac76aeb2221d9e7038229fd352a8694e8cde7e9/src/main/java/com/springml/salesforce/wave/util/HTTPHelper.java#L129
I built an application to use salesforce wave api as dependency and I tried to execute the Jar I am getting SSL handshake issue.
I passed in javax.net.ssl.truststore,javax.net.ssl.keyStore,https.protocols of my cluster still having problems.
Did anyone had similar issue ? did anyone tried to use this library in cloudera cluster ?
Run Book:-
java -cp httpclient-4.5.jar:SFWaveApiTest-1.0-SNAPSHOT-jar-with-dependencies.jar com.az.sfget.SFGetTest "username" "passwordwithtoken" "https://test.salesforce.com/services/Soap/u/35" "select id,OWNERID from someobject" "enterpiseproxyhost" "9400" "TLSv1.1,TLSv1.2" "/usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts" "/opt/cloudera/security/jks/uscvlpcldra-keystore.jks"
Error:-
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:946)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:388)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at com.springml.salesforce.wave.util.HTTPHelper.execute(HTTPHelper.java:122)
at com.springml.salesforce.wave.util.HTTPHelper.get(HTTPHelper.java:88)
at com.springml.salesforce.wave.util.HTTPHelper.get(HTTPHelper.java:92)
at com.springml.salesforce.wave.impl.ForceAPIImpl.query(ForceAPIImpl.java:120)
at com.springml.salesforce.wave.impl.ForceAPIImpl.query(ForceAPIImpl.java:36)
at com.az.sfget.SFGetTest.main(SFGetTest.java:54)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:482)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
... 21 more

Import expired SSL certificate into Java keystore

I need to be able to point JMeter at a test server which has an expired SSL certificate (it will be some time before we are able to renew it). JMeter is quite rightly throwing an exception when it tries to connect:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(Unknown Source)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:284)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1075)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1064)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:426)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:255)
at java.lang.Thread.run(Unknown Source)
Does anybody know of any way that I can import said certificate into the keystore and in the process change the expiry date (and would this even help or would the fact that the server certificate has expired still cause this exception to be thrown)?
I've tried to set the validity as part of the import but this is ignored:
keytool -import ... -validity 100
P.S - I know I could implement my own TrustManager which ignores these checks but my fingers won't allow me to write such evil code and I'd much rather get to a solution that I can install on the server without having to modify JMeter!
Thanks for your time.

JMeter does not validate certificate so this is not the cause of your issue.
Which implementation do you use, java, HC3 or HC4 ?
Your issue could come from error in Socket version negotation.
Try setting this in user.properties:
https.socket.protocols=SSLv2Hello SSLv3 TLSv1
You may have to play with them depending on your server configuration, for example only set this:
https.socket.protocols=SSLv3

Receiving SSLHandshakeException: handshake_failure despite my client ignoring all certs

I have a Java program that connects to a webserver using SSL/TLS, and sends various HTTP requests over that connection. The server is localhost and is using a self-signed cert, but my code is using custom TrustManagers, and ignores invalid certificates. It has worked perfectly until now.
The only difference on the server is that it used to run jboss 6 and is now running jboss 7. I'm not sure if this is a configuration issue, or whether there is a problem with my code, but I get the same errors if I try to connect using other Java-based programs like WebScarab or ZAP.
In any case, is there anything I can do to my code to get around this problem? Here is the error in full:
Received fatal alert: handshake_failure
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)
Here are the debug messages before the failure:
main, WRITE: TLSv1 Handshake, length = 75
main, WRITE: SSLv2 client hello message, length = 101
main, READ: TLSv1 Alert, length = 2
main, RECV TLSv1 ALERT: fatal, handshake_failure

So I found the problem. There might be a bug in Java, but the client seems to initiate a TLSv1 Handshake, but then sends an SSLv2 client hello message, at which point the server rejects the connection.
This happens even if you create your SSLContext with an instance of TLS:
SSLContext sslContext = SSLContext.getInstance("TLS");
The solution is to set a system property before any connection attempts are made:
System.setProperty("https.protocols", "TLSv1");
There are probably other solutions to it, but this one worked for me.

The info you provide is very little as well as your stack trace.
I'll take a guess here.
What I suspect is that in the new server the protocol is TLSv1 while your clients try to connect with SSLv3 (or less) and as a result the handshake fails.
Change you clients to use higher version of TLS
or
Make your webserver support SSLv3 as well. I know how to do this in Tomcat but not in JBoss.
If this doesn't work update the post with more info (and a full stack trace).
You should enable ssl debug info -Djavax.net.debug=ssl

Was this ever resolved?
I had the exact same problem, essentially I was receiving a handshake exception immediately following the clientHello. So The chain of events was
I would present my certificate to the server
Server would imediately respond with a handshake failure. (I would not even get a Server Hello back).
Eventually I found that the server was requiring a stronger encryption/decryption algorithm than what I Was supplying in the initial handshake phase (Ie. Client and Server could not agree on a mutual encryption algorithm to use for the ssl communication).
I need to install the Unlimited Java JCE (Java Cryptography Extension Policy). There are export rules on using this, so if you ship your code overseas that may have implications..however this is what solved my problem.
This link explains how to install the updated policies
http://suhothayan.blogspot.com/2012/05/how-to-install-java-cryptography.html
This was also a great link that helped me understand exactly what was going on
https://support.f5.com/kb/en-us/solutions/public/15000/200/sol15292.html#id
This may or may not be the issue, but when the handshake fails immediately after the client Hello, it looks like the client and the server can not agree on something (in many cases its the encryption algorithms that they will mutually need to communicate).

You are seeing this error most probably because the keystore that your JBoss 6 had access to is not accessible to your JBoss 7 instance.
What I would recommend is the following.
Your self-signed server certificate must be imported into a truststore
keytool -import -alias gridserver -file server.crt -storepass $YOUR_PASSWORD_HERE -keystore server.keystore
Add the following properties to your run.conf
-Djavax.net.ssl.keyStoreType=pkcs12
-Djavax.net.ssl.trustStoreType=jks
-Djavax.net.ssl.keyStore=clientcertificate.p12
-Djavax.net.ssl.trustStore=server.keystore
-Djavax.net.debug=ssl # very verbose debug. Turn this off after everything looks good.
-Djavax.net.ssl.keyStorePassword=$YOUR_PASSWORD_HERE
-Djavax.net.ssl.trustStorePassword=$YOUR_PASSWORD_HERE

The stack trace is from you client code and your client 'Received [a] fatal alert'. In other words, the SSL error happened in Jboss, not your client.
Your client side custom TrustManagers have therefore nothing to do with it. My wild guess is that your new Jboss 7 is configured to require client certificate and your client did not present any.
To debug your SSL connection, use openssl and try this:
openssl s_client -connect jboss.server.com:443
or is it is an SSLV3 server
openssl s_client -connect jboss.server.com:443 -ssl3
This should print a lot of interesting information.

I think this is related to a Java 7 bug. It is hard to be sure without more details.

For me solution was : System.setProperty("https.protocols", "TLSv1.1,TLSv1.2");