Develop Reference

Trying to understand how mutual authentication key pairs and certificates work in a gRPC context

so after reading countless articles on how and what to generate key pairs, certificates, trust managers i'm incredibly confused.
This is my situation, i have a client:
SslContextBuilder builder = GrpcSslContexts.forClient();
// builder.trustManager(new File(trustCertCollectionFilePath)); //i've read this should be ignored for the client
builder.keyManager(new File(clientCertChainFilePath), new File(clientPrivateKeyFilePath));
and a server:
SslContextBuilder sslClientContextBuilder = SslContextBuilder.forServer(new
File(certChainFilePath), new File(privateKeyFilePath));
sslClientContextBuilder.trustManager(new File(trustCertCollectionFilePath));
sslClientContextBuilder.clientAuth(ClientAuth.REQUIRE);
I'm using these from an example found here:
https://github.com/grpc/grpc-java/tree/master/examples/example-tls/src/main/java/io/grpc/examples/helloworldtls
As far as I've understood it should work like this:
For the client:
1. You have to generate a RSA key pair.
2. Generate a certificate.
3. Put the public key inside the certificate.
clientCertChainFilePath = certificate with public key inside
clientPrivateKeyFilePath = client private key
For the server:
1. You have to generate a trusted authority certificate(CA) with a server private key
2. Get the certificate from the client
3. Register the client certificate inside the trusted authority somehow.
certChainFilePath = certificate from the client with public key inside
privateKeyFilePath = private server key for the trust authority certificate(CA)
trustCertCollectionFilePath = trusted authority certificate(CA)
Please correct me or tell me how exactly all of this binds together to make this work, if you have any specific links on how to generate everything properly it's highly appreciated.

Client and server both have their own public and private keys. These are two separate pairs of public-private keys.
On the server side, certChainFilePath is the server certificate with server's public key inside. privateKeyFilePath is server's private key.
Without mutual TLS, client only needs the CA certificate to verify server's certificate received during handshake.
With mutual TLS, client is requested to send its certificate (with client public key inside) to server.

Java JSSE TLS - Is this connection safely encrypted in both directions?

In Java using JSSE with TLS. I have created a secure socket between the server and client. After finally getting the sockets to connect securely, I still have a fundamental question about my existing code's security. I followed instructions in a tutorial, and sometimes the documentation in the JavaDoc is very precise but a little vague unless you speak the Swaheli dialect of Jargon....
I have been network programming for quite awhile now in C++. The transition to Java was easy. Recently, however, I have found it prudent to make the traffic secure. This being said:
I want to create a secure socket in the same way a web browser creates a secure socket, so traffic in both direction is encrypted. A client can see their personal account information sent from the server (very bad if intercepted), and a client can send their username and password to the server securely (also very bad if intercepted).
I know all about how public key cryptography works, but there's a side effect to public key cryptography alone. You send your public key to a client, the client encrypts with the public key, and sends data to the server only the server can decrypt. Now from what I understand, the server uses the private key to encrypt messages going to the client, and another layer of security needs to be added to prevent anyone with the public key from being able to decrypt it.
I have a public / private key pair stored in files public.key and private.key (i made these using JSSE's keytool utility
I included public.key in the client
I included private.key in the server
Client Class:
KeyStore keyStore;
TrustManagerFactory tmf;
KeyManagerFactory kmf;
SSLContext sslContext;
SecureRandom secureRandom = new SecureRandom();
secureRandom.nextInt();
keyStore = KeyStore.getInstance("JKS");
keyStore.load(this.getClass().getClassLoader().getResourceAsStream("server.public"),"public".toCharArray());
tmf = TrustManagerFactory.getInstance("SunX509");
tmf.init(keyStore);
kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(keyStore, "public".toCharArray());
sslContext = SSLContext.getInstance("TLS");
sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), secureRandom);
SSLSocketFactory sslsocketfactory = sslContext.getSocketFactory();
SSLSocket sslsocket = (SSLSocket)sslsocketfactory.createSocket("localhost", 9999);
Server Class:
String passphrase = "secret"
KeyStore keyStore;
TrustManagerFactory tmf;
KeyManagerFactory kmf;
SSLContext sslContext;
SecureRandom secureRandom = new SecureRandom();
secureRandom.nextInt();
keyStore = KeyStore.getInstance("JKS");
keyStore.load(this.getClass().getClassLoader().getResourceAsStream("server.private"),passphrase.toCharArray());
tmf = TrustManagerFactory.getInstance("SunX509");
tmf.init(keyStore);
kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(keyStore, passphrase.toCharArray());
sslContext = SSLContext.getInstance("TLS");
sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), secureRandom);
SSLServerSocketFactory sslserversocketfactory = sslContext.getServerSocketFactory();
SSLServerSocket sslserversocket =
(SSLServerSocket)sslserversocketfactory.createServerSocket(9999);
/ ******* THE QUESTION ********/
Everything works! I attach the sockets to BufferedReader and BufferedWriter and begin talking beautifully back and forth after accept(); ing the connection from the client and starting my client and server send / receive loops.
Now I know that at this point client to server communication is secure. Only the server key can decrypt traffic coming from the client. But what about server to client communication? The client's key can decrypt messages coming from the server, but in Public Key Crypto 101 you learn that the client is now supposed to send a public key to the server. Is this happening behind the scenes in this code? Did SSLContext take care of this? Or now that I have an encrypted connection from the client to the server, am I now expected to generate a private/public key pair for the client as well?
Let me know if the traffic being sent and received in the above code is actually secure in both directions.

The certificates (and their private keys) in SSL/TLS are only used for authenticating the parties in SSL/TLS (often, only the server uses a certificate).
The actual encryption is done using shared/symmetric keys that are negotiated during the handshake, derived from the pre master key exchanged using a form of authenticated key exchange (see TLS Specification, Section F.1.1.
How this authenticated key exchange is done depends on the cipher suite, but the end result is the same: a shared pre-master secret between the two parties, guaranteed to be known only to the client and the server with the private key for its certificate.
Following that pre-master secret exchange, the master secret itself is calculated, from which a pair of secret keys is derived (as described in the Key Calculation section): one for the client to write (and for the server to read) and one for the server to write (and for the client to read). (MAC secrets are also generated, to guarantee the connection integrity.)
In principle, not all cipher suites provide encryption and authenticated key exchange (see Cipher Suite Definitions section), but all those enabled by default in JSSE with the SunJSSE provider do (see Cipher Suite tables in the SunJSSE provider documentation). In short, don't enable cipher suites with anon or NULL in their names.
Regarding your code:
There are multiple example of code around that fix the Key/TrustManagerFactory algorithm like this ("SunX509"). This is typically code that hard-codes the Java 1.4 defaults. Since Java 5, the default TMF algorithm is PKIX (see Customization section of the JSSE Reference Guide). The best way to get around this is to use TrustManagerFactory.getDefaultAlgorithm() (same for KMF), which will also allow your code to run on other JREs that don't support SunX509 (e.g. IBM's).
Since you're not using client-certificate authentication, there's no point having a KeyManagerFactory on the client side. Your initialising it with a keystore that probably doesn't have a private key anyway, which makes it pointless. You might as well use sslContext.init(null, tmf.getTrustManagers(), null). (Same thing for the secure random in both cases, let the JSSE use its default values.)

You do have an understanding of how PKI works, but you are missing two key pieces of SSL implementation. First most PKI algorithms allow for encrypting traffic both ways. You can send encrypt message using public key and only whoever has a private key can read it, this is called encryption. You can also encrypt the message using a private key and anybody who has a public key can decrypt it, this is called digital signature.
Another missing piece is that SSL doesn't use PKI to send network traffic between client and server. It uses symmetric encryption algorithm. However the key for the symmetric encryption (called session key) is establish using rather complicated challenge-response protocol that employs PKI, and certificates. During this phase server proves to the client that it is not man in in the middle, client can optionally proved it's certificate to the server if it has any for stronger authentication, and the symmetric session key is established. More details are here https://www.rfc-editor.org/rfc/rfc5246
The symmetric key is used for encrypting traffic using algorithms like RC5 or AES

How can I effect 'known_hosts' vs self-certified servers for my app's SSL usage?

Oh wise and noble Oracle,
I'm adding SSL to a TCP client I've written on my Android phone. I can
successfully connect to servers with properly signed certificates, and I can
connect to self-certifying hosts by cooking up a TrustManager implementation
that always thinks everything is fine.
I now have a decorator TrustManager capturing the certificates (before
delegating to its decoratee) for self-certifying hosts and presenting them for
my breathless perusal, but what I can't work out is how to implement ssh's
behaviour of warning that a host is unknown and offering to remember it for
next time - and doing so.
I presumed all I needed was to store the public key - as ssh does with
known_hosts - and re-represent it, but with this code and 'sslTrust' holding
the public key:
TrustManagerFactory tmf = TrustManagerFactory.getInstance(
TrustManagerFactory.getDefaultAlgorithm());
KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
ks.load(null, null); // initialise!
ks.setKeyEntry("dbentry", Base64.decode(sslTrust, Base64.NO_WRAP), null);
tmf.init(ks);
tms = tmf.getTrustManagers();
ss.stm = new SnoopyTrustManager((X509TrustManager) tms[0]);
// ...
SLContext context = SSLContext.getInstance("SSL");
context.init(null, new TrustManager[] { ss.stm } , null);
ss.factory = context.getSocketFactory();
// ...
SocketFactory factory = ss.getFactory();
mSocket = factory.createSocket(host, port);
attempting to establish a connection results in
SSLHandshakeException: InvalidAlgorithmParameterException: trustAnchors.isEmpty()
which is fair enough: I don't know how to cook things up from the certificate
offered by the remote server. I'm also fairly sure this isn't how I tell a
TrustManager about a remote server's public key anyway.
Since the site is self-certifying, I imagine could probably just verify that
the public keys match in a trivial TrustManager, but I'd like to understand
how this 'should' be done - adding a CA on a per-connection basis, since
I won't trust that CA for anything else.

You need to use your own trust store on pre-ICS version, and add the serer's certificates to it on first error. Subsequent connections will load it from the trust store and thus trust the remote certificate. This is not a complete solution, but here's one way to do it (code on Github), along with some discussion:
http://nelenkov.blogspot.jp/2011/12/using-custom-certificate-trust-store-on.html

Why do I get the error "Cannot store non-PrivateKeys" when creating an SSL Socket in Java?

I am working on an older IBM iSeries (IBM-i, i5OS, AS/400, etc), with a Java 5 JVM (Classic, not ITJ J9) on O/S version V5R3M0.
Here is the scenario in a nutshell:
I created a key-store of type JKS using Portecle 1.7 (Note: I did try converting my key-store to JCEKS but that was rejected as an unsupported format, so it appears that JKS is the only option with the iSeries machine (at least the version I am on).
I then created a key-pair and CSR and sent the CSR to Thawte to be signed.
I imported the signed certificate from Thawte successfully using the PKCS#7 format to import the entire certificate chain, which included my certificate, the Thawte intermediary and the Thawte server root.
This all worked as expected.
However, when I ran up the JVM, configured properly to point to the store and supply it's password (which I have done in the past with self-signed certificates created in Portecle for testing), and try to start my web server on 443, I get the following security exception:
java.security.KeyStoreException: Cannot store non-PrivateKeys
Can anyone tell me where I went wrong, or what I should check next?

The "Cannot store non-PrivateKeys" error message usually indicates you are trying to use secret symmetric keys with a JKS keystore type. The JKS keystore type only supports asymmetric (public/private) keys. You would have to create a new keystore of type JCEKS to support secret keys.

As it turns out, this was a subtle problem, and it's worth giving the answer here in case someone else has something similar.
The TLDR answer is that I did not check that my key and certificate were not null and as a result attempted to add a null key and certificate to a key-store. The longer answer follows.
The way we have our web server set up to use SSL, specifically to support our user's typical configuration where the IP address is used to configure the web site listen address rather than a DNS name, is that it locates the certificate in the master key-store using the alias, and creates an ephemeral key-store containing just the certificate for that web site, using that key-store to configure an SSL context and an SSL socket factory, like so:
// CREATE EPHEMERAL KEYSTORE FOR THIS SOCKET USING THE DESIRED CERTIFICATE
try {
final char[] BLANK_PWD=new char[0];
SSLContext ctx=SSLContext.getInstance("TLS");
KeyManagerFactory kmf=KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
Key ctfkey=mstkst.getKey(svrctfals,BLANK_PWD);
Certificate[] ctfchn=mstkst.getCertificateChain(svrctfals);
KeyStore sktkst;
sktkst=KeyStore.getInstance("jks");
sktkst.load(null,BLANK_PWD);
sktkst.setKeyEntry(svrctfals,ctfkey,BLANK_PWD,ctfchn);
kmf.init(sktkst,BLANK_PWD);
ctx.init(kmf.getKeyManagers(),null,null);
ssf=ctx.getServerSocketFactory();
}
catch(java.security.GeneralSecurityException thr) {
throw new IOException("Cannot create server socket factory using ephemeral keystore ("+thr+")",thr);
}
Notice that it uses a blank password for extracting the private key and certificates from the master key-store. That was my problem - I had, out of habit from using keytool, created the private key-pair with a password (the same password as the key-store).
Because I had a password on the certificate, the key and certificate were not extracted, and null was passed to sktkst.setKeyEntry(svrctfals,ctfkey,BLANK_PWD,ctfchn); However, setKeyEntry checks the passed Key using instanceof and concludes (correctly) that null is not an instanceof PrivateKey, resulting in the misleading error I was seeing.
The corrected code checks that a key and certificate are found and sends appropriate errors:
// CREATE EPHEMERAL KEYSTORE FOR THIS SOCKET USING THE DESIRED CERTIFICATE
try {
final char[] BLANK_PWD=new char[0];
SSLContext ctx=SSLContext.getInstance("TLS");
KeyManagerFactory kmf=KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
Key ctfkey=mstkst.getKey(svrctfals,BLANK_PWD);
Certificate[] ctfchn=mstkst.getCertificateChain(svrctfals);
KeyStore sktkst;
if(ctfkey==null) {
throw new IOException("Cannot create server socket factory: No key found for alias '"+svrctfals+"'");
}
if(ctfchn==null || ctfchn.length==0) {
throw new IOException("Cannot create server socket factory: No certificate found for alias '"+svrctfals+"'");
}
sktkst=KeyStore.getInstance("jks");
sktkst.load(null,BLANK_PWD);
sktkst.setKeyEntry(svrctfals,ctfkey,BLANK_PWD,ctfchn);
kmf.init(sktkst,BLANK_PWD);
ctx.init(kmf.getKeyManagers(),null,null);
ssf=ctx.getServerSocketFactory();
}
catch(java.security.GeneralSecurityException thr) {
throw new IOException("Cannot create server socket factory using ephemeral keystore ("+thr+")",thr);
}

Instead of using an ephemeral keystore, you could handle everything within a single SSLContext.
You would need to initialise your SSLContext using an custom X509KeyManager instead of using the one given by the default KeyManagerFactory. In this X509KeyManager,chooseServerAlias(String keyType, Principal[] issuers, Socket socket) should return a different alias depending on the local address obtained from the socket.
This way, you wouldn't have to worry about copying the private key from one keystore to another, and this would even work for keystore types from which you can't extract (and thus copy) but only use the private key, e.g. PKCS#11.

SSL and SocketChannel

Ideally, I only need a simple SSLSocketChannel.
I already have a component that reads and writes message over ordinary SocketChannel, but for some of these connections, I have to use SSL over the wire; the operations over these connections, however, are the same.
Does anyone knows a free SSLSocketChannel implementation (with the appropriate selector) or something similar? I've found this, but the selector doesn't accept it since its vendor isn't SUN.
I'm decoupling the reading_from/writing_to net logic from the insertion and retrieval of network data via a simple object, in order to use a SSLEngine without getting mad, but it's really tricky to implement that correctly, given the fact that I don't know the internals of SSL protocol...

Jetty has an NIO SSL implementation for their server: SslSelectorChannelConnector. You might want to peek at it for details on what its doing.
There is also an old (but decent) article from O'Reilly that explains the details about NIO + SSL along with example code.

TLS Channel is a simple library that does exactly that: wrapping a SSLContext (or SSLEngine) and exposing a ByteChannel interface, doing the heavy lifting internally.
(Disclaimer: I am the library's main author).

Check out Restlet's implementation it may do what you need, and it's all about NIO.
Restlet Engine Javadoc
Specifically the HttpClientCall. SetProtocol(HTTPS) - getResponseEntityChannel returns a ReadableByteChannel (getEntityChannel returns a WriteableByteChannel)

Not sure if this is what you're looking for, but may help... To create SSL/TLS enabled server sockets, I'm currently using code like the following (keystore.jks contains a self signed private/public key pair used for securing confirmation) - clients have a similar trust store which contains the signed certificate with the public key of that pair.
A bit of googling around getting that configured should get you underway.
String keyStorePath = "keystore.jks";
String keyStorePassword = "password";
KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
KeyStore keyStore = new KeyStore();
keyStore.load(new FileInputStream(keyStorePath), keyStorePassword);
keyManagerFactory.init(keyStore, keyStorePassword.toCharArray());
sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagerFactory.getKeyManagers(), null, new SecureRandom());
SSLContext sslContext = getServerSSLContext(namespace.getUuid());
SSLServerSocketFactory serverSocketFactory = sslContext.getServerSocketFactory();
// Create sockets as necessary