Develop Reference

Securing TCP traffic and certificate pinning?

We develop a Java application that serves requests over TCP. We also develop client libraries for the application, where each library supports a different language/platform (Java, .NET, etc.).
Until recently, TCP traffic was confined to a secure network. To support usage on an insecure network we implemented TLS using the recipes in java-plain-and-tls-socket-examples. There are recipes here for both server and client, and a script to generate an X.509 certificate. Below is a summary of the recipe for TLS with server-only authentication:
Create an X.509 root certificate that is self-signed.
Configure the server with a keystore file that contains the certificate's identifying data plus the public and private keys.
Configure the client with a trust store file that contains the same identifying data plus only the public key.
When connecting, the client validates the certificate received from the server by comparing its identifying data with the corresponding data in the client's trust store. (This looks like certificate pinning.)
We assume for now that this approach is valid for securing TCP traffic. Signing by a certificate authority seems unnecessary because we control both server and client.
Initial testing shows that the implementation is working in our Java server and Java client:
Client accepts a server certificate that matches data in the client's trust store.
Client rejects a non-matching server certificate.
TCP packets captured by tcpdump contain encrypted data.
.NET Client
We use SslStream to encrypt TCP traffic. As the documentation suggests, we do not specify a TLS version; instead we throw an exception if the version is below 1.2.
We're not confident about how to use X509Chain.ChainPolicy.CustomTrustStore correctly, because the documentation omits information like use cases for this type, and for option types like X509KeyStorageFlags and X509VerificationFlags.
The code below aims to mimic the recipe outlined above, i.e. configure a trust store data structure for the client to use when validating a server certificate. This approach seems equivalent to importing the certificate into the operating system's trust store.
// Import the trust store.
private X509Certificate2Collection GetCertificates(string storePath, string storePassword)
{
byte[] bytes = File.ReadAllBytes(storePath);
var result = new X509Certificate2Collection();
result.Import(bytes, storePassword, X509KeyStorageFlags.EphemeralKeySet);
return result;
}
// Callback function to validate a certificate received from the server.
// fCertificates stores the result of function GetCertificates.
private bool ValidateServerCertificate(
object sender,
X509Certificate certificate,
X509Chain chain,
SslPolicyErrors sslPolicyErrors)
{
// Do not allow this client to communicate with unauthenticated servers.
//
// With a self-signed certficate, sslPolicyErrors should be always equal to
// SslPolicyErrors.RemoteCertificateChainErrors.
var result = (SslPolicyErrors.RemoteCertificateChainErrors == sslPolicyErrors);
if (result)
{
// The values below are default values: set them to be explicit.
chain.ChainPolicy.VerificationFlags = X509VerificationFlags.NoFlag;
chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;
chain.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;
chain.ChainPolicy.CustomTrustStore.AddRange(fCertificates);
result = chain.Build((X509Certificate2)certificate);
}
return result;
}
// Initialize SslStream.
private SslStream GetStream(TcpClient tcpClient, string targetHost)
{
SslStream sslStream = new SslStream(
tcpClient.GetStream(),
false,
new RemoteCertificateValidationCallback(ValidateServerCertificate),
null
);
try
{
sslStream.AuthenticateAsClient(targetHost);
// require TLS 1.2 or higher
if (sslStream.SslProtocol < SslProtocols.Tls12)
{
throw new AuthenticationException($"The SSL protocol ({sslStream.SslProtocol}) must be {SslProtocols.Tls12} or higher.");
}
}
catch (AuthenticationException caught)
{
sslStream.Dispose();
throw caught;
}
return sslStream;
}
Initial testing has yielded results that vary depending on the operating system:
When deployed to Ubuntu on WSL2, this code:
Accepts a valid server certificate.
Rejects an invalid server certificate.
Encrypts TCP packets.
Automatically uses TLS 1.3.
Given a valid server certificate, the callback function argument sslPolicyErrors is equal to SslPolicyErrors.RemoteCertificateChainErrors (expected).
When deployed to MacOS:
This code automatically uses TLS 1.2.
Given a valid server certificate, the callback function argument sslPolicyErrors includes these values:
SslPolicyErrors.RemoteCertificateNameMismatch (unexpected).
SslPolicyErrors.RemoteCertificateChainErrors (expected).
Questions
In what ways could security be compromised with this .NET code?
Upon reviewing discussions about "certificate name mismatch" (see SslPolicyErrors.RemoteCertificateNameMismatch above), it seems that our server certificate should include a subjectAltName field to specify allowed DNS names. Is this necessary, or would it be reasonable, as we are using certificate pinning, to ignore sslPolicyErrors when validating the server certificate?

I can't answer your specific questions but here is some thoughts:
You have not mentioned anything about how the server authenticates clients. So you might consider implementing something like client certificates. If you control both you probably want some way to ensure random attackers cannot connect.
You might consider creating a threat model. In many cases it is the things that you haven't thought about that cause problems.
If you are handling national security data or financial data you might want an external audit. Such might even be required in some cases.
If there is no way an attacker could sell, use, or ransom the data, then you will probably not be directly targeted. So you might worry more about mass attacks against known vulnerabilities, i.e. keep all your software up to date.
Consider other ways to mitigate risks. Are your server/client running in least privilege possible? Are you using a DMZ? Are firewalls correctly configured? Are credentials for support etc well managed?

TLS on Embedded Jetty without JKS

My goal: I am working on an integration of Jetty Embedded that would make it simple to use. The interface would allow, among others, for integration of external sources for TLS certificates, without the use of the Java KeyStore.
This would allow for greater flexibility when building distributed web services (in my case an experimental, self-hosted CDN).
However, I am having problems building the integration. The stub implementation is in this repository.
What I've tried: I have tried replacing the key manager and the trust manager and set break points to every function in it. However, when trying to access the server, these break points are never triggered. Instead, I'm encountering this error:
javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:621)
at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:322)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:231)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:112)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:261)
at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:150)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:112)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:748)
I have tried analyzing the "standard" Jetty setup which a certificate from a keystore, but without much luck. I'm failing to find the point where Jetty is obtaining the cipher / certificate information that I should override.
My question: How can I get Jetty to use my own certificate source instead of the Java KeyStore and TrustStore?

#EJP pointed me in the right direction, so here's how to do it:
Here's how it needs to be done.
First, set up Jetty for TLS:
HttpConfiguration https = new HttpConfiguration();
https.addCustomizer(new SecureRequestCustomizer());
SslContextFactory sslContextFactory = new JettySslContextFactory(configuration.getSslProviders());
ServerConnector sslConnector = new ServerConnector(
server,
new SslConnectionFactory(sslContextFactory, "http/1.1"),
new HttpConnectionFactory(https)
);
sslConnector.setPort(httpsPort);
Note the class JettySslContextFactory. This class extends the built-in X509ExtendedKeyManager and needs to override the protected KeyManager[] getKeyManagers(KeyStore keyStore) throws Exception method in order to provider a custom KeyManager, like this:
#Override
protected KeyManager[] getKeyManagers(KeyStore keyStore) throws Exception {
return new KeyManager[] {
new JettyX509ExtendedKeyManager(certificateProviders)
};
}
In addition to that, the following steps are run in every connection:
The SNI matcher is consulted with the SNI host name. This seems to be the only place where the SNI host name is even available.
The key manager is consulted to get the alias (sort of a key ID) for a certain key type (EC or RSA). Here we need grab the host name from the SNI matcher, because otherwise we wouldn't know which host name to match on.
Based on the alias (key ID) we can then return the private key and certificate.
At least this is what I gathered from debugging this issue. The full code is online here.

How to read SNI extension in Java?

I want to find host name from TLS Client Hello Message. I want to find host name before java does complete handshake for transparent ssl proxy.
Is there any way to find SNI extension value without writing whole ssl handshake logic ?
Is Java supports ssl handshake with initial memory buffer ?
My Idea is:
Read TLS client hello message, parse it and find SNI value
Call sslSocket.startHandshake(initialBuffer) Initial buffer will contain TLS client hello packet data. So Java can do handshake.
Second Idea is to use SSLEngine class. But it seems a lot more implementation than requirement. I assume SSLEngine is used most of async in case which I don't require it.
Third idea is to implement complete TLS protocol.
Which idea is better ?

Both SSLSocket and SSLEngine lack (quite inexplicable) proper SNI support for server connections.
I came across the same problem myself and ended up writing a library: TLS Channel. It does not only that, it is actually a complete abstraction for SSLEngine, exposed as a ByteChannel. Regarding SNI, the library does the parsing of the first bytes before creating the SSLEngine. The user can then supply a function to the server channel, to select SSLContexts depending on the received domain name.

The accepted answer to this question is a bit old and does not really provide an answer to the question.
You should use a custom KeyManager when initializing the SSLContext and the trick is to use a javax.net.ssl.X509ExtendedKeyManager (note the Extended term). This will offer the possibility to expose the SSLEngine during the key alias selection process instead of the plain (useless) socket.
The method chooseEngineServerAlias(...) will be called during the SSL handshake process in order to select the proper alias (cert/key) from the KeyStore. And the SNI information is already populated and available in the SSLEngine. The trick here is to cast the SSLSession as an ExtendedSSLSession (note the Extended term) and then to getRequestedServerNames().
KeyManager[] km = new KeyManager[]
{
new X509ExtendedKeyManager()
{
public String chooseEngineServerAlias(String keyType, Principal[] issuers, SSLEngine engine)
{
if( engine.getHandshakeSession() instanceof ExtendedSSLSession )
{
List<SNIServerName> sni = ((ExtendedSSLSession)engine.getHandshakeSession()).getRequestedServerNames();
// select the proper certificate alias based on SNI here
}
}
}
}
SSLContext context = SSLContext.getInstance("TLS");
context.init(km, null, null);

Kerberos for Thrift?

I have a simple Thrift based java application I have written. It is really very simple, not much more than a "Hello World" message transport using Thrift in java. I have been told that I need to add Kerberos support to my message. I have done some googling and am surprised that Thrift does not already have Kerberos support in some form (or if it does, I cannot find it). I thought about writing my own wrapper using GSSAPI, but I cannot wrap/unwrap my Thrift message as that screws up the Thrift message format.
Has anyone ever Kerberized Thrift?.. or know how it would be done?
Thanks, in advance.

**So, I guess there is a way to do this via the SASL/GSS APIs. It confuses me as to why I don't see any great examples of this on the internet anywhere. However, I post an example of what I have created in the hopes that it will be a help to others... or that someone can correct my delusion of doing something useful here.
Sample Server code:
TServerSocket serverTransport = new TServerSocket(7911); // new server on port 7911
HelloWorldService.Processor<Iface> processor = new HelloWorldService.Processer<Iface>(new ThriftServerImpl()); // This is my thrift implementation for my server
Map<String, String> saslProperties = new HashMap<String, String>(); // need a map for properties
saslProperties.put(Sasl.QOP, "true");
saslProperties.put(Sasl.QOP, "auth-conf"); // authorization and confidentiality
TSaslServerTransport.Factory saslTransportFactory = new TSaslServerTransport.Factory(); // Creating the server definition
saslTransportFactory.addServerDefinition(
"GSSAPI", // tell SASL to use GSSAPI, which supports Kerberos
"myserviceprincipal", // base kerberos principal name - myprincipal/my.server.com#MY.REALM
"my.server.com", // kerberos principal server - myprincipal/my.server.com#MY.REALM
saslProps, // Properties set, above
new SaslRpcServer.SaslGssCallbackHandler())); // I don't know what this really does... but I stole it from Hadoop and it works.. so there.
Tserver server = new TThreadPoolServer(newTThreadPoolSErver.Args(serverTransport).transportFactory(saslTrasnportFactory).processor(processor));
server.serve(); // Thrift server start
Sample Client Code
TTransport transport = new TSocket("my.server.com", 7911); // client to connect to server and port
saslProperties.put(Sasl.QOP, "true");
saslProperties.put(Sasl.QOP, "auth-conf"); // authorization and confidentiality
TTransport saslTransport = new TSaslTransport(
"GSSAPI", // tell SASL to use GSSAPI, which supports Kerberos
null, // authorizationid - null
"myserviceprincipal", // base kerberos principal name - myprincipal/my.client.com#MY.REALM
"my.server.com", // kerberos principal server - myprincipal/my.server.com#MY.REALM
saslProps, // Properties set, above
null, // callback handler - null
transport); // underlying transport
TProtocol protocol = new TBinaryProtocol(saslTransport); // set up our new Thrift protocol
HelloWorldService.Client client = new HelloWorldService.Client(protocol); // Setup our thrift client
saslTransport.open();
String response = client.hello("Hi There"); // send message
System.out.println("response = " + response);
transport.close();
Other condsiderations:
* I set several java properties on both the client and the server.
- java.security.krb5.realm = MY.REALM // realm name
- java.security.krb5.kdc = my.kdc.com // kdc server
- javax.security.auth.useSubjectCredsOnly = false // Allow JAAS to get the TGT.
- java.security.auth.login.config = /etc/myapp/conf/jaas.conf - required jaas file
- sun.security.krb5.debug = true // helped with diagnosing problems.
* The jaas.conf file specified, above, needs to have two entries (maybe only one per server...). I cannot remember where I gleaned this information from.. but here is my file:
com.sun.security.jgss.initiate {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/myapp/conf/myapp.keytab"
useTicketCache=true
principal="myuserprincipal"
debug=true;
};
com.sun.security.jgss.accept {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/myapp/conf/myapp.keytab"
useTicketCache=false
principal="myserviceprincipal/my.server.com"
debug=true;
};
(back to considerations....)
* Despite having a Sasl.QOP of "auth-conf".. the first(?) message that gets transmitted is not encrypted. Maybe this is just a handshake, or something. The remaining messages appear to be encrypted, but this first one prints an ugly message to the console of "No encryption was performed by peer". It would be nice to not get that message, as it will cause grief down the road (warranted or not).
Anyway, I hope this helps someone... or alternatively can provoke some improvements that will help me. :) Hard to believe I spend 2-3 days doing this, and only a small amount of code came out of it , but I knew neither Kerberos or Thrift very well when I started.
Thanks for reading.

Choosing SSL client certificate in Java

Our system communicates with several web services providers. They are all invoked from a single Java client application. All the web services up until now have been over SSL, but none use client certificates. Well, a new partner is changing that.
Making the application use a certificate for the invocation is easy; setting javax.net.ssl.keyStore and javax.net.ssl.keyStorePassword will do it. However, the problem is now how to make it so that it only uses the certificate when invoking that particular web service. I guess more generally speaking, we'd like to be able to choose the client certificate to be used, if any.
One quick solution could be setting the system properties, invoking the methods, and then unsetting them. The only problem with that is that we're dealing with a multi-threaded application, so now we would need to deal with synchronization or locks or what have you.
Each service client is supposed to be completely independent from each other, and they're individually packaged in separate JARs. Thus, one option that has occurred to me (although we haven't properly analyzed it) is to somehow isolate each JAR, maybe load each one under a different VM with different parameters. That's merely an idea that I don't know how to implement (or if it's even possible, for that matter.)
This post suggests that it is possible to select an individual certificate from a key store, but how to attach it to the request seems to be a different issue altogether.
We're using Java 1.5, Axis2, and client classes generated with either wsimport or wsdl2java.

The configuration is done via an SSLContext, which is effectively a factory for the SSLSocketFactory (or SSLEngine). By default, this will be configured from the javax.net.ssl.* properties. In addition, when a server requests a certificate, it sends a TLS/SSL CertificateRequest message that contains a list of CA's distinguished names that it's willing to accept. Although this list is strictly speaking only indicative (i.e. servers could accept certs from issuers not in the list or could refuse valid certs from CAs in the list), it usually works this way.
By default, the certificate chooser in the X509KeyManager configured within the SSLContext (again you normally don't have to worry about it), will pick one of the certificates that has been issued by one in the list (or can be chained to an issuer there).
That list is the issuers parameter in X509KeyManager.chooseClientAlias (the alias is the alias name for the cert you want to picked, as referred to within the keystore). If you have multiple candidates, you can also use the socket parameter, which will get you the peer's IP address if that helps making a choice.
If this helps, you may find using jSSLutils (and its wrapper) for the configuration of your SSLContext (these are mainly helper classes to build SSLContexts). (Note that this example is for choosing the server-side alias, but it can be adapted, the source code is available.)
Once you've done this, you should look for the documentation regarding the axis.socketSecureFactorysystem property in Axis (and SecureSocketFactory). If you look at the Axis source code, it shouldn't be too difficult to build a org.apache.axis.components.net.SunJSSESocketFactory that's initialized from the SSLContext of your choice (see this question).
Just realized you were talking about Axis2, where the SecureSocketFactory seems to have disappeared. You might be able to find a workaround using the default SSLContext, but this will affect your entire application (which isn't great). If you use a X509KeyManagerWrapper of jSSLutils, you might be able to use the default X509KeyManager and treat only certain hosts as an exception. (This is not an ideal situation, I'm not sure how to use a custom SSLContext/SSLSocketFactory in Axis 2.)
Alternatively, according to this Axis 2 document, it looks like Axis 2 uses Apache HTTP Client 3.x:
If you want to perform SSL client
authentication (2-way SSL), you may
use the Protocol.registerProtocol
feature of HttpClient. You can
overwrite the "https" protocol, or use
a different protocol for your SSL
client authentication communications
if you don't want to mess with regular
https. Find more information at
http://jakarta.apache.org/commons/httpclient/sslguide.html
In this case, the SslContextedSecureProtocolSocketFactory should help you configure an SSLContext.

Java SSL clients will only send a certificate if requested by the server. A server can send an optional hint about what certificates it will accept; this will help a client choose a single certificate if it has multiple.
Normally, a new SSLContext is created with a specific client certificate, and Socket instances are created from a factory obtained from that context. Unfortunately, Axis2 doesn't appear to support the use of an SSLContext or a custom SocketFactory. Its client certificate settings are global.

I initialized EasySSLProtocolSocketFactory and Protocol instances for different endpoints and register the protocol with unique key like this:
/**
* This method does the following:
* 1. Creates a new and unique protocol for each SSL URL that is secured by client certificate
* 2. Bind keyStore related information to this protocol
* 3. Registers it with HTTP Protocol object
* 4. Stores the local reference for this custom protocol for use during furture collect calls
*
* #throws Exception
*/
public void registerProtocolCertificate() throws Exception {
EasySSLProtocolSocketFactory easySSLPSFactory = new EasySSLProtocolSocketFactory();
easySSLPSFactory.setKeyMaterial(createKeyMaterial());
myProtocolPrefix = (HTTPS_PROTOCOL + uniqueCounter.incrementAndGet());
Protocol httpsProtocol = new Protocol(myProtocolPrefix,(ProtocolSocketFactory) easySSLPSFactory, port);
Protocol.registerProtocol(myProtocolPrefix, httpsProtocol);
log.trace("Protocol [ "+myProtocolPrefix+" ] registered for the first time");
}
/**
* Load keystore for CLIENT-CERT protected endpoints
*/
private KeyMaterial createKeyMaterial() throws GeneralSecurityException, Exception {
KeyMaterial km = null;
char[] password = keyStorePassphrase.toCharArray();
File f = new File(keyStoreLocation);
if (f.exists()) {
try {
km = new KeyMaterial(keyStoreLocation, password);
log.trace("Keystore location is: " + keyStoreLocation + "");
} catch (GeneralSecurityException gse) {
if (logErrors){
log.error("Exception occured while loading keystore from the following location: "+keyStoreLocation, gse);
throw gse;
}
}
} else {
log.error("Unable to load Keystore from the following location: " + keyStoreLocation );
throw new CollectorInitException("Unable to load Keystore from the following location: " + keyStoreLocation);
}
return km;
}
When I have to invoke the web service, I do this (which basically replace "https" in the URL with https1, or https2 or something else depending on the Protocol you initialized for that particular endpoint):
httpClient.getHostConfiguration().setHost(host, port,Protocol.getProtocol(myProtocolPrefix));
initializeHttpMethod(this.url.toString().replace(HTTPS_PROTOCOL, myProtocolPrefix));
It works like a charm!