I'm trying to test my SSL implementation in unit tests and have one scenario I can't quite understand.
When I connect to a host once and fail, every following connection will fail also, even if it has the correct certificates. I'm assuming somewhere along the way I'd have to flush a cache.
Here is my code, server and client are running locally. I use one jks-File for both trustStore and keyStore. The error occurrs no matter what the initial error was, I'll always get the first error the next time.
If I don't perform the first request the second one works.
If you're wondering what the use case is here, we have some local servers that use https certificates from an internal PKI, when someone misconfigures a server or a certificate, we'd like to be able to change them obviously, without shutting down the whole VM.
//attempt a connection without certificates, will fail
try (final InputStream stream = new URL("https://localhost:" + port).openStream()){
System.out.println(IOUtils.toString(stream, Charset.defaultCharset()));
} catch (IOException e){
System.out.println("Failed to load: " + StackTraceUtil.getStackTrace(e));
}
//copies the jks file to a temporary location
final File jksFile = copyJKSFile();
//ignore host names, running locally, won't use this in production
HttpsURLConnection.setDefaultHostnameVerifier((hostname, sslSession) -> hostname.equalsIgnoreCase("localhost"));
//set the system properties
System.setProperty("javax.net.ssl.keyStore", jksFile.getAbsolutePath());
System.setProperty("javax.net.ssl.keyStorePassword", password);
System.setProperty("javax.net.ssl.trustStore", jksFile.getAbsolutePath());
System.setProperty("javax.net.ssl.trustStorePassword", password);
//this should work now
try (final InputStream stream = new URL("https://localhost:" + port).openStream()){
System.out.println(IOUtils.toString(stream, Charset.defaultCharset()));
} catch (IOException e){
System.out.println("Failed to load: " + StackTraceUtil.getStackTrace(e));
}
Thanks for any help!
So I found a solution and I thought I'd share it in case someone else would have this problem at some point.
The class javax.net.ssl.HttpsURLConnection uses a javax.net.ssl.SSLSocketFactory to load the Key- and TrustStore, which uses a javax.net.ssl.SSLContext internally. When you don't overwrite anything, it uses the default implementation, which loads the files and can't be reset once loaded.
So what I did was not to use the default implementation, but to set my own SSLContext, when I knew the files would change.
final SSLContext sslContext = SSLContext.getInstance("TLSv1.2");
sslContext.init(null, null, null);
HttpsURLConnection.setDefaultSSLSocketFactory(sslContext.getSocketFactory());
If you want to use older versions of TLS, the full list should be here
Related
I tried to submit a flink job that is already packaged in a JAR. Basically it consumes a kafka topic protected by SASL authentication, thus it requires a .jks file which I already include them in JAR and read in the code as:
try(InputStream resourceStream = loader.getResourceAsStream(configFile)){
properties.load(resourceStream);
properties.setProperty("ssl.truststore.location",
loader.getResource(properties.getProperty("ssl.truststore.location")).toURI().getPath());
}
catch(Exception e){
System.out.println("Failed to load config");
}
I tried to submit the job on two different (different VM specs) standalone server for the sake of testing. One server runs succesfully, but another throw a java.nio.file.NoSuchFileException, saying that my .jks file is not found. Can someone please point out the possible issue on it?
Here, the flink is deployed on a standalone cluster mode with the following version:
Flink version: 1.14.0
Java version: 11.0.13
I realize my question was really silly. This part actually returns null and trigger exception.
loader.getResource(properties.getProperty("ssl.truststore.location")).toURI().getPath()
The problem was that I submit the job through web UI thus I couldn't see the printed message. Thus, the filename resolves to the original one stored under the configFile, which is a relative path. Why one machine works and another one doesn't? Cause I previously somehow has the .jks on my homedir for another testing :).
For others to not jump into this mistake, here is the summary of what will .getResource() resolve if run from IDE (gradle run task) and jar, respectively.
// file:home/gradle-demo/build/resources/main/kafka-client.truststore.jks
// jar:file:home/gradle-demo/build/libs/gradle-demo-1.0-SNAPSHOT.jar!/kafka-client.truststore.jks
System.out.println(loader.getResource("kafka-client.trustore.jks").toString());
// home/gradle-demo/build/resources/main/kafka-client.truststore.jks
// file:home/gradle-demo/build/libs/gradle-demo-1.0-SNAPSHOT.jar!/kafka-client.truststore.jks
System.out.println(loader.getResource("kafka-client.trustore.jks").getPath());
// home/gradle-demo/build/resources/main/kafka-client.truststore.jks
// null
System.out.println(loader.getResource("kafka-client.trustore.jks").toURI().getPath());
// file:home/gradle-demo/build/resources/main/kafka-client.truststore.jks
// jar:file:home/gradle-demo/build/libs/gradle-demo-1.0-SNAPSHOT.jar!/kafka-client.truststore.jks
System.out.println(loader.getResource("kafka-client.trustore.jks").toURI());
kafka-client:2.4.1
org.apache.kafka.common.security.ssl.SslEngineBuilder#285
try (InputStream in = Files.newInputStream(Paths.get(path))) {
KeyStore ks = KeyStore.getInstance(type);
// If a password is not set access to the truststore is still available, but integrity checking is disabled.
char[] passwordChars = password != null ? password.value().toCharArray() : null;
ks.load(in, passwordChars);
return ks;
} catch (GeneralSecurityException | IOException e) {
throw new KafkaException("Failed to load SSL keystore " + path + " of type " + type, e);
}
It looks like we should put jks file in file system(nfs or hdfs) where task manager can access by absolute path.
I have a code which calls a rest api protected with certificate, and that code was working for some time without issues, until I migrate application from Wildfly 10 to Wildfly 16.
The code itself is straight forward, it creates http request and set custom socket factory: `
private SSLSocketFactory getSSLSocketFactory() {
char[] certPassword = {}; // password hidden from you
try {
final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
final SSLContext sslContext = SSLContext.getInstance("TLS");
final KeyStore keyStore = KeyStore.getInstance("PKCS12");
try (FileInputStream fileInputStream = new FileInputStream(new File("cert.pk12"))) {
keyStore.load(fileInputStream, certPassword);
} catch (final Exception e) {
logger.error("....", e);
}
this.readLoadCertificateFile(keyStore);
kmf.init(keyStore, certPassword);
sslContext.init(kmf.getKeyManagers(), new TrustManager[]{new AnyTrust()}, null);
return sslContext.getSocketFactory();
} catch (Exception e) {
logger.error(".....", e);
}
throw new IllegalStateException("....");
}
HTTPRequest req = ....
req.setSSLSocketFactory(getSSLSocketFactory());
tokenHttpResp = req.send();`
`
All seems good but when I run this code from within WF16 it throws
IOException: Failed to load .p12 keystore:C:\Cert\cert.p12; error
constructing MAC: java.lang.SecurityException: JCE cannot authenticate
the provider BC;
org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.engineLoad::-1
java.security.KeyStore.load in KeyStore.java::1445
I checked for the bouncy castle lib, and I don't have it in the application itself...
If any one knows what the issue could be I would appreciate any input.
Alternatively, I'm looking to move this socket factory creation in to the container itself, seems like Wildfly Elytron subsystem specifically designed for this, is it a good idea?
Answering my own question.
The error message like "JCE cannot authenticate the provider BC" indicating that the jar file, the Security Provider is loaded from, cannot be verified by JVM. Either the jar is not signed or signature cannot be verified. In my case, the newer Wildfly version has a newer version of bouncy castle library, which for some reason, cannot be verified by Java 8. Interesting enough, that it is fine with Java 10. Some people on the Internet says that this issue only occurs in Oracle's JVM and does not exist for Open JDK, I haven't tested it, just think it worth to mention.
To overcome issue you need to tell JVM to trust the security provider, for that, make sure that the Security Provider you want to use/JVM decided to use, mentioned in jre/lib/security/java.security file, it should have line like:
security.provider.11=org.bouncycastle.jce.provider.BouncyCastleProvider
Then copy Jars with Security Provider into /jre/lib/ext folder
Can anybody tell how we interact with security enables Riak KV Cluster using java client.
I tried with following java code to interact with security enabled riak cluster, but getting SSLEngine Problem, Below is the java code ......
InputStream inputStream = null;
KeyStore ks = null;
try {
inputStream = new FileInputStream("/etc/ssl/certs/rootCA.pem");
CertificateFactory certFactory = CertificateFactory.getInstance("X.509");
X509Certificate caCert = (X509Certificate) certFactory.generateCertificate(inputStream);
inputStream.close();
ks = KeyStore.getInstance(KeyStore.getDefaultType());
ks.load(null, "password".toCharArray());
ks.setCertificateEntry("cacert", caCert);
} catch (Exception e) {
e.printStackTrace();
}
synchronized (RiakConfig.class) {
List<RiakNode> riakNodeList = new ArrayList<RiakNode>();
for (final String riakServer : riakServerArray) {
RiakNode node = new RiakNode.Builder()
.withMinConnections(10)
.withMinConnections(50)
.withRemoteAddress(riakServer.split(":")[0])
.withRemotePort(Integer.parseInt(riakServer.split(":")[1]))
.withAuth("riakuser", "riakuser", ks)
.build();
riakNodeList.add(node);
}
cluster = new RiakCluster.Builder(riakNodeList).build();
cluster.start();
}
suggenst anyone how we do that???
getting SSLEngine Problem
Please, always provide the specific problem you are getting. "getting SSLEngine Problem" is a useless bit of information.
Based on this document it appears you are using the correct steps. Without more information about the security issue you are getting, further help is impossible. Most likely you do not have your certificates set up correctly. The RabbitMQ documentation includes a comprehensive TLS/SSL troubleshooting guide that can help you determine if your certificates were created correctly.
Additionally, I suggest that you review how the Riak Java Client sets up certificates and then uses them.
This part of the Makefile is where certs are imported with keytool:
https://github.com/basho/riak-java-client/blob/develop/Makefile#L43-L62
This is a class that uses the key store to create connections for use in tests:
https://github.com/basho/riak-java-client/blob/develop/src/test/java/com/basho/riak/client/core/operations/itest/RiakJKSConnection.java
Achually this is working fine for single node, in cluster i am facing SSL Engine Problem.
I am new to opc ua and not a pro in java. While setting up a client in java I'm having trouble with the certificate dealing. I want to connect to the server via Basic 256, SignAndEncrypt. As I understand, in this stage of security a certificate, created or loaded by the client, is send to the server, where it must be accepted. The server then sends a certificate back to the client, which then needs to be accepted by the client. Please, correct me, if I'm wrong.
Creating/Loading a certificate on the client side and sending it to the server does already work fine (see code below) and I can then accept it on the server side manually. But after that I'm stuck: How can I see this certificate validation in my code and how can I find the server certificate, let alone accept it?
I used the SampleConsoleClient of opc ua for some orientation during implementation. But in contrast to there, I do not use any user input.
Here's some of my code so far.
Initialization:
try {
client = new UaClient(serverUri);
} catch (final URISyntaxException e) {
throw new InitializationException("The server uri has an invalid syntax.", e);
}
try {
client.setApplicationIdentity(createApplicationIdentity());
} catch (final SecureIdentityException e) {
throw new InitializationException(
"Application Identity could not be created due to a Security Identity Exception.", e);
} catch (final IOException e) {
throw new InitializationException("Application Identity could not be created due to an IO Exception.",
e);
}
createApplicationIdentity():
final ApplicationDescription appDescription = new ApplicationDescription();
appDescription.setApplicationName(new LocalizedText(APPLICATION_NAME, Locale.ENGLISH));
appDescription.setApplicationUri(APPLICATION_URI);
appDescription.setProductUri(PRODUCT_URI);
appDescription.setApplicationType(ApplicationType.Client);
// Setting security features
client.setSecurityMode(SecurityMode.BASIC256_SIGN_ENCRYPT);
client.setCertificateValidator(validator);
validator.setValidationListener(myValidationListener); //myValidationListener is similar to most lines in MyCertificateValidationListener in the opc ua samples
final File privatePath = new File(validator.getBaseDir(), "private");
final KeyPair issuerCertificate = null;
final int[] keySizes = null;
final ApplicationIdentity identity = ApplicationIdentity.loadOrCreateCertificate(appDescription,
"Sample Organisation", "opcua", privatePath, issuerCertificate, keySizes, true);
identity.setApplicationDescription(appDescription);
return identity;
After initializing, I try to connect like this (with annotation, how I imagine the connection could work properly):
final String securityPolicy = client.getEndpoint() == null
? client.getSecurityMode().getSecurityPolicy().getPolicyUri()
: client.getEndpoint().getSecurityPolicyUri();
client.setSessionName(String.format("%s#%s/Session%d", APPLICATION_NAME,
ApplicationIdentity.getActualHostNameWithoutDomain(), ++sessionCount));
try {
//Idea: catch the server certificate and accept it. Only if that was possible: connect
client.connect();
} catch (final ServiceException e) {
e.printStackTrace();
}
client.setKeepSubscriptions(false);
// After that resolving namespace index (works fine)
}
And the error, that is thrown:
WARN (?:?): /<IPofServer> Error org.opcfoundation.ua.common.ServiceResultException: Bad_SecurityChecksFailed
(0x80130000) "An error occurred verifying security." at
org.opcfoundation.ua.transport.tcp.io.TcpConnection$ReadThread.run(Unknown Source)
com.prosysopc.ua.client.ConnectException: Failed to create secure channel to server: : opc.tcp://<IPofServer>
[http://opcfoundation.org/UA/SecurityPolicy#Basic256,SignAndEncrypt]
ServiceResult=Bad_SecurityChecksFailed (0x80130000) "An error occurred verifying security."
at com.prosysopc.ua.client.UaClient.n(Unknown Source)
at com.prosysopc.ua.client.UaClient.connect(Unknown Source)
at *lineOfCode*
Caused by: org.opcfoundation.ua.common.ServiceResultException:
Bad_SecurityChecksFailed (0x80130000) "An error occurred verifying security."
at org.opcfoundation.ua.transport.tcp.io.TcpConnection$ReadThread.run(Unknown Source)
With the lineOfCode being client.connect().
Thanks in advance for the help!!
The server will send it's certificate to the client. The client then has to
verify the validity of the certificate. This amounts to verifying the signature of the certificate, checking the validity, whether the hostname in the certificate matches the hostname in the endpoint, checking CRLs and so forth. Usually the SDK (the validator) should do this for you, but you might need to feed some parameters into the validator which checks should actually be performed. The security policy Basic256 imposes some minimal requirements on the certificate which the certificate should meet, of course. You can check the requirements here: http://opcfoundation-onlineapplications.org/profilereporting/ -- go to Security Category -> Facets -> Security policy.
check whether the server certificate is trusted. This usually amounts to checking whether a copy of the (puclic key) certificate has been put into some certicate store chosen as a trust store. If you write the client it's up to you to say which store to choose, but you will need to tell the validator where to look. I don't know that much about OPc UA development in Java, but you should check which certificate stores the validator expects. Maybe there is a default keyfile.
(On server side the same happens with the client certificate).
This asssumes you are starting out with self-signed certificates. If you are using certificates signed by a CA both applications (server and client) will need to be able to verify the whole chain of the other party. It can be stored locally in some store or can be send by the other party. At least one certificate in the chain has to be trustest (has to be put into the trust store).
For a general description on how UA security works have a look at this link:
https://opcfoundation.org/wp-content/uploads/2014/08/11_OPC_UA_Security_How_It_Works.pdf
For a detailed account you should consult the specification, available at GitHub.
Edit: one addtional remark which may help here: you seem to be using some SDK for the purpose in question. While validation of certificates, i.e. doing the signature checks etc, is usually covered by such an SDK the configuration of the application is the task of the application (programmer). This includes the location where you store trusted certificates and where and how you gather together missing parts of certificate chains. You might first try to check how demo clients and servers deal with this task, in other words check out the configuration tasks for such applications by trying to create a secure connection from, say, UA Expert to the sample servers from the OPC foundation. In the .Net SDK of the OPC foundation the location for the trust store defaults to a certain directory in the file system (a subfolder of C:\ProgramData\OpcFoundation, it's Windows only). You can, however, overwrite this when you initialize the validator. Other clients use their own directory structure for the storage of trusted certificates
You are obviously referring to the Prosys OPC UA Java SDK.
What always happens first, when you try to establish a secure connection for the first time, is that the server will deny access to your client application - and returns Bad_SecurityChecksFailed.
Only, after you have told the server to trust (the certificate of) your client application, will you get to the phase where the client application will try to verify the server's certificate - and your 'validationListener' will be triggered.
Thank you all for your answers. In the meantime, I tried basically copy/paste and modifying the connect() and initalize() methods from the SampleConsoleClient in the Prosys SDK samples And that worked. Guess it had something to do with renewing some information, but I'm not quite sure of that... Fact is, my application is now working, but thanks for your efforts!
I have an Apache web server that runs several TLS virtualhosts with different certs and SNI.
I can access the various virtual hosts just fine using curl (presumably SNI makes it work). I can also access them fine with a little command-line Java program that basically just openConnection()s on a URL.
In my Tomcat application, the basic same client-side code accesses the same Apache server as a client, but always ends up with the default cert (defaulthost.defaultdomain) instead of the cert of the virtual host that was specified in the URL that it attempts to access. (This produces a SunCertPathBuilderException -- basically it can't verify the certificate path to the cert, which of course is true as it is a non-official cert. But then the default cert should not be used anyway.)
It's just as if SNI had been deactivated client-side in my application / Tomcat. I am at a loss why it should behave differently between my app and the command-line; same JDK, same host etc.
I found property jsse.enableSNIExtension, but I verified that it is set to true for both cases. Questions:
Any ideas, even wild ones, why these two programs behave differently?
Any ideas how I would debug this?
This is Arch Linux on 86_64, JDK 8u77, Tomcat 8.0.32.
This answer comes late, but we just have hit the problem (I can't believe it, it seems a very big bug).
All what it said seems true, but it's not default HostnameVerifier the culprit but the troubleshooter. When HttpsClient do afterConnect first try to establish setHost (only when socket is SSLSocketImpl):
SSLSocketFactory factory = sslSocketFactory;
try {
if (!(serverSocket instanceof SSLSocket)) {
s = (SSLSocket)factory.createSocket(serverSocket,
host, port, true);
} else {
s = (SSLSocket)serverSocket;
if (s instanceof SSLSocketImpl) {
((SSLSocketImpl)s).setHost(host);
}
}
} catch (IOException ex) {
// If we fail to connect through the tunnel, try it
// locally, as a last resort. If this doesn't work,
// throw the original exception.
try {
s = (SSLSocket)factory.createSocket(host, port);
} catch (IOException ignored) {
throw ex;
}
}
If you use a custom SSLSocketFactory without override createSocket() (the method without parameters), the createSocket well parametrized is used and all works as expected (with client sni extension). But when second way it's used (try to setHost en SSLSocketImpl) the code executed is:
// ONLY used by HttpsClient to setup the URI specified hostname
//
// Please NOTE that this method MUST be called before calling to
// SSLSocket.setSSLParameters(). Otherwise, the {#code host} parameter
// may override SNIHostName in the customized server name indication.
synchronized public void setHost(String host) {
this.host = host;
this.serverNames =
Utilities.addToSNIServerNameList(this.serverNames, this.host);
}
The comments say all. You need to call setSSLParameters before client handshake. If you use default HostnameVerifier, HttpsClient will call setSSLParameters. But there is no setSSLParameters execution in the opposite way. The fix should be very easy for Oracle:
SSLParameters paramaters = s.getSSLParameters();
if (isDefaultHostnameVerifier) {
// If the HNV is the default from HttpsURLConnection, we
// will do the spoof checks in SSLSocket.
paramaters.setEndpointIdentificationAlgorithm("HTTPS");
needToCheckSpoofing = false;
}
s.setSSLParameters(paramaters);
Java 9 is working as expected in SNI. But they (Oracle) seem not to want fix this:
https://bugs.openjdk.java.net/browse/JDK-8072464
https://bugs.openjdk.java.net/browse/JDK-8144566
and many more
After some hours of debugging the JDK, here is the unfortunate result. This works:
URLConnection c = new URL("https://example.com/").openConnection();
InputStream i = c.getInputStream();
...
This fails:
URLConnection c = new URL("https://example.com/").openConnection();
((HttpsURLConnection)c).setHostnameVerifier( new HostnameVerifier() {
public boolean verify( String s, SSLSession sess ) {
return false; // or true, won't matter for this
}
});
InputStream i = c.getInputStream(); // Exception thrown here
...
Adding the setHostnameVerifier call has the consequence of disabling SNI, although the custom HostnameVerifier is never invoked.
The culprit seems to be this code in sun.net.www.protocol.https.HttpsClient:
if (hv != null) {
String canonicalName = hv.getClass().getCanonicalName();
if (canonicalName != null &&
canonicalName.equalsIgnoreCase(defaultHVCanonicalName)) {
isDefaultHostnameVerifier = true;
}
} else {
// Unlikely to happen! As the behavior is the same as the
// default hostname verifier, so we prefer to let the
// SSLSocket do the spoof checks.
isDefaultHostnameVerifier = true;
}
if (isDefaultHostnameVerifier) {
// If the HNV is the default from HttpsURLConnection, we
// will do the spoof checks in SSLSocket.
SSLParameters paramaters = s.getSSLParameters();
paramaters.setEndpointIdentificationAlgorithm("HTTPS");
s.setSSLParameters(paramaters);
needToCheckSpoofing = false;
}
where some bright mind checks whether the configured HostnameVerifier's class is the default JDK class (which, when invoked, just returns false, like my code above) and based on that, changes the parameters for the SSL connection -- which, as a side effect, turns off SNI.
How checking the name of a class and making some logic depend on it is ever a good idea escapes me. ("Mom! We don't need virtual methods, we can just check the class name and dispatch on that!") But worse, what in the world does SNI have to do with the HostnameVerifier in the first place?
Perhaps the workaround is to use a custom HostnameVerifier with the same name, but different capitalization, because that same bright mind also decided to do case-insensitive name comparison.
'nuff said.
This is a Java 8 bug (JDK-8144566) fixed by 8u141. See Extended server_name (SNI Extension) not sent with jdk1.8.0 but send with jdk1.7.0 for more.