I am trying to create self-signed certificate for usage with a Netty (4.1.86) Websocket Server. It's running fine without SSL as well as with a certificate created with the SelfSignedCertificate class. But I am struggling when creating a self signed certificate with openssl.
This:
SslContext sslCtx = SslContextBuilder.forServer(new File(certFile), new File(keyFile), password).build();
Throws the following error:
ERROR Thread-5 com..application.NettyWSServer - Exception caught:
java.lang.IllegalArgumentException: File does not contain valid private key: /home/johnny/testbench/application/app.pkcs8.key
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:386)
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120)
at com..application.NettyWSServer.start(NettyWSServer.java:78)
at com..application.ApplicationLauncher$2.run(ApplicationLauncherncher.java:315)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
at java.base/javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122)
at java.base/javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168)
at io.netty.handler.ssl.SslContext.generateKeySpec(SslContext.java:1084)
at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1170)
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1133)
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384)
... 4 more
Here's how I create certFile (app.pem) and keyFile (app.pkcs8.key):
openssl genrsa -out app.key 2048
openssl pkcs8 -topk8 -in app.key -out app.pkcs8.key
openssl req -x509 -new -nodes -key app.key -sha256 -days 1024 -out app.pem
As per my understanding Netty needs a pkcs8 format key:
https://netty.io/4.1/api/io/netty/handler/ssl/SslContextBuilder.html#forServer-java.io.File-java.io.File-
Working solution thanks to dave_thompson_085:
openssl genrsa -out app.key 2048
openssl pkcs8 -topk8 -v1 PBE-SHA1-3DES -nocrypt -in app.key -out app.pkcs8.key
openssl req -x509 -new -nodes -key app.key -sha256 -days 1024 -out app.pem
More exactly the default SSLContext path needs either PKCS8-unencrypted or PKCS8-encrypted using a password-based encryption (PBE) scheme that is defined uniquely by the OID at the beginning of the AlgorithmIdentifier, like the schemes in PKCS5v1 (now retronymed PBES1) or PKCS12, but not the newer/better family of schemes in PKCS5v2 named PBES2, because the OID for PBES2 -- 1.2.840.113549.1.5.13 which you see in your exception message -- does not identify a single scheme and thus is not sufficient for Java to instantiate a SecretKeyFactory (causing the exception). And openssl pkcs8 -topk8 by default uses a PBES2 scheme since 1.1.0 in 2016.
But the netty 4.1 SSLContext.toPrivateKey method(s) (which you can see and your stacktrace confirms SSLContextBuilder calls) will instead use BouncyCastlePemReader if available, and Bouncy can handle both older PBE schemes and PBES2, and also OpenSSL 'traditional' format files that are not PKCS8 at all (and if/when encrypted use a scheme based on OpenSSL's own PBKDF EVP_BytesToKey).
Thus I think (but can't test) you have 3 options:
supply BouncyCastle. I think you need bcprov (or bcprov-ext) and bcpkix; you may now need bcutil (due to a reorg in recent versions I haven't fully grokked).
encrypt your file with a non-PBES2 scheme. The PBES1 schemes are all now broken or easily breakable; the 'best' (least bad) PKCS12 scheme is officially named pbeWithSHA1And3-KeyTripleDES-CBC (!) but OpenSSL has a more convenient 'shortname':
openssl pkcs8 -topk8 -v1 PBE-SHA1-3DES [-in oldfile -out newfile]
Technically you could also accomplish this by leaving the default on a very old version of OpenSSL, but that's usually more difficult(*) and in my view not really different. (* not for me, because I have my own archive of old versions, but I'm weird)
don't encrypt your keyfile, (only!) if you can adequately secure it otherwise, for example by access controls on your system(s) (and any backups). Use:
openssl pkcs8 -topk8 -nocrypt [-in oldfile -out newfile]
or a bit more simply:
openssl pkey [-in oldfile -out newfile]
Related
I need to read an encrypted ID token in an OpenID Connect flow with a private and public key. Don't need a cert.
First I ran:
openssl req -new -newkey rsa:2048 -keyout key.pem -pubkey -out pubreq.pem -subj "/CN=MyKey"
Got the public and private keys.
Then I ran:
openssl pkcs12 -export -inkey key.pem -out encryptedprivatekeytest.p12 -nocerts
So far so good. The file encryptedprivatekeytest.p12 contains the private key. Now I just need to create a jks truststore file from the public key in pubreq.pem.
Literally no matter how hard I try to find out how to, I just can't.
Any takers?
You have actually created a certificate signing request with the above command.
To generate just a public-private key pair, you can use:
openssl genrsa -out private.key 2048
openssl rsa -in private.key -pubout -out public.key
However, a trust store contains certificates, not public keys.
I found that Kafka 2.7.0 supports PEM certificates and I decided to try setting up the broker with DigiCert SSL certificate. I used new options and I did everything like in example in KIP-651. But I get the error:
[2021-01-20 17:54:55,787] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.config.ConfigException: Invalid value javax.net.ssl.SSLHandshakeException: no cipher suites in common for configuration A client SSLEngine created with the provided settings can't connect to a server SSLEngine created with those settings.
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:98)
at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:72)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)
at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:97)
at kafka.network.Processor.<init>(SocketServer.scala:790)
at kafka.network.SocketServer.newProcessor(SocketServer.scala:415)
at kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:288)
at kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:287)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:254)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:251)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551)
at scala.collection.AbstractIterable.foreach(Iterable.scala:920)
at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:251)
at kafka.network.SocketServer.startup(SocketServer.scala:125)
at kafka.server.KafkaServer.startup(KafkaServer.scala:303)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
openssl x509 -in certificate.pem -text:
Certificate:
...
Signature Algorithm: ecdsa-with-SHA384
...
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
I've tried using different versions of Oracle JDK and OpenJDK, but no effect.
I also tried to set certificate with options:
ssl.keystore.type=PEM
ssl.keystore.location=/opt/kafka/certs/certificate.pem
ssl.key.password=null
And I got а new error:
[2021-01-23 20:33:21,552] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.errors.InvalidConfigurationException: Failed to load PEM SSL keystore /opt/kafka/certs/certificate.pem
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: Invalid PEM keystore configs
Caused by: java.io.IOException: overrun, bytes = 111
at javax.crypto.EncryptedPrivateKeyInfo.<init>(EncryptedPrivateKeyInfo.java:92)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(DefaultSslEngineFactory.java:512)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.createKeyStoreFromPem(DefaultSslEngineFactory.java:462)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.<init>(DefaultSslEngineFactory.java:435)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedPemStore.load(DefaultSslEngineFactory.java:412)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:349)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedPemStore.<init>(DefaultSslEngineFactory.java:405)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:293)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:161)
at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:136)
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:93)
at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:72)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)
at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:97)
at kafka.network.Processor.<init>(SocketServer.scala:790)
at kafka.network.SocketServer.newProcessor(SocketServer.scala:415)
at kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:288)
at kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:287)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:254)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:251)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551)
at scala.collection.AbstractIterable.foreach(Iterable.scala:920)
at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:251)
at kafka.network.SocketServer.startup(SocketServer.scala:125)
at kafka.server.KafkaServer.startup(KafkaServer.scala:303)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
But if convert this certificate to jks:
openssl pkcs12 -export -in certificate.pem -out certificate.p12
keytool -importkeystore -srckeystore certificate.p12 -srcstoretype pkcs12 -destkeystore certificate.jks
echo 'ssl.keystore.location=/opt/kafka/certs/certificate.jks' >>server.properties
echo 'ssl.keystore.password=password' >>server.properties
Broker works correctly.
Could this be a bug or am I doing anything wrong? Full log here
I think this might be because the private key you are using is encrypted with a PBES2 scheme. You can use OpenSSL to convert the original key and use PBES1 instead:
openssl pkcs8 -in old_kafka.key -passout "pass:password" -topk8 -v1 PBE-SHA1-3DES -out kafka.key
You can find more info of which PKCS#5 encryption algorithms are available with PBES1 and PBES2 in OpenSSL's PKCS#8 man page.
Also, PBES2 support in OpenJDK is mentioned in this issue.
I would like to have SSL termination on HAProxy, using my own self-signed certificates, and to validate client access using client certificates I create.
I create the server (which is also the CA) certificates the following way:
openssl genrsa -out ca.key 1024
openssl req -new -key ca.key -out ca.csr
openssl x509 -req -days 365 -in ca.csr -signkey ca.key -out ca.crt
and:
cat ca.crt ca.key > haproxy.pem
at HAProxy, I configure:
bind *:443 ssl crt /path/server.pem ca-file /path/ca.crt verify required crt-ignore-err all
I create the client certificates in a similar way:
openssl req -new -key client.key -out client.csr
openssl x509 -req -days 365 -in client.csr -signkey ca.key -out client.crt
cat client.crt client.key > client.pem
My logic is: I'm creating a client key, a certificate signing request for it, and then I sign it using the CA (which is also the server certificate, so there's a simple chain that the server would recognize).
To test, I first try with the server certificate as the client cert:
curl https://my.service:443/ping -E ./haproxy.pem -k
pong
ok, it works. Now I try with the client certificate as the client certificate:
curl https://my.service:443/ping -E ./client.pem -k
curl: (58) unable to set private key file: './client.pem' type PEM
My question:
1) I would like to create a client certificate that this server will accpet, and test it using curl.
2) I would like to import this certificate and the CA into a new java keystore / truststore using keytool, so that Java (Jersey client) code could access the same content.
I have spent 2 days on 1/2.
I'm pretty sure someone that's done this before could answer this in 5m. Or so I hope. :)
Thanks!
1. create client cert
wrong: openssl x509 -req -signkey creates a self-signed cert, which by definition means the key in the cert (the subject key) is the public half of the same key whose private half signs the cert. The documentation for the cert (not req) case is clear that it replaces the key previously in the cert with the signing key. The -req doc is less clear, but it does the same thing; it puts in the cert the subject name from the CSR, also as the issuer, and the key from -signkey. You have used a CSR containing the client name, but a -signkey containing the CA key, producing an unusable chimera.
right: to sign a "child" (not self-signed) cert with x509, use -CA and possibly -CAkey as described in the documentation https://www.openssl.org/docs/apps/x509.html#SIGNING-OPTIONS (or man [where] x509 on any Unix with openssl doc installed). If there is or ever will be more than one child cert for a given CA (defined by its DN), either use the serial-number file scheme to automatically and conveniently assign sequential serial numbers, or use -set_serial to manually assign unique serial numbers (sequential is the easiest way to do unique, but if you a prefer another way that's okay).
aside: for the self-signed CA (and server?!) cert, you don't need separate req -new and x509 -req -signkey steps, you can do it in one req -new -x509. See the doc/manpage for req. In fact you don't need a separate genrsa step, req -newkey [-nodes] -x509 can do that as well. One note: in OpenSSL 1.0.0+ this generates the generic PKCS#8 format keyfile instead of the "legacy" PKCS#1 format used by genrsa (and rsa); all OpenSSL functions can accept either, but some other things might not. In particular last I checked (a while ago) the Wireshark option to decrypt SSL/TLS using server key for akRSA (there are other options too) accepted only PKCS#1 not PKCS#8.
2. use in Java (Jersey). Note that any SSL/TLS client doing client authentication, including Java, needs both the certificate and the privatekey, and in most cases the certificate uses "chain" or "intermediate" certs which you need also. Some people (cough) Microsoft (cough) encourage you to misunderstand and ignore this important distinction, but if you try to use only a certificate it won't work at all. On the other hand a truststore entry needs only the certificate, almost always only the root (CA) certificate, and usually must have only the certificate. Your situation where the same person operates the CA and server and client(s) is somewhat unusual for PKC.
2a. maybe just convert to pkcs12. Java does not directly support the openssl format(s) for keys, but both Java and openssl support PKCS#12 (and so do Microsoft, Mozilla, Apple, and probably others). Since you combined client key and (leaf) cert in client.pem do
openssl pkcs12 -export <client.pem -CA ca.crt [-name whatever] >client.p12
# if you use separate key,cert files see the doc about -in and -inkey
Java crypto (JCE and JSSE) can use this PKCS#12 as a keystore, if you can configure the keystore "type" (as pkcs12). The default SSLSocketFactory supports this, and so do other apps I've used, but I don't use Jersey and don't know what it does here. PKCS#12 isn't generally supported to carry "separate" certs (without privatekey), but in your case the CA cert for the client is also the cert for the server, so it will happen to work as your truststore as well; otherwise you would need to import the server CA or server selfsigned cert (only cert not privatekey) into a JKS truststore (which might be the default truststore in JRE/lib/security/[jsse]cacerts).
2b. maybe further convert to JKS. If Jersey cannot use PKCS#12 directly, Java can convert it to JKS which any sane Java code can use, like:
keytool -importkeystore -srckeystore client.p12 -srcstoretype pkcs12 -destkeystore client.jks
UPDATE 2018: after this answer was written Java support of PKCS12 increased, making it less often necessary to convert to JKS. 8u60 released fall 2017 and up still defaults to keystore type JKS, but as a special feature(?) type JKS can actually read (though not write) PKCS12; see the release notes and item keystore.type.compat in file JRE/lib/security/java.security. Java9 released 2017 makes the default keystore type PKCS12 which (as expected) reads and writes PKCS12, although explicit JKS no longer reads PKCS12. But if you do need to convert with Java9 for some reason, you now need to specify -deststoretype jks but no longer need to specify -srcstoretype pkcs12.
I am writing a .NET client app that consumes a Java web service and need to sign sent requests (related to this other question).
I have been supplied with a private.key file (and a .X509 certificate) and a Java source example. The certificate looks like the public key of service, and the private.key is what I use to sign requests.
In the Java source, I can see they convert the file to a byte array and pass it into the constructor of the PKCS8EncodedKeySpec class.
A bit of googling suggests this file is a private key hash (though I may be wrong).
Is there any way to use this in .Net or convert it to something .Net can use?
This link mentions converting a public/private key, but I don't have both, or if it would work. Does anyone have more information to work on? such as what this file is exactly?
If I read this in as a byte array and convert it to a string, I get a load of HEX (e.g. AA-BB-06 etc) but I can't convert this to anything useful no matter the encoding I use.
This documentation suggests it is in PKCS #8 standard.
I tried (suggested by #gtrig) the command:
openssl rsa -in pkcs8privatekey.der -inform der -out privatekey.pem
but this gives me the following:
unable to load Private Key
32096:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1306:
32096:error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:830:
32096:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:749:Field=n, Type=RSA
32096:error:0D09A00D:asn1 encoding routines:d2i_PrivateKey:ASN1 lib:d2i_pr.c:99:
I also get similar errors with NET and PEM -inform args.
and:
openssl asn1parse -in private.key
gives me the error:
"Error: offset too large"
I've just found that if I convert it to a base 64 string
Dim ba As Byte() = IO.File.ReadAllBytes("C:\private.key")
Dim toString1 As String = System.Convert.ToBase64String(ba)
which gives me a string which starts MIICdgIBADANB and is 924 characters long.
trying the following command gives me
openssl rsa -in private.key -text -noout
unable to load Private Key
17978:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:647:Expec ting: ANY PRIVATE KEY
Any further suggestions?
It's probably not a "hash" of the private key. It's most likely the private key in PKCS#8 format.
You can use the openssl command line tool to create a PKCS#12 keystore that should then be able to be used to construct an X509Certificate2 object.
First you will likely have to convert your private key from DER to PEM format, which can also be done in openssl:
openssl rsa -in pkcs8privatekey.der -inform der -out privatekey.pem
Then create the PKCS#12 keystore with:
openssl pkcs12 -export -name myalias -in mycert.crt -inkey privatekey.pem -out keystore.p12
Finally, you should be able to import this into X509Certificate2 object:
X509Certificate2 cert = X509Certificate2("C:\Path\keystore.p12", "password");
you can use the key tool UI. You need to know the type of the certificate they gave you , typically either a JKS key of PEM.
The following commands turn this into a format usable in windows:
Convert the private key from pkcs8/DER to a PEM file format
openssl pkcs8 -nocrypt -in dealerPrivate.key -inform der -outform pem -out private.pem
Convert the certificate from x509/DER to a PEM file format
openssl x509 -inform der -in dealerCertificate.x509 -out public.pem
Merge the two files into a pkcs12 file – you will be prompted for password to protect the p12 with
openssl pkcs12 -export -inkey private.pem -in public.pem -out mycert.p12
pkcs12 can be used directly in windows.
Customer has created key and certificate using openssl command below
openssl req -newkey rsa:1024 -sha1 -keyout OCkey.pem -out OCreq.pem -subj "/C=country/L=city/O=OCserver/OU=myLab/CN=OCserverName/" -config req.conf
openssl ca -in OCreq.pem -cert CAcert.pem -keyfile CAkey.pem -out OCcert.pem -days 3650 -config sign.conf -extfile sign.ext -md sha1 -notext
So they have given me the certificate(OCcert.pem) and private key (OCkey.pem).
How to ensure that OCcert.pem is created through OCkey.pem using Java? Note :I can't ask customer to change the ssl command.
Please help.
they have given me the certificate(OCcert.pem) and private key
(OCkey.pem)
Why? They should throw that private key away immediately, it is compromised. What was the purpose of this exercise?