Nifi and Postgresql with certificate: how to set client certificate and key? - java

I have a NIFI image running in openshift and a postgres in the cloud "owned" by another department.
They sent to us a certificate(".crt") and a key(".key"), since the log-in is made trough client certificate, instead of username and password. I have succeeded to log-in in pgAdmin 4, but not to connect the NIFI to the Postgres with certificate and key.
I have uploaded the certificate and the key to the image (using a secret and mounting it) so if I go to the pod terminal I can access it.
But, when I pass to a DBCPConnectionPoll service the connection string bellow and activates an ExecuteSQL processor, I receive an exception that the certificate is not valid, as follows:
ERROR
ExecuteSQL[id=...] Unable to execute SQL query <...>;
due to java.sql.SQLException: Cannot create a PoolableConnectionFactory (FATAL: connection requires a valid client certificate).
No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Cannot create a PoolableConnectionFactory (FATAL: connection requires a valid client certificate)
I have tried to pass the certificate in two ways to the DBCPConnectionPoll service:
1) as parameters in the connection string ("database connection url" property:
jdbc:postgresql://<ip>:<port>/<username>?user=<username>&sslTrue&sslcert=/etc/.../mycerts/mycert.der&sslkey=/etc/.../mycerts/mykey.key.pk8
2) adding properties in the service (+ button and the just parameter name and the path as the value) and just passing this as url:
jdbc:postgresql://<ip>:<port>/<username>
Both seems to work generally speaking, since I can connect to another postgres I have which not requires ssl certification.
Some considerations:
1) My assumption here is that the connection string in the NIFI does not know to read properly the file path for the certificate and key.
2) I have converted the certificates a bunch of times to different types that java can receive in order to see if that was the problem, but I still receive the same exception. So it seems that the connection pool just does not "achieve" the files at all. Nevertheless, if some one has a say in this topic, it can be handy, after the main problem is solved. So appreciate some tips here as well.
3) I have also read the NIFI source code and it seems that NIFI uses normally JDBC classes to create the connection pool, so a connection string as I passed would have worked in java code, but somehow doesn't work in NIFI (which is written in java).
4) The jdbc driver and everything else is configured properly, since I can work with a non-secure postgres in NIFI.
Thank you very much.

A co-worker found the 'simple solution' to the problem I asked and I would like to share so it can help others.
What was missing was the property sslmode = require. After including that, the service worked perfectly. Actually, I am not sure why it didn't work with sslmode = prefer, since it is what my pgadmin is using for the same database and there it works perfectly. It seems like we must 'force' nifi to use ssl in this case - see documentation here: https://jdbc.postgresql.org/documentation/head/ssl-client.html.
Moreover, some insights:
It worked with certificate in '.der' and key in '.pk8' formats (didn't have to use trust-store and key-store as needed in other services).
One can add in the 'plus' button the properties and give them the right name as we would do in java code, instead of concatenating every property in the connection string (see second option in the question above).
Make it helps others as well.

Related

How do I fix this SSL error when making a SQL request from a jetty server

I'm in the process of upgrading a java application. Originally, the application was built with jdk 8 and the server was jetty 9. Since upgrading to jetty 10 and jdk 11, I'm running into an issue when trying to make requests to our sql datasource. When the application attempts to query the database, it fails with the error:
java.sql.SQLException: Cannot create PoolableConnectionFactory
(The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target". ClientConnectionId:redacted)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:653)
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:531)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:731)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator$ConnectionProviderJdbcConnectionAccess.obtainConnection(JdbcEnvironmentInitiator.java:181)
I understand that either the jetty server or the sql server are missing a security certificate, but I'm not sure what to do about it. I read through the documentation here:
https://www.eclipse.org/jetty/documentation/jetty-10/operations-guide/index.html#og-keystore but the sql server is a remote server that I don't have admin access to, so I'm not sure I can even do what they describe. Do I need to get the certificate and key from the sql server somehow? What am I missing here?
Your program is failing because it is trying to connect to an "unsecure" location, you have to connect to the destination manually, get the certificate/or certificates, install them into your local keystore, and restart your program.
Some instructions on how to do that can be gound here.
https://www.thesslstore.com/knowledgebase/ssl-install/jetty-java-http-servlet-webserver-ssl-installation/
If you connect to multiple locations, yes, you need to have an entry for each, unless that they have a wildcard certificate (a certificate that applies for all of them).
Alternatively! (not sure but handy) you can start the connection without enforcing certificate validation, in this case the connection will happen no matter is the destination is actually false, this is a security issue, but there are cases when this is needed, in fact, this happens a lot when you run balancers where they validate the certs for you and you connect in http to them, automatically the validation is getting dropped since most of those balancers won't fail to connect even if the validation fails, but this is a different topic!
Cheers.

Is it possible to force SSLHandshake to always use the hostname, not IP for HttpsUrlConnection

So I have this situation: I try to download an image from somedomain.com using HTTPS. The domain is probably misconfigured, but unfortunately I can't change that. What exactly is happening:
When I browse to https://somedomain.com/animage.jpg I get a valid certificate issued for somedomain.com, which is perfect. But when I call the same site using it's IP address, say https://123.123.123.123 - I get a (also valid) certificate for *.hostingcompany.com - the certificate of the hosting company.
Now, I try to download the contents of the file using Java's HttpsUrlConnection, nothing special:
var urlConnection = new URL(imageUrl).openConnection();
((HttpURLConnection) urlConnection).getResponseCode();
(I want to first check the response code, but it's not important here.)
This code runs inside a Spring Boot App and is run on request. It works fine for the first request since booting the app. Each subsequent request fails with java.security.cert.CertificateException: No subject alternative DNS name matching somedomain.com found. It's because on each subsequent request the SSL Handshake is sent to the IP, not hostname, and get's the hosting company's certificate.
I was trying to find different settings for the SSL classes, but to no avail. I know there is a workaround where I could supply my own HostnameVerifier which could just return true, but that won't be secure, so I don't want to do that.
Did anyone encounter such problem? Maybe I'm searching in the wrong places? Maybe it's something with DNSes? I will appreciate any help.
Turns out it is a bug in Java 11.01. It is fixed since 11.02. After switching to 11.03. the behaviour I described above is gone. Each request gets a proper certificate.
Here are the details of the bug: https://bugs.openjdk.java.net/browse/JDK-8211806

How to connect to Sybase ASE using JDBC driver and SSL connection

I'm trying to establish an SSL connection to a Sybase ASE 15.7 using JDBC driver with no luck.
I tried the following options:
Using JTDS 1.25 driver (jtds-1.2.5.jar)
With the following connection string: jdbc:jtds:sybase://host:port;databaseName=dbname;ssl=request
I got Network error IOException: Connection refused
Using Jconnect 4 (jconn4.jar)
with the following connection string:
jdbc:sybase:Tds:host:port/dbname?ENABLE_SSL=true
I got java.sql.SQLException: JZ00L: Login failed. Examine the SQLWarnings chained to this exception for the reason(s)
...
java.sql.SQLException: I/O Error: DB server closed connection.
I checked the Sybase log see the following error:
kernel SSL or Crypto Error Message: 'The SSL handshake failed. Root error: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol'.
it looks like the Sybase server expects SSL connection but the java client still not using SSL (although connection string property marks ssl=true).
tried searching Sybase documentation with not much luck, neither for this error nor for SSL with JConnect.
Any answer will be much appreciated - i'm flexible with the type of driver and the configuration.
Thanks
After much investigation, I found a solution. 2 actually.
Using trust all certificate JDBC connection string parameter:
if you don't mind to trust all certificates (do this only if you entirely trust the network you're working in, especially not anything going on the public internet), you may add a connection string indicating the SSLSocketFactory creating the connection to trust all certificates. The connection string will look as follows: jdbc:sybase:Tds:host:port/dbname?ENABLE_SSL=true&SSL_TRUST_ALL_CERTS=true
Using the sybase certificate:
the certificate needs to be imported to the java application trust store.
in case you're not working with a designated trust store, it may be imported to the Java default trust store found under $JAVA_HOME\jreX\lib\security\cacerts. The certificate may be imported using keytool as explained here.
Although zuckermanori's answer provides some key details, adding more steps below which are required -
Provide right jdbc jar which supports ssl while creating spark session. I was earlier using jconn3-6.0.0.jar which doesnt support ssl. Later, I used jconnect-16.0_SP02.jar which worked fine.
Example pyspark command to pass the driver jar would be -
pyspark --jars
/path/to/your/jdbc/driver/jar
Provide additional java args as follows in the same command to provide trust store location(which has the certificates) -
--conf spark.driver.extraJavaOptions="-Djavax.net.ssl.trustStore=/path/to/truststore -Djavax.net.ssl.trustStorePassword=your_truststore_password"​
​
3. Use right string to load driver in option. Earlier I was using 'com.sybase.jdbc4.jdbc.SybDriver' which didnt work. Then below worked for me -
.option("driver", "com.sybase.jdbc4.jdbc.SybDriver")
​
4. Use right connection string to provide additional ssl options as per the driver. E.g. -
.option("url", "jdbc:sybase:Tds:host_name:ssl_port/database_name?ENABLE_SSL=true&SSL_TRUST_ALL_CERTS=true&ssl=request")
Additional options required -
.option("ssl", True).option("sslmode", "require")
To sum it up, this is how your pyspark command should look like(if you are using yarn mode, then truststore should be accessible on all nodes. Below is the example of spark local mode) -
pyspark --jars /path/to/your/jdbc/driver/jar --conf spark.driver.extraJavaOptions="-Djavax.net.ssl.trustStore=/path/to/truststore -Djavax.net.ssl.trustStorePassword=your_truststore_password"​ --master local
And this is how your jdbc read look like -
spark.read.format("jdbc").option("url", "jdbc:sybase:Tds:host_name:ssl_port/database_name?ENABLE_SSL=true&SSL_TRUST_ALL_CERTS=true&ssl=request").option("driver", "com.sybase.jdbc4.jdbc.SybDriver").option("ssl", True).option("sslmode", "require").option("user", "your_user_name").option("password", "your_password").option("dbtable", "db.dbo.table_name").load().show(5)

Error connecting to Oracle database using JDBC [duplicate]

I am new to Oracle, and am trying to run a simple example code with Java, but am getting this error when executing the code.. I am able to start up the listener via CMD and am also able to run SQL Plus. Can anyone give me a hand and tell me what I might be doing wrong?
Update:
I am using JDBC.
Database is local, and I actually had it working but it stopped working just today. I'm not really sure why though. Would you mind giving me some procedures to follow by since I don't know much.
Either:
The database isn't running
You got the URL wrong
There is a firewall in the way.
(This strange error message is produced by Oracle's JDBC driver when it can't connect to the database server. 'Network adapter' appears to refer to some component of their code, which isn't very useful. Real network adapters (NICs) don't establish connections at all: TCP protocol stacks do that. It would have been a lot more useful if they had just let the original ConnectException be thrown, or at least used its error message and let it appear in the stack trace.)
I had the same problem, and this is how I fixed it.
I was using the wrong port for my connection.
private final String DB_URL = "jdbc:oracle:thin:#localhost:1521:orcll"; // 1521 my wrong port
go to your localhost
(my localhost address) : https://localhost:1158/em
login
user name
password
connect as --> normal
Below 'General' click on LISTENER_localhost
look at you port number
Net Address (ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1522))
Connect to port 1522
Edit you connection
change port 1521 to 1522.
done
Another thing you might want to check that the listener.ora file matches the way you are trying to connect to the DB. If you were connecting via a localhost reference and your listener.ora file got changed from:
HOST = localhost
to
HOST = 192.168.XX.XX
then this can cause the error that you had unless you update your hosts file to accommodate for this. Someone might have made this change to allow for remote connections to the DB from other machines.
I figured out that in my case, my database was in different subnet than the subnet from where i was trying to access the db.
I had this error when i renamed the pc in the windows-properties. The pc-name must be updated in the listener.ora-file
Most probably you have listener configured wrongly, the hostname you specify in connection string must be the same as in the listener.
First check the Firewall and network related issues.
Check if Oracle Listener service is available and running. If not you may use Oracle Net Configuration Assistant tool to add and register new listener.
If the above steps are ok then you need to configure Oracle Listener appropriately. You may use Oracle Net Manager tool or edit “%ORACLE_HOME%\network\admin\listener.ora” file manually.
There are 2 options that need to be considered carefully:
Listening Locations associated with the Listener – Hostname(IP) and Port in Listening Location must exactly match the ones used in the connection string.
For example, if you use 192.168.74.139 as target hostname, then there must be Listening Location registered with the same IP address.
Also make sure the you use the same SID as indicated in Database Service associated with the Listener.
https://adhoctuts.com/fix-oracle-io-error-the-network-adapter-could-not-establish-the-connection-error/
IO Error: The Network Adapter could not establish the connection (CONNECTION_ID=iKQM6lBbSLiArrYuDqud8A==)
if you are facing this issue
1- make sure you have downloaded oracle databases like oracle 11g,19c, 21c, or any latest databases.
2- search for services in your computer or type win+r then services.mis then search for oracleservice you will find orcl or xe or any other sid like oracleserviceorcl;
after that you can test your connection using sql developer, sql plus or cmd
To resolve the Network Adapter Error I had to remove the - in the name of the computer name.
In my case, I needed to specify a viahost and viauser. Worth trying if you're in a complex system. :)
For me the basic oracle only was not installed. Please ensure you have oracle installed and then try checking host and port.
I was having issues with this as well. I was using the jdbc connection string to connect to the database. The hostname was incorrectly configured in the string. I am using Mac, and the same string was being used on Windows machines without an issue. On my connection string, I had to make sure that I had the full url with the appending "organizationname.com" to the end of the hostname.
Hope this helps.
Just try to re-create connection. In my situation one of jdbc connection stopped working for no reason. From console sqlplus was working ok.
It took me 2 hours to realize that If i create the same connection - it works.

Java7 sqljdbc4 - SQL error 08S01 on getConnection()

I'm trying to write a really simple GUI app for inserting some records
into a database, and reading back some records (nothing fancy, just 1 table with 3 rows, no relations).
The source...
package EntryProg;
import java.sql.*;
import com.microsoft.sqlserver.jdbc.*;
public class CourseDataEntryHandler
{
private Connection connect;
private CallableStatement callState;
private ResultSet rSet;
private SQLServerDataSource dSource;
public CourseDataEntryHandler()
{
rSet = null;
callState = null;
dSource = new SQLServerDataSource();
dSource.setUser(REDACTED);
dSource.setPassword(REDACTED);
dSource.setServerName(REDACTED);
dSource.setPortNumber(REDACTED);
dSource.setDatabaseName(REDACTED);
dSource.setEncrypt(true);
dSource.setTrustServerCertificate(true);
try
{
Error here
connect = dSource.getConnection();
end error
}
catch (SQLServerException e)
{
//TODO Figure out how to handle -- logging for now, console
do
{
System.out.println(e.getErrorCode());
System.out.println(e.getMessage());
System.out.println(e.getSQLState());
e = (SQLServerException) e.getNextException();
} while (e != null);
System.out.println("END");
System.out.println();
}
}
I get the following error...
(code)0
(message)SQL Server did not return a response. The connection has been closed.
(state)08S01
I've verified that the user,pass,server name,port, and DB name are all accurate.
If I change the username to a non-valid one, I get a "could not log in" error reported back so I know I'm hitting the server.
I've not been able to fully connect once, so I know it's not a "too many connections" issue, as the only person currently logged into the server is me via sql management studio. It doesn't work when I log out of that either so definitely not a connections # issue.
The applications user has datareader/datawriter permissions as well.
(I'm using Eclipse, if that matters. And am referencing the sqljdbc4.jar library).
I'm at a loss as to where to go with troubleshooting this. Any help would be greatly appreciated.
EDIT
Update - I've also tried a connection string and using DriverManager.getConnection(connString) to set the connection, that didn't work either. The result is the same.
Also, SQL server 2008 r2 is the sql server version I'm using.
EDIT
I wrote a quick C# program to test the connection, sure enough the connection works fine in .net, unfortunately I have to use java for this project (it's a project I've chosen to do on my own for a class, only requirement is it be in Java...teacher has no clue what's going on either).
Comment the line with setEncrypt(true):
...
dSource.setDatabaseName(REDACTED);
//dSource.setEncrypt(true);
dSource.setTrustServerCertificate(true);
...
You might have trouble with the encryption setting. From the setEncrypt(...) documentation:
If the encrypt property is set to true, the Microsoft SQL Server JDBC Driver uses the JVM's default JSSE security provider to negotiate SSL encryption with SQL Server. The default security provider may not support all of the features required to negotiate SSL encryption successfully. For example, the default security provider may not support the size of the RSA public key used in the SQL Server SSL certificate. In this case, the default security provider might raise an error that will cause the JDBC driver to terminate the connection. In order to resolve this issue, do one of the following:
Configure the SQL Server with a server certificate that has a smaller RSA public key
Configure the JVM to use a different JSSE security provider in the "/lib/security/java.security" security properties file
Use a different JVM
Update
With Java versions 1.6.0_29 and 7.0.0_1 Oracle introduced a security fix for the SSL/TLS BEAST attack that very likely will cause the very same problem. The above security fix is known to make trouble for database connections to MSSQL Server with both the jTDS driver and the Microsoft driver. You can either
decide not to use encryption by not using setEncrypt(true) (as specified above)
or, if it is enforced by MSSQL Server, you could turn off the Java fix in your JVM by setting the -Djsse.enableCBCProtection=false system property. Be warned, it will affect all SSL connections within the same VM.
Sometimes, the engine configuration is modified in such a way it doesn't accept external connections. After some research, the following worked for me:
Open SQL Server Configuration Manager
Get to SQL SERVER Network Configuration
MSSQLSERVER Protocols (Double click)
View TCP/IP (must be enabled) (Open)
Go to tab "IP Addresses"
Type into "TCP Port" :1433
The enabled and activated options must be Enabled: Yes
Restart the service

Categories

Resources