Develop Reference

How can I rerun an Apache Flink Postgres JDBC job without getting "No suitable driver found" exception

I have a Flink job derived from the starter Maven project. That job has a source that opens a Postgres JDBC connection. I am executing the job on my own Flink session cluster using the example docker-compose.yml.
When I submit the job for the first time it executes successfully. When I try to submit it again I get the following error:
Caused by: java.sql.SQLException: No suitable driver found for jdbc:postgresql://host.docker.internal:5432/postgres?user=postgres&password=mypassword
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at com.myorg.project.JdbcPollingSource.run(JdbcPollingSource.java:25)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)
I have to restart my cluster in order to rerun my job. Why is this happening? How can I submit my job again without having to restart the cluster?
The only addition to the Maven starter project is:
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.2.24</version>
</dependency>
The Flink source does nothing but open a JDBC connection and is as follows:
package com.mycompany;
import org.apache.flink.streaming.api.functions.source.RichSourceFunction;
import java.sql.Connection;
import java.sql.DriverManager;
public class JdbcSource extends RichSourceFunction<Integer> {
private final String connString;
public JdbcSource(String connString) {
this.connString = connString;
}
#Override
public void run(SourceContext<Integer> ctx) throws Exception {
try (Connection conn = DriverManager.getConnection(this.connString)) {
}
}
#Override
public void cancel() {
}
}
I have tested this on Flink version 1.14.0 and 1.13.2 with the same results.
Note that this question provides a solution of using Class.forName("org.postgresql.Driver"); within my RichSourceFunction. However I would like to know what is going on.

The first question you can refer JDBC driver cannot be found when reading a DataSet from an SQL database in Apache Flink.
Second, if you use session mode. It can be easy to rerun the Flink job without restart the cluster. you can log in job manager shell then use the command rerun job.
Class.forName("org.postgresql.Driver"); will trigger static method block, so you DriverManager can get driver class. see:
// from org.postgresql.Driver
static {
try {
register();
} catch (SQLException var1) {
throw new ExceptionInInitializerError(var1);
}
}

I have this pom.xml dependency for Postgres for Apache Flink 1.13:
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>9.4-1201-jdbc41</version>
</dependency>
you can have a Postgres connector class for example:
public class PostgreSQLConnector {
private static volatile PostgreSQLConnector instance;
private Connection connectionDB = null;
public PostgreSQLConnector(your params) {
...
}
public static PostgreSQLConnector getInstance() {
PostgreSQLConnector postgreSQLConnector = instance;
if (postgreSQLConnector != null)
return postgreSQLConnector;
synchronized (PostgreSQLConnector.class) {
if (instance == null) {
instance = new PostgreSQLConnector(your params);
}
return instance;
}
}
public Connection getConnectionDB() throws SQLException {
if (checkNullConnection()) CreateConnection();
return connectionDB;
}
public void CheckConnection() throws SQLException {
if (checkNullConnection()) CreateConnection();
}
public void CreateConnection() throws SQLException {
try {
Class.forName(sink.driverName);
connectionDB = DriverManager.getConnection(fullUrl, username, password);
} catch (Exception e) {
...
}
}
public boolean checkNullConnection() throws SQLException {
return (connectionDB == null || connectionDB.isClosed());
}
}
then you can create a RichSourceFunction and create the connection in the overrides open method, not in the run
public class JdbcSource extends RichSourceFunction<Integer> {
private final String connString;
private static Connection dbConnection;
private static final PostgreSQLConnector postgreSQLConnector = PostgreSQLConnector.getInstance();
public JdbcSource(String connString) {
this.connString = connString;
}
#Override
public void open(Configuration parameters) throws SQLException {
dbConnection = postgreSQLConnector.getConnectionDB();
}
#Override
public void close() throws Exception {
if (dbConnection != null) dbConnection.close();
}
#Override
public void run(SourceContext<Integer> ctx) throws Exception {
do something here with the connection
}
#Override
public void cancel() {
}
}
Something like that you could maybe try and it should work

According to the official documentation of PostgreSQL JDBC driver, if you are using Java 1.6+, you can just put the driver's jar file into the classpath. The driver will be loaded by the JVM automatically. So the question is how to place the driver's jar file into the classpath.
Since you are using docker to deploy a session cluster, there's two way that may works:
Put the driver's jar file into docker image
Run and access the image with the command:
docker docker run -it -v $PWD:/tmp/flink <address to image> -- bash
Copy the driver's jar file into the folder /opt/flink/lib.
Create a new image from the container. Since /opt/flink/lib is loaded as classpath by default, now the driver's jar file is located at the classpath.
Package the driver's jar into your user jar
Add maven-assembly-plugin to the pom.xml of your maven project. Recompile your project and get a jar file with dependencies. In this jar, the PostgreSQL JDBC driver is packaged together.

How to load a JDBC driver dynamically during runtime since Java 9?

I'm currently migrating my Java 8 code to Java 11 and stumbled across a problem. I'm looking for jar files in a directory and add them to the classpath in order to use them as JDBC drivers.
After doing so I can easily use DriverManager.getConnection(jdbcString); to get a connection to any database I loaded a driver beforehand.
I used to load drivers using this bit of code which no longer works since the SystemClassLoader is no longer a URLClassLoader.
Method method = URLClassLoader.class.getDeclaredMethod("addURL", new Class[] { URL.class });
method.setAccessible(true);
method.invoke(ClassLoader.getSystemClassLoader(), new Object[] { jdbcDriver.toURI().toURL() });
So after looking around for alternatives I found this answer on SO:
https://stackoverflow.com/a/14479658/10511969
Unfortunately for this approach I'd need the drivers class name, i.e. "org.postgresql.Driver" which I don't know.
Is there just no way to do this anymore, or am I missing something?

Using a Shim is a good way to load the JDBC driver when the driver is, for some reason, not accessibile via the system class loader context. I have ran into this a few times with multi-threaded scripts that have their own separated classpath context.
http://www.kfu.com/~nsayer/Java/dyn-jdbc.html

Not knowing the driver's class seems like an odd constraint.
I would go for a custom class loader that after ever class initialisation (I think you can do that), calls DriverManager.getDrivers and registers any new drivers it finds. (I have no time at the moment to write the code.)
The hacky alternative would be to load all your code (except a bootstrap) in a URLClassLoader and addURL to that.
Edit: So I wrote some code.
It creates a class loader for the drivers that also contains a "scout" class that forwards DriverManager.drivers (which is a naughty caller sensitive method (a newish one!)). A fake driver within the application class loader forwards connect attempts onto any dynamically loaded drivers at the time of request.
I don't have any JDBC 4.0 or later drivers conveniently around to test this on. You'll probably want to change the URL - you'll need the Scout class and the driver jar.
import java.lang.reflect.*;
import java.net.*;
import java.sql.*;
import java.util.*;
import java.util.logging.*;
import java.util.stream.*;
class FakeJDBCDriver {
public static void main(String[] args) throws Exception {
URLClassLoader loader = URLClassLoader.newInstance(
new URL[] { new java.io.File("dynamic").toURI().toURL() },
FakeJDBCDriver.class.getClassLoader()
);
Class<?> scout = loader.loadClass("Scout");
Method driversMethod = scout.getMethod("drivers");
DriverManager.registerDriver(new Driver() {
public int getMajorVersion() {
return 0;
}
public int getMinorVersion() {
return 0;
}
public Logger getParentLogger() throws SQLFeatureNotSupportedException {
throw new SQLFeatureNotSupportedException();
}
public DriverPropertyInfo[] getPropertyInfo(String url, Properties info) {
return new DriverPropertyInfo[] { };
}
public boolean jdbcCompliant() {
return false;
}
public boolean acceptsURL(String url) throws SQLException {
if (url == null) {
throw new SQLException();
}
for (Iterator<Driver> iter=drivers(); iter.hasNext(); ) {
Driver driver = iter.next();
if (
driver.getClass().getClassLoader() == loader &&
driver.acceptsURL(url)
) {
return true;
}
}
return false;
}
public Connection connect(String url, Properties info) throws SQLException {
if (url == null) {
throw new SQLException();
}
for (Iterator<Driver> iter=drivers(); iter.hasNext(); ) {
Driver driver = iter.next();
if (
driver.getClass().getClassLoader() == loader &&
driver.acceptsURL(url)
) {
Connection connection = driver.connect(url, info);
if (connection != null) {
return connection;
}
}
}
return null;
}
private Iterator<Driver> drivers() {
try {
return ((Stream<Driver>)driversMethod.invoke(null)).iterator();
} catch (IllegalAccessException exc) {
throw new Error(exc);
} catch (InvocationTargetException exc) {
Throwable cause = exc.getTargetException();
if (cause instanceof Error) {
throw (Error)cause;
} else if (cause instanceof RuntimeException) {
throw (RuntimeException)cause;
} else {
throw new Error(exc);
}
}
}
});
// This the driver I'm trying to access, but isn't even in a jar.
Class.forName("MyDriver", true, loader);
// Just some nonsense to smoke test.
System.err.println(DriverManager.drivers().collect(Collectors.toList()));
System.err.println(DriverManager.getConnection("jdbc:mydriver"));
}
}
Within a directory dynamic (relative to current working directory):
import java.sql.*;
public interface Scout {
public static java.util.stream.Stream<Driver> drivers() {
return DriverManager.drivers();
}
}
I would always suggest avoiding setting the thread context class loader to anything other than a loader that denies everything, or perhaps null.
Modules may well allow you to load drivers cleanly, but I've not looked.

if you don`t know the driver name, you cannot use reflect to use urlLoader to load jar, which you exactly want.
I have same problem with dynamically load driver, because of jars are conflict.
Even though, I have to know the driver name to jar, which i want to load use my url class loader.
DriverManager use class loader to load jar, so it could find jdbc driver by name. As usual we use: class.forName。
We use self defined class loader to load our driver, so that it can solve the conflict of jars.

Accessing AWS RDS using IAM Authentication and Spring JDBC (DataSource and JdbcTemplace)

I am not able to figure out how to implement this. Any help and/or pointers will be greatly appreciated.
Currently, my Java/Spring application backend is deployed on EC2 and accessing MySQL on RDS successfully using the regular Spring JDBC setup. That is, storing database info in application.properties and configuring DataSource and JdbcTemplate in #Configuration class. Everything works fine.
Now, I need to access MySQL on RDS securely. RDS instance has IAM Authentication enabled. I have also successfully created IAM role and applied inline policy. Then, following the AWS RDS documentation and Java example on this link, I am able to access the database from a standalone Java class successfully using Authentication Token and the user I created instead of regular db username and password. This standalone Java class is dealing with "Connection" object directly.
The place I am stuck is how I translate this to Spring JDBC configuration. That is, setting up DataSource and JdbcTemplate beans for this in my #Configuration class.
What would be a correct/right approach to implement this?
----- EDIT - Start -----
I am trying to implement this as a library that can be used for multiple projects. That is, it will be used as a JAR and declared as a dependency in a project's POM file. This library is going to include configurable AWS Services like this RDS access using general DB username and password, RDS access using IAM Authentication, KMS (CMK/data keys) for data encryption, etc.
Idea is to use this library on any web/app server depending on the project.
Hope this clarifies my need more.
----- EDIT - End -----
DataSource internally has getConnection() so I can basically create my own DataSource implementation to achieve what I want. But is this a good approach?
Something like:
public class MyDataSource implements DataSource {
#Override
public Connection getConnection() throws SQLException {
Connection conn = null;
// get a connection using IAM Authentication Token for accessing AWS RDS, etc. as in the AWS docs
return conn;
}
#Override
public Connection getConnection(String username, String password) throws SQLException {
return getConnection();
}
//other methods
}

You can use the following snippet as a replacement for the default connection-pool provided by SpringBoot/Tomcat. It will refresh the token password every 10 minutes, since the token is valid for 15 minutes. Also, it assumes the region can be extracted from the DNS hostname. If this is not the case, you'll need to specify the region to use.
public class RdsIamAuthDataSource extends org.apache.tomcat.jdbc.pool.DataSource {
private static final Logger LOG = LoggerFactory.getLogger(RdsIamAuthDataSource.class);
/**
* The Java KeyStore (JKS) file that contains the Amazon root CAs
*/
public static final String RDS_CACERTS = "/rds-cacerts";
/**
* Password for the ca-certs file.
*/
public static final String PASSWORD = "changeit";
public static final int DEFAULT_PORT = 3306;
#Override
public ConnectionPool createPool() throws SQLException {
return pool != null ? pool : createPoolImpl();
}
protected synchronized ConnectionPool createPoolImpl() throws SQLException {
return pool = new RdsIamAuthConnectionPool(poolProperties);
}
public static class RdsIamAuthConnectionPool extends ConnectionPool implements Runnable {
private RdsIamAuthTokenGenerator rdsIamAuthTokenGenerator;
private String host;
private String region;
private int port;
private String username;
private Thread tokenThread;
public RdsIamAuthConnectionPool(PoolConfiguration prop) throws SQLException {
super(prop);
}
#Override
protected void init(PoolConfiguration prop) throws SQLException {
try {
URI uri = new URI(prop.getUrl().substring(5));
this.host = uri.getHost();
this.port = uri.getPort();
if (this.port < 0) {
this.port = DEFAULT_PORT;
}
this.region = StringUtils.split(this.host,'.')[2]; // extract region from rds hostname
this.username = prop.getUsername();
this.rdsIamAuthTokenGenerator = RdsIamAuthTokenGenerator.builder().credentials(new DefaultAWSCredentialsProviderChain()).region(this.region).build();
updatePassword(prop);
final Properties props = prop.getDbProperties();
props.setProperty("useSSL","true");
props.setProperty("requireSSL","true");
props.setProperty("trustCertificateKeyStoreUrl",getClass().getResource(RDS_CACERTS).toString());
props.setProperty("trustCertificateKeyStorePassword", PASSWORD);
super.init(prop);
this.tokenThread = new Thread(this, "RdsIamAuthDataSourceTokenThread");
this.tokenThread.setDaemon(true);
this.tokenThread.start();
} catch (URISyntaxException e) {
throw new RuntimeException(e.getMessage());
}
}
#Override
public void run() {
try {
while (this.tokenThread != null) {
Thread.sleep(10 * 60 * 1000); // wait for 10 minutes, then recreate the token
updatePassword(getPoolProperties());
}
} catch (InterruptedException e) {
LOG.debug("Background token thread interrupted");
}
}
#Override
protected void close(boolean force) {
super.close(force);
Thread t = tokenThread;
tokenThread = null;
if (t != null) {
t.interrupt();
}
}
private void updatePassword(PoolConfiguration props) {
String token = rdsIamAuthTokenGenerator.getAuthToken(GetIamAuthTokenRequest.builder().hostname(host).port(port).userName(this.username).build());
LOG.debug("Updated IAM token for connection pool");
props.setPassword(token);
}
}
}
Please note that you'll need to import Amazon's root/intermediate certificates to establish a trusted connection. The example code above assumes that the certificates have been imported into a file called 'rds-cacert' and is available on the classpath. Alternatively, you can also import them into the JVM 'cacerts' file.
To use this data-source, you can use the following properties for Spring:
datasource:
url: jdbc:mysql://dbhost.xyz123abc.us-east-1.rds.amazonaws.com/dbname
username: iam_app_user
driver-class-name: com.mysql.cj.jdbc.Driver
type: com.mydomain.jdbc.RdsIamAuthDataSource
Using Spring Java config:
#Bean public DataSource dataSource() {
PoolConfiguration props = new PoolProperties();
props.setUrl("jdbc:mysql://dbname.abc123xyz.us-east-1.rds.amazonaws.com/dbschema");
props.setUsername("iam_dbuser_app");
props.setDriverClassName("com.mysql.jdbc.Driver");
return new RdsIamAuthDataSource(props);
}
UPDATE: When using MySQL, you can also decide to use the MariaDB JDBC driver, which has builtin support for IAM authentication:
spring:
datasource:
host: dbhost.cluster-xxx.eu-west-1.rds.amazonaws.com
url: jdbc:mariadb:aurora//${spring.datasource.host}/db?user=xxx&credentialType=AWS-IAM&useSsl&serverSslCert=classpath:rds-combined-ca-bundle.pem
type: org.mariadb.jdbc.MariaDbPoolDataSource
The above requires MariaDB and AWS SDK libraries, and needs the CA-bundle in the classpath

I know this is an older question, but after a some searching I found a pretty easy way you can now do this using the MariaDB driver. In version 2.5 they added an AWS IAM credential plugin to the driver. It will handle generating, caching and refreshing the token automatically.
I've tested using Spring Boot 2.3 with the default HikariCP connection pool and it is working fine for me with these settings:
spring.datasource.url=jdbc:mariadb://host/db?credentialType=AWS-IAM&useSsl&serverSslCert=classpath:rds-combined-ca-bundle.pem
spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
spring.datasource.username=iam_username
#spring.datasource.password=dont-need-this
spring.datasource.hikari.maxLifetime=600000
Download rds-combined-ca-bundle.pem and put it in src/main/resources so you can connect via SSL.
You will need these dependencies on the classpath as well:
runtime 'org.mariadb.jdbc:mariadb-java-client'
runtime 'com.amazonaws:aws-java-sdk-rds:1.11.880'
The driver uses the standard DefaultAWSCredentialsProviderChain so make sure you have credentials with policy allowing IAM DB access available wherever you are running your app.
Hope this helps someone else - most examples I found online involved custom code, background threads, etc - but using the new driver feature is much easier!

There is a library that can make this easy. Effectively you just override the getPassword() method in the HikariDataSource. You use STS to assume the role and send a "password" for that role.
<dependency>
<groupId>io.volcanolabs</groupId>
<artifactId>rds-iam-hikari-datasource</artifactId>
<version>1.0.4</version>
</dependency>

Loading jars at runtime

I am trying to add jar file to classpath at runtime. I use this code
public static void addURL(URL u) throws IOException {
URLClassLoader sysloader = (URLClassLoader) ClassLoader
.getSystemClassLoader();
Class<URLClassLoader> sysclass = URLClassLoader.class;
try {
Method method = sysclass.getDeclaredMethod("addURL", parameters);
method.setAccessible(true);
method.invoke(sysloader, new Object[] { u });
System.out.println(u);
} catch (Throwable t) {
t.printStackTrace();
throw new IOException("Error");
}
}
System out prints this url:
file:/B:/Java/Tools/mysql-connector-java-5.1.18/mysql-connector-java-5.1.18/mysql-connector-java-5.1.18-bin.jar
I was check this path carefully, this jar exist. Even this test show that com.mysql.jdbc.
Driver class exists.
javap -classpath "B:\Java\Tools\mysql-connector-java-5.1.18\
mysql-connector-java-5.1.18\mysql-connector-java-5.1.18-bin.jar" com.mysql.jdbc.
Driver
Compiled from "Driver.java"
public class com.mysql.jdbc.Driver extends com.mysql.jdbc.NonRegisteringDriver i
mplements java.sql.Driver{
public com.mysql.jdbc.Driver() throws java.sql.SQLException;
static {};
}
But I still get java.lang.ClassNotFoundException when I use this Class.forName(driver).
What is wrong with this code?

The URL is ok, nevertheless you try to load a jar from classpath, so it means that yo need to have the file in cp first.
In your case you want to load a jar that is not in classpath so you have to use
URLClassLoader and for JAR you can use also the JARClassLoader
If you want some sample lesson on it:
http://docs.oracle.com/javase/tutorial/deployment/jar/jarclassloader.html
Here a sample I ran by myself see if helps you. It search the Logger class of Log4j that is not in my classpath, of course i got exception on invocation of the constructor since i did not pass the right params to the constructor
package org.stackoverflow;
import java.io.File;
import java.net.URL;
import java.net.URLClassLoader;
public class URLClassLoaderSample
{
public static void main(String[] args) throws Exception
{
File f = new File("C:\\_programs\\apache\\log4j\\v1.1.16\\log4j-1.2.16.jar");
URLClassLoader urlCl = new URLClassLoader(new URL[] { f.toURL()},System.class.getClassLoader());
Class log4jClass = urlCl.loadClass("org.apache.log4j.Logger");
log4jClass.newInstance();
}
}
Exception in thread "main" java.lang.InstantiationException: org.apache.log4j.Logger
at java.lang.Class.newInstance0(Class.java:357)
at java.lang.Class.newInstance(Class.java:325)
at org.stackoverflow.URLClassLoaderSample.main(URLClassLoaderSample.java:19)
Exception due to the wrong invocation, nevertheless at this stage we already found the class

Ok try the alternative approach with DataSource and not directly the Driver
Below is the code (working with oracle driver, i don't have my sql db, but the properties are the same)
Generally using the DataSource interface is the preferred approach since JDBC 2.0
The DataSource jar was not in the classpath neither for the test below
public static void urlCLSample2() throws Exception
{
File f = new File("C:\\_programs\\jdbc_drivers\\oracle\\v11.2\\ojdbc6.jar");
URLClassLoader urlCl = new URLClassLoader(new URL[] { f.toURL() }, System.class.getClassLoader());
// replace the data source class with MySQL data source class.
Class dsClass = urlCl.loadClass("oracle.jdbc.pool.OracleDataSource");
DataSource ds = (DataSource) dsClass.newInstance();
invokeProperty(dsClass, ds, "setServerName", String.class, "<put your server here>");
invokeProperty(dsClass, ds, "setDatabaseName", String.class, "<put your db instance here>");
invokeProperty(dsClass, ds, "setPortNumber", int.class, <put your port here>);
invokeProperty(dsClass, ds, "setDriverType",String.class, "thin");
ds.getConnection("<put your username here>", "<put your username password here>");
System.out.println("Got Connection");
}
// Helper method to invoke properties
private static void invokeProperty(Class dsClass, DataSource ds, String propertyName, Class paramClass,
Object paramValue) throws Exception
{
try
{
Method method = dsClass.getDeclaredMethod(propertyName, paramClass);
method.setAccessible(true);
method.invoke(ds, paramValue);
}
catch (Exception e)
{
throw new Exception("Failed to invoke method");
}
}

Loading JDBC Driver at Runtime

I'm using the following code to load a driver class:
public class DriverLoader extends URLClassLoader {
private DriverLoader(URL[] urls) {
super(urls);
File driverFolder = new File("driver");
File[] files = driverFolder.listFiles();
for (File file : files) {
try {
addURL(file.toURI().toURL());
} catch (MalformedURLException e) {
}
}
}
private static DriverLoader driverLoader;
public static void load(String driverClassName) throws ClassNotFoundException {
try {
Class.forName(driverClassName);
} catch (ClassNotFoundException ex) {
if (driverLoader == null) {
URL urls[] = {};
driverLoader = new DriverLoader(urls);
}
driverLoader.loadClass(driverClassName);
}
}
}
Although the class loads fine I can't establish a Database connection (No suitable driver found for ...) no matter which driver I try.
I assume this is because I'm not loading the driver class using Class.forName (which wouldn't work since I'm using my own ClassLoader). How can I fix this?

You need to create an instance of the driver class before you can connect:
Class drvClass = driverLoader.loadClass(driverClassName);
Driver driver = drvClass.newInstance();
Once you have the instance you can either use that instance to connect:
Properties props = new Properties();
props.put("user", "your_db_username");
props.put("password", "your_db_password");
Connection con = driver.connect("jdbc:postgresql:...", props);
As an alternative, if you want to keep using DriverManager you must register the driver with the DriverManager manually:
DriverManager.registerDriver(driver);
Then you should be able to use the DriverManager to establis a connection.
If I recall it correctly there was a problem with the DriverManager refusing to connect if the driver itself was not loaded by the same classloader as the DriverManager. If that (still) is the case, you need to use Driver.connect() directly.

You should establish connection in a class loaded by your DriverLoader. So, load the connection establishment code using DriverLoader and then call JDBC from it.

You need to add a Classpath reference in the manifest. Follow these simple steps:
add a folder "lib" to your application
place "mysql-connector-java-5.1.18-bin" in lib
now open your "MANIFEST.MF" and go to tab "RUNTIME"
on bottom right, you would see "classpath" ; click "Add"
now add the folder lib [created in step 1] along with the jar file
in this way, whenever a runtime EclipseApplication /OSGi Application is started this jar file is exported along too. So the connectivity would then be available there too.