Develop Reference

How can I rerun an Apache Flink Postgres JDBC job without getting "No suitable driver found" exception

I have a Flink job derived from the starter Maven project. That job has a source that opens a Postgres JDBC connection. I am executing the job on my own Flink session cluster using the example docker-compose.yml.
When I submit the job for the first time it executes successfully. When I try to submit it again I get the following error:
Caused by: java.sql.SQLException: No suitable driver found for jdbc:postgresql://host.docker.internal:5432/postgres?user=postgres&password=mypassword
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at com.myorg.project.JdbcPollingSource.run(JdbcPollingSource.java:25)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)
I have to restart my cluster in order to rerun my job. Why is this happening? How can I submit my job again without having to restart the cluster?
The only addition to the Maven starter project is:
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.2.24</version>
</dependency>
The Flink source does nothing but open a JDBC connection and is as follows:
package com.mycompany;
import org.apache.flink.streaming.api.functions.source.RichSourceFunction;
import java.sql.Connection;
import java.sql.DriverManager;
public class JdbcSource extends RichSourceFunction<Integer> {
private final String connString;
public JdbcSource(String connString) {
this.connString = connString;
}
#Override
public void run(SourceContext<Integer> ctx) throws Exception {
try (Connection conn = DriverManager.getConnection(this.connString)) {
}
}
#Override
public void cancel() {
}
}
I have tested this on Flink version 1.14.0 and 1.13.2 with the same results.
Note that this question provides a solution of using Class.forName("org.postgresql.Driver"); within my RichSourceFunction. However I would like to know what is going on.

The first question you can refer JDBC driver cannot be found when reading a DataSet from an SQL database in Apache Flink.
Second, if you use session mode. It can be easy to rerun the Flink job without restart the cluster. you can log in job manager shell then use the command rerun job.
Class.forName("org.postgresql.Driver"); will trigger static method block, so you DriverManager can get driver class. see:
// from org.postgresql.Driver
static {
try {
register();
} catch (SQLException var1) {
throw new ExceptionInInitializerError(var1);
}
}

I have this pom.xml dependency for Postgres for Apache Flink 1.13:
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>9.4-1201-jdbc41</version>
</dependency>
you can have a Postgres connector class for example:
public class PostgreSQLConnector {
private static volatile PostgreSQLConnector instance;
private Connection connectionDB = null;
public PostgreSQLConnector(your params) {
...
}
public static PostgreSQLConnector getInstance() {
PostgreSQLConnector postgreSQLConnector = instance;
if (postgreSQLConnector != null)
return postgreSQLConnector;
synchronized (PostgreSQLConnector.class) {
if (instance == null) {
instance = new PostgreSQLConnector(your params);
}
return instance;
}
}
public Connection getConnectionDB() throws SQLException {
if (checkNullConnection()) CreateConnection();
return connectionDB;
}
public void CheckConnection() throws SQLException {
if (checkNullConnection()) CreateConnection();
}
public void CreateConnection() throws SQLException {
try {
Class.forName(sink.driverName);
connectionDB = DriverManager.getConnection(fullUrl, username, password);
} catch (Exception e) {
...
}
}
public boolean checkNullConnection() throws SQLException {
return (connectionDB == null || connectionDB.isClosed());
}
}
then you can create a RichSourceFunction and create the connection in the overrides open method, not in the run
public class JdbcSource extends RichSourceFunction<Integer> {
private final String connString;
private static Connection dbConnection;
private static final PostgreSQLConnector postgreSQLConnector = PostgreSQLConnector.getInstance();
public JdbcSource(String connString) {
this.connString = connString;
}
#Override
public void open(Configuration parameters) throws SQLException {
dbConnection = postgreSQLConnector.getConnectionDB();
}
#Override
public void close() throws Exception {
if (dbConnection != null) dbConnection.close();
}
#Override
public void run(SourceContext<Integer> ctx) throws Exception {
do something here with the connection
}
#Override
public void cancel() {
}
}
Something like that you could maybe try and it should work

According to the official documentation of PostgreSQL JDBC driver, if you are using Java 1.6+, you can just put the driver's jar file into the classpath. The driver will be loaded by the JVM automatically. So the question is how to place the driver's jar file into the classpath.
Since you are using docker to deploy a session cluster, there's two way that may works:
Put the driver's jar file into docker image
Run and access the image with the command:
docker docker run -it -v $PWD:/tmp/flink <address to image> -- bash
Copy the driver's jar file into the folder /opt/flink/lib.
Create a new image from the container. Since /opt/flink/lib is loaded as classpath by default, now the driver's jar file is located at the classpath.
Package the driver's jar into your user jar
Add maven-assembly-plugin to the pom.xml of your maven project. Recompile your project and get a jar file with dependencies. In this jar, the PostgreSQL JDBC driver is packaged together.

SQLException: No suitable driver found for url=jdbc:derby

I am having problems establishing a connection to a local Apache Derby (Java DB) database for a small game which I've been working on. The code that throws the exception looks like this:
public class DatabaseController {
private static Connection conn;
private final String url = "url=jdbc:derby://localhost:1527/GameDB;create=true";
private final String username = "pdc";
private final String password = "123";
/**
* Connects to the database.
*/
public void initialize() {
try{
//Open a connection
conn = DriverManager.getConnection(url, username, password);
} catch(SQLException e){
//Handle errors
Logger.getLogger(DatabaseController.class.getName()).log(Level.SEVERE, null, e);
}
}
}
When I run the code I get the following exception:
Oct 08, 2015 2:27:40 PM pdc.project.Controller.DatabaseController initialize
SEVERE: null
java.sql.SQLException: No suitable driver found for url=jdbc:derby://localhost:1527/GameDB;create=true
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at pdc.project.Controller.DatabaseController.initialize(DatabaseController.java:30)
at pdc.project.Controller.Main.main(Main.java:35)
Exception in thread "main" java.lang.NullPointerException
at pdc.project.Controller.DatabaseController.createTable(DatabaseController.java:63)
at pdc.project.Controller.Main.main(Main.java:36)
I have tried the following:
Adding derbyclient.jar to libraries (and checking that it's in the classpath under Project > Properties > Libraries)
Adding Java DB Driver to libraries (and checking that it's in the classpath under Project > Properties > Libraries)
Using Class.forName("org.apache.derby.jdbc.ClientDriver") to register the driver
Using Class.forName("org.apache.derby.jdbc.EmbeddedDriver") to register the driver
Adding a dummy driver
As well as combinations of the above. None of it has worked and I think my hair is starting to fall out. Can anyone provide me the answer to why it won't register the driver?
Thank you!
I'm using Netbeans and JDK 1.8.

Your connection URL should not begin with the characters "url=". Instead of
private final String url = "url=jdbc:derby://localhost:1527/GameDB;create=true";
your URL should start with "jdbc:", e.g.,
private final String url = "jdbc:derby://localhost:1527/GameDB;create=true";

Connection issue in Hadoop MapReduce

In a Hadoop MapReduce program, I am trying to read JDBC connection details from connection.properties file, that is also in proper classpath.
Code:
public synchronized Connection getConnection() {
return getDetails(); // inside try catch block
}
public Connection getDetails() throws SQLException, IOException {
Properties props = new Properties();
FileInputStream in = new FileInputStream("connection.properties");
props.load(in);
in.close();
String drivers = props.getProperty("jdbc.drivers");
if (drivers != null) {
System.setProperty("jdbc.drivers",drivers);
}
String url = props.getProperty("jdbc.url");
String username = props.getProperty("jdbc.username");
String password = props.getProperty("jdbc.password");
return DriverManager.getConnection(url, username, password);
}
But when running my driver main program it gives an error:
java.io.FileNotFoundException: connection.properties (No such file or directory)
Any idea?

In order to run the MapReduce that uses properties file, you need to package it with your code, and instead of using the traditional reading the file streams from local system, you can read it from the jar you already built
reading properties from jar:
InputStream stream = this.getClass().getResourceAsStream("foo.properties"))

Class.forName("com.mysql.jdbc.Driver").newInstance()

I am having this error on Netbeans 7.2, it says that ClassNotFoundexception and InstantationException. I am really stuck on this matter. Kindly help me.
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
try {
String driver = "com.mysql.jdbc.Driver";
con = null;
String username = "";
String password = "";
Class.forName("com.mysql.jdbc.Driver").newInstance();
con = DriverManager.getConnection("jdbc:mysql://localhost:3306/dbName", "root", "password");
Statement st = con.createStatement();
ResultSet mar = st.executeQuery("SELECT * FROM table");
Gson gson = new GsonBuilder().create();
response.setContentType("application/json");
response.setCharacterEncoding("utf-8");
} catch (SQLException e) {
String message = e.getMessage();
}

What about this simple way?!
java.sql.Driver d=new com.mysql.jdbc.Driver();
I also wondered why do you connect to database with such this way?! It's better let server manage it.
First config the context.xml (if you are using tomcat) like this:
<context>
<Resource name="_ds" auth="Container" type="javax.sql.DataSource"
maxActive="128" maxIdle="32" username="_admin" password="qwerty" driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://127.0.0.1:3306/dbname"/>
</context>
Then, simple get a connection from this resource in servlet/etc, like this:
public void init() {
try {
_ds = (DataSource) InitialContext.lookup("java:/comp/env/_ds");
} catch (Exception ex) {
}
}
private javax.sql.DataSource _ds;
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{
try {
/*String driver = "com.mysql.jdbc.Driver";
con = null;
String username = "";
String password = "";
Class.forName("com.mysql.jdbc.Driver").newInstance();
con = DriverManager.getConnection("jdbc:mysql://localhost:3306/dbName", "root", "password");*/
Connection con=_ds.getConnection();
Statement st = con.createStatement();
ResultSet mar = st.executeQuery("SELECT * FROM table");
Gson gson = new GsonBuilder().create();
response.setContentType("application/json");
response.setCharacterEncoding("utf-8");
con.close();
} catch (SQLException e) {
String message = e.getMessage();
}
By the way, don't forget to compy the MySQL JDBC driver jar-file in <CATALINA_BASE>/lib folder.

All you need is Class.forName("com.mysql.jdbc.Driver")
This acts like class loader and load your driver class for you. For that you need to add the corresponding jar file(which has the driver implementation). So download and add mysql-connector.jar in your class path.
Note : If you are using Java 7 then there is no need to even add the Class.forName("com.mysql.jdbc.Driver") statement.Automatic Resource Management (ARM) is added in JDBC 4.1 which comes by default in Java 7.

You might want to solve a bigger problem. You ought not enter configuration data such as database connection information directly in your servlet.
Are you using Tomcat? You can simply use JNDI. You will be able to change database details and drivers without having to recompile your servlet.
Here is the Tomcat 7.0 JNDI Datasource HOW-TO shows various ways in which you can get a Connection to your database.
On that page, you have a code example of how to get a Connection (Oracle 8i, 9i & 10g -> Part 3), and how to write a MySQL specific configuration.
Make sure to download a correct MySQL jar and place it in your Tomcat's lib/ directory (or alternatively your WAR's WEB-INF/lib).

You need to add mysqlconnector.jar file found here: ( http://dev.mysql.com/downloads/connector/j/ ) into your lib folder of your project. Include it with your project and then you can access your connection with database.
Add that jar into the buildpath.

Loading JDBC Driver at Runtime

I'm using the following code to load a driver class:
public class DriverLoader extends URLClassLoader {
private DriverLoader(URL[] urls) {
super(urls);
File driverFolder = new File("driver");
File[] files = driverFolder.listFiles();
for (File file : files) {
try {
addURL(file.toURI().toURL());
} catch (MalformedURLException e) {
}
}
}
private static DriverLoader driverLoader;
public static void load(String driverClassName) throws ClassNotFoundException {
try {
Class.forName(driverClassName);
} catch (ClassNotFoundException ex) {
if (driverLoader == null) {
URL urls[] = {};
driverLoader = new DriverLoader(urls);
}
driverLoader.loadClass(driverClassName);
}
}
}
Although the class loads fine I can't establish a Database connection (No suitable driver found for ...) no matter which driver I try.
I assume this is because I'm not loading the driver class using Class.forName (which wouldn't work since I'm using my own ClassLoader). How can I fix this?

You need to create an instance of the driver class before you can connect:
Class drvClass = driverLoader.loadClass(driverClassName);
Driver driver = drvClass.newInstance();
Once you have the instance you can either use that instance to connect:
Properties props = new Properties();
props.put("user", "your_db_username");
props.put("password", "your_db_password");
Connection con = driver.connect("jdbc:postgresql:...", props);
As an alternative, if you want to keep using DriverManager you must register the driver with the DriverManager manually:
DriverManager.registerDriver(driver);
Then you should be able to use the DriverManager to establis a connection.
If I recall it correctly there was a problem with the DriverManager refusing to connect if the driver itself was not loaded by the same classloader as the DriverManager. If that (still) is the case, you need to use Driver.connect() directly.

You should establish connection in a class loaded by your DriverLoader. So, load the connection establishment code using DriverLoader and then call JDBC from it.

You need to add a Classpath reference in the manifest. Follow these simple steps:
add a folder "lib" to your application
place "mysql-connector-java-5.1.18-bin" in lib
now open your "MANIFEST.MF" and go to tab "RUNTIME"
on bottom right, you would see "classpath" ; click "Add"
now add the folder lib [created in step 1] along with the jar file
in this way, whenever a runtime EclipseApplication /OSGi Application is started this jar file is exported along too. So the connectivity would then be available there too.