How can I rerun an Apache Flink Postgres JDBC job without getting "No suitable driver found" exception

The first question you can refer JDBC driver cannot be found when reading a DataSet from an SQL database in Apache Flink.

Second, if you use session mode. It can be easy to rerun the Flink job without restart the cluster. you can log in job manager shell then use the command rerun job.

Class.forName("org.postgresql.Driver"); will trigger static method block, so you DriverManager can get driver class. see:

// from org.postgresql.Driver
    static {
        try {
            register();
        } catch (SQLException var1) {
            throw new ExceptionInInitializerError(var1);
        }
    }

I have this pom.xml dependency for Postgres for Apache Flink 1.13:

        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <version>9.4-1201-jdbc41</version>
        </dependency>

you can have a Postgres connector class for example:

public class PostgreSQLConnector {
    private static volatile PostgreSQLConnector instance;
    private Connection connectionDB = null;

    public PostgreSQLConnector(your params) {
        ...
    }

    public static PostgreSQLConnector getInstance() {
        PostgreSQLConnector postgreSQLConnector = instance;
        if (postgreSQLConnector != null)
            return postgreSQLConnector;
        synchronized (PostgreSQLConnector.class) {
            if (instance == null) {
                instance = new PostgreSQLConnector(your params);
            }
            return instance;
        }
    }

    public Connection getConnectionDB() throws SQLException {
        if (checkNullConnection()) CreateConnection();
        return connectionDB;
    }

    public void CheckConnection() throws SQLException {
        if (checkNullConnection()) CreateConnection();
    }

    public void CreateConnection() throws SQLException {
            try {
                Class.forName(sink.driverName);
                connectionDB = DriverManager.getConnection(fullUrl, username, password);
            } catch (Exception e) {
                ...
            }
    }

public boolean checkNullConnection() throws SQLException {
        return (connectionDB == null || connectionDB.isClosed());
    }
}

then you can create a RichSourceFunction and create the connection in the overrides open method, not in the run

public class JdbcSource extends RichSourceFunction<Integer> {

    private final String connString;
    private static Connection dbConnection;
    private static final PostgreSQLConnector postgreSQLConnector = PostgreSQLConnector.getInstance();

    public JdbcSource(String connString) {
        this.connString = connString;
    }

    @Override
    public void open(Configuration parameters) throws SQLException {
        dbConnection = postgreSQLConnector.getConnectionDB();
    }

    @Override
    public void close() throws Exception {
        if (dbConnection != null) dbConnection.close();
    }

    @Override
    public void run(SourceContext<Integer> ctx) throws Exception {
       do something here with the connection
    }

    @Override
    public void cancel() {
    }
}

Something like that you could maybe try and it should work

According to the official documentation of PostgreSQL JDBC driver, if you are using Java 1.6+, you can just put the driver's jar file into the classpath. The driver will be loaded by the JVM automatically. So the question is how to place the driver's jar file into the classpath.

Since you are using docker to deploy a session cluster, there's two way that may works:

1. Put the driver's jar file into docker image

Run and access the image with the command:

docker docker run -it -v $PWD:/tmp/flink <address to image> -- bash

Copy the driver's jar file into the folder /opt/flink/lib.

Create a new image from the container. Since /opt/flink/lib is loaded as classpath by default, now the driver's jar file is located at the classpath.

2. Package the driver's jar into your user jar

Add maven-assembly-plugin to the pom.xml of your maven project. Recompile your project and get a jar file with dependencies. In this jar, the PostgreSQL JDBC driver is packaged together.