Multi-tenant schema-based application with r2dbc

user_4028399 0
Viewed 1973

I'm working on a multi-tenant reactive application using Spring-Webflux + Spring-data-r2dbc with the r2dbc driver to connect to a Postgresql database. The multi-tenant part is schema-based: one schema per tenant. So depending of the context (the user logged in for instance) the requests will hit a certain schema of the database.

I'm struggling on how to accomplish that in r2dbc. Ideally it would be the way Hibernate does with MultiTenantConnectionProvider (see example 16.3).

What I found and what I've done so far:

  • Could use AbstractRoutingConnectionFactory as mentioned here. But I'm force to create a ConnectionFactory by tenant/schema. It seems to me that is far from efficient/scalable, I would rather use a connection pool like r2dbc-pool

  • I looked at PostgresqlConnectionFactory. Interesting thing here is that on prepareConnection there's a call on setSchema(connection):

    private Mono<Void> setSchema(PostgresqlConnection connection) {
    if (this.configuration.getSchema() == null) {
        return Mono.empty();
    }

    return connection.createStatement(String.format("SET SCHEMA '%s'", this.configuration.getSchema()))
        .execute()
        .then();
}

May be I need to find a way to override this in order to get the schema dynamically from the context instead of the configuration ?

  • Otherwise I could try to specify the schema in the request as a table prefix:

        String s = "tenant-1";
    databaseClient.execute("SELECT * FROM \"" + s + "\".\"city\"")
            .as(City.class)
            .fetch()
            .all()

But I can't use SpringData anymore or I need to override every request to pass the tenant as a parameter.

Any hints/help appreciated :)

spring-data-r2dbc r2dbc multi-tenant
4 Answers

I ran into this as well.

Here is what I am currently doing:

  • Publish the PostgresqlConnectionConfigurationBuilder and the PostgresqlConnectionFactory as a Bean:

    @Bean
public PostgresqlConnectionConfiguration.Builder postgresqlConnectionConfiguration() {
    return PostgresqlConnectionConfiguration.builder()
            .host("localhost")
            .port(5432)
            .applicationName("team-toplist-service")
            .database("db")
            .username("user")
            .password("password");
}

@Bean
@Override
public PostgresqlConnectionFactory connectionFactory() {
    return new PostgresqlConnectionFactory(postgresqlConnectionConfiguration()
            .build());
}

So that I can later (in my business method) create a new PostgresqlConnectionFactory using the injected PostgresqlConnectionConfigurationBuilder instance - but now with the "schema" setter called on the builder as well (after extracting tenant information from the incoming org.springframework.web.reactive.function.server.ServerRequest which i passed down from my route bean.

My db schemas follow the pattern appname_tenantId, so we have an "appName" statically configured to ie "app_name", so i end up with schema names like "app_name_foo_bar123"

next, we have a tenant identifier which in my case would come from a request header which is guaranteed to be set by an apache server sitting upstream (passing along a X-Tenant-Id header for incoming requests so to not rely on URLs to do tenant specific routing)

So my "logic" currently looks a bit like this: 

public Flux<TopTeam> getTopTeams(ServerRequest request) {

    List<String> tenantHeader = request.headers().header("X-Tenant-Id");
    // resolve relevant schema name on the fly
    String schema = (appName+ "_" + tenantHeader.iterator().next()).replace("-", "_");
    System.out.println("Using schema: " + schema);
    // configure connfactory with schema set on the builder
    PostgresqlConnectionFactory cf = new PostgresqlConnectionFactory(postgresqlConnectionConfiguration.schema(schema).build());
    // init new DatabaseClient with tenant specific connection
    DatabaseClient cli = DatabaseClient.create(cf);


        return cli
                .execute("select * from top_teams ").fetch().all()
                .flatMap(map -> {

                    ...
                    });
                });
    }

This logic of course can be abstracted away, not sure however where to put it, maybe this can be moved to a MethodArgumentResolver so we can just inject an already configured DatabaseClient

ps: this only solves the multitenancy issue when using the DatabaseClient. I am not sure how one would make this work with R2dbcRepositories

user_7610140

Thanks for the answers. I finally ended up with this solution:

Build a ConnectionFactory by tenant/schema:

public class CloudSpringUtilsConnectionFactoryBuilder implements ConnectionFactoryBuilder {

@Override
public ConnectionFactory buildConnectionFactory(String schema) {
    PostgresqlConnectionConfiguration configuration = getPostgresqlConnectionConfigurationBuilder(schema)
            .build();
    return new PostgresqlConnectionFactory(configuration);
}

@Override
public ConnectionFactory buildSimpleConnectionFactory() {
    PostgresqlConnectionConfiguration configuration = getPostgresqlConnectionConfigurationBuilder(null)
            .build();
    return new PostgresqlConnectionFactory(configuration);
}

protected PostgresqlConnectionConfiguration.Builder getPostgresqlConnectionConfigurationBuilder(String schema) {
    return PostgresqlConnectionConfiguration
            .builder()
            .username(dbUser)
            .password(dbPassword)
            .host(dbHost)
            .port(dbPort)
            .database(dbName)
            .schema(schema);
}

Create a TenantRoutingConnectionFactory to get the right ConnectionFactory depending of the tenant. In our case tenant is extracted from the authentication Principal (a token convert to a UserProfile):

public class TenantRoutingConnectionFactory extends AbstractRoutingConnectionFactory {

private final DatabaseMigrationService databaseMigrationService;
private final ConnectionFactoryBuilder connectionFactoryBuilder;

private final Map<String, ConnectionFactory> targetConnectionFactories = new ConcurrentHashMap<>();

@PostConstruct
private void init() {
    setLenientFallback(false);
    setTargetConnectionFactories(new HashMap<>());
    setDefaultTargetConnectionFactory(connectionFactoryBuilder.buildConnectionFactory());
}

@Override
protected Mono<Object> determineCurrentLookupKey() {
    return ReactiveSecurityContextHolder.getContext()
            .map(this::getTenantFromContext)
            .flatMap(tenant -> databaseMigrationService.migrateTenantIfNeeded(tenant)
                    .thenReturn(tenant));
}

private String getTenantFromContext(SecurityContext securityContext) {
    String tenant = null;
    Object principal = securityContext.getAuthentication().getPrincipal();
    if (principal instanceof UserProfile) {
        UserProfile userProfile = (UserProfile) principal;
        tenant = userProfile.getTenant();
    }
    ...
    log.debug("Tenant resolved: " + tenant);
    return tenant;
}

@Override
protected Mono<ConnectionFactory> determineTargetConnectionFactory() {
    return determineCurrentLookupKey().map(k -> {
        String key = (String) k;
        if (!targetConnectionFactories.containsKey(key)) {
            targetConnectionFactories.put(key, connectionFactoryBuilder.buildConnectionFactory(key));
        }
        return targetConnectionFactories.get(key);
    });
}

Note that we use Flyway in DatabaseMigrationService to create and migrate schema for each tenant we get.

user_40283990

I have created a multi-tenancy example for r2dbc but using a per-databases strategy.

Check the complete example codes here.

In some databases, the schema and database concept is equivalent. If you are stick on using a per-schema strategy, add a SQL to select schema(please do research the database you are using, and determine the right clause to set a schema) when obtaining the connection.

user_8938980

Thanks/Based on the @charlie carver answer, That's how I solved this problem:

Controller:

    @PostMapping(MAP + PATH_DDL_PROC_DB)  //PATH_DDL_PROC_DB = "/database/{db}/{schema}/{table}"
    public Flux<Object> createDbByDb(
            @PathVariable("db") String db,
            @PathVariable("schema") String schema,
            @PathVariable("table") String table) {
        return ddlProcService.createDbByDb(db,schema,table);

Service:

    public Flux<Object> createDbByDb(String db,String schema,String table) {
        return ddl.createDbByDb(db,schema,table);
    }

Repository:

    @Autowired
    PostgresqlConnectionConfiguration.Builder connConfig;

    public Flux<Object> createDbByDb(String db,String schema,String table) {
        return createDb(db).thenMany(
                Mono.from(connFactory(connConfig.database(db)).create())
                    .flatMapMany(
                            connection ->
                                    Flux.from(connection
                                                      .createBatch()
                                                      .add(sqlCreateSchema(db))
                                                      .add(sqlCreateTable(db,table))
                                                      .add(sqlPopulateTable(db,table))
                                                      .execute()
                                             )));
    }

    private Mono<Void> createDb(String db) {

        PostgresqlConnectionFactory
                connectionFactory = connFactory(connConfig);

        DatabaseClient ddl = DatabaseClient.create(connectionFactory);

        return ddl
                .execute(sqlCreateDb(db))
                .then();
    }

Connection Class:

@Slf4j
@Configuration
@EnableR2dbcRepositories
public class Connection extends AbstractR2dbcConfiguration {

    /*
     **********************************************
     * Spring Data jdbc:
     *      DDL: does support JPA.
     *
     * Spring Data R2DBC
     *      DDL:
     *          -does no support JPA
     *          -To achieve DDL, uses R2dbc.DataBaseClient
     *
     *      DML:
     *          -it uses R2dbcREpositories
     *          -R2dbcRepositories is different than
     *          R2dbc.DataBaseClient
     * ********************************************
     */
    @Bean
    public PostgresqlConnectionConfiguration.Builder connectionConfig() {
        return PostgresqlConnectionConfiguration
                .builder()
                .host("db-r2dbc")
                .port(5432)
                .username("root")
                .password("root");
    }

    @Bean
    public PostgresqlConnectionFactory connectionFactory() {
        return
                new PostgresqlConnectionFactory(
                        connectionConfig().build()
                );
    }
}

DDL Scripts:

@Getter
@NoArgsConstructor(access = AccessLevel.PRIVATE)
public final class DDLScripts {

    public static final String SQL_GET_TASK = "select * from tasks";

    public static String sqlCreateDb(String db) {
        String sql = "create database %1$s;";
        String[] sql1OrderedParams = quotify(new String[]{db});
        String finalSql = format(sql,(Object[]) sql1OrderedParams);
        return finalSql;
    }

    public static String sqlCreateSchema(String schema) {
        String sql = "create schema if not exists %1$s;";
        String[] sql1OrderedParams = quotify(new String[]{schema});
        return format(sql,(Object[])  sql1OrderedParams);
    }

    public static String sqlCreateTable(String schema,String table) {

        String sql1 = "create table %1$s.%2$s " +
                "(id serial not null constraint tasks_pk primary key, " +
                "lastname varchar not null); ";
        String[] sql1OrderedParams = quotify(new String[]{schema,table});
        String sql1Final = format(sql1,(Object[])  sql1OrderedParams);

        String sql2 = "alter table %1$s.%2$s owner to root; ";
        String[] sql2OrderedParams = quotify(new String[]{schema,table});
        String sql2Final = format(sql2,(Object[])  sql2OrderedParams);

        return sql1Final + sql2Final;
    }

    public static String sqlPopulateTable(String schema,String table) {

        String sql = "insert into %1$s.%2$s values (1, 'schema-table-%3$s');";
        String[] sql1OrderedParams = quotify(new String[]{schema,table,schema});
        return format(sql,(Object[]) sql1OrderedParams);
    }

    private static String[] quotify(String[] stringArray) {

        String[] returnArray = new String[stringArray.length];

        for (int i = 0; i < stringArray.length; i++) {
            returnArray[i] = "\"" + stringArray[i] + "\"";
        }
        return returnArray;
    }
}
user_76816960
Related