Kafka/questDB JDBC Sink Connector: tables not created using "topics.regex"

user_8383962 0
Viewed 310

I am using "confluentinc/kafka-connect-jdbc:10.2.6" as my JDBC connector to transfer Kafka topics into my questDB.

When I provide explicit topic names it is working as expected. But when I use topic names based on regex then it's not working, the tables are not being created in my questDB database.

What am I missing in my JDBC settings?

Thanks!

explicit version (working):

{
  "name": "jdbc_sink_ftx",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "topics": "trades-FTX-ETH-USD-PERP, trades-FTX-FTM-USD-PERP",
    "table.name.format": "${topic}",
    "connection.url": "jdbc:postgresql://questdb:8812/qdb?useSSL=false",
    "connection.user": "admin",
    "connection.password": "quest",
    "auto.create": "true",
    "insert.mode": "insert",
    "dialect.name": "PostgreSqlDatabaseDialect"
  }
}

regex-version (not working):

{
  "name": "jdbc_sink_ftx_regex",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "topics.regex": "FTX",
    "table.name.format": "${topic}",
    "connection.url": "jdbc:postgresql://questdb:8812/qdb?useSSL=false",
    "connection.user": "admin",
    "connection.password": "quest",
    "auto.create": "true",
    "insert.mode": "insert",
    "dialect.name": "PostgreSqlDatabaseDialect"
  }
}
questdb confluent-kafka-python jdbc
2 Answers

It would be best to check QuestDB logs. Grepping for ' E ' should show errors that usually include description and OS error code in this case. Topic names in Kafka Connector are used as table names. QuestDB uses table names as directories on disk and validates them for security purposes. '.', '/' and '' are not allowed characters in table name. Specific OS can also apply additional restrictions on file names.

It would be best to check QuestDB logs though.

user_36465350

The regex needs to match with the complete topic names like so:

"topics.regex": "trades-FTX-.*"
user_83839620
Related