Kafka Connect How to make namespace agnostic to database name

Viewed 1018

My Environment

MySQL(5.7): We have multiple schemas and the naming convention is {application_name}_env.

Example: Consider we have two apps: app1 and app2

Dev Environment: The database names would be app1_dev, app2_dev

QA Environment: The database names would be app1_qa, app2_qa.

Debezium(0.8.3). The plugin is used to CDC MySQL Logs.

Connector Configuration is:

{
"name": "connector-1",
"config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",
    "database.hostname": "mysql",
    "database.port": "3306",
    "database.user": "debezium",
    "database.password": "dbz",
    "database.server.id": "184054",
    "database.server.name": "dbserver1",
    "database.history.kafka.bootstrap.servers": "kafka:9092",
    "database.history.kafka.topic": "schema-changes.inventory",
    "decimal.handling.mode": "double",
    "snapshot.mode": "when_needed",
    "table.whitelist":"{database_name}.account",
    "key.converter": "io.confluent.connect.avro.AvroConverter",
    "value.converter": "io.confluent.connect.avro.AvroConverter",
    "key.converter.schema.registry.url": "http://schema-registry:8081",
    "value.converter.schema.registry.url": "http://schema-registry:8081",
    "transforms" : "setSchema",
    "transforms.setSchema.type" : "org.apache.kafka.connect.transforms.SetSchemaMetadata$Value",
    "transforms.setSchema.schema.name" : "com.test.Account"
}

}

Spring Java Application I am using Kafka Consumer(@KafkaListener) to read the changes from the Debezium event.

I provide the avsc files and used gradle avro plugin to generate the classes.

Schema from Dev env

{
"type":"record",
"name":"Accounts",
"namespace":"com.test",
"fields":[
  {
     "name":"before",
     "type":[
        "null",
        {
           "type":"record",
           "name":"Value",
           "namespace":"dbserver1.app1_dev.account",
           "fields":[
              {
                 "name":"id",
                 "type":"long"
              }
           ],
           "connect.name":"dbserver1.app1_dev.account.Value"
        }
     ],
     "default":null
  },
  {
     "name":"after",
     "type":[
        "null",
        "dbserver1.app1_dev.account.Value"
     ],
     "default":null
  },
  {
     "name":"source",
     "type":{
        "type":"record",
        "name":"Source",
        "namespace":"io.debezium.connector.mysql",
        "fields":[
           {
              "name":"version",
              "type":[
                 "null",
                 "string"
              ],
              "default":null
           },
           {
              "name":"name",
              "type":"string"
           },
           {
              "name":"server_id",
              "type":"long"
           },
           {
              "name":"ts_sec",
              "type":"long"
           },
           {
              "name":"gtid",
              "type":[
                 "null",
                 "string"
              ],
              "default":null
           },
           {
              "name":"file",
              "type":"string"
           },
           {
              "name":"pos",
              "type":"long"
           },
           {
              "name":"row",
              "type":"int"
           },
           {
              "name":"snapshot",
              "type":[
                 {
                    "type":"boolean",
                    "connect.default":false
                 },
                 "null"
              ],
              "default":false
           },
           {
              "name":"thread",
              "type":[
                 "null",
                 "long"
              ],
              "default":null
           },
           {
              "name":"db",
              "type":[
                 "null",
                 "string"
              ],
              "default":null
           },
           {
              "name":"table",
              "type":[
                 "null",
                 "string"
              ],
              "default":null
           },
           {
              "name":"query",
              "type":[
                 "null",
                 "string"
              ],
              "default":null
           }
        ],
        "connect.name":"io.debezium.connector.mysql.Source"
     }
   },
   {
     "name":"op",
     "type":"string"
   },
   {
     "name":"ts_ms",
     "type":[
        "null",
        "long"
     ],
     "default":null
   }
 ],
 "connect.name":"com.test.Account"
 }

Issue: Since my database schemas are dynamic i.e they end with env suffix.

The Schema generated in each environment has a different namespace.

Dev: dev.app1_dev.accounts QA: dev.app1_qa.accounts

Because of the different namespace, I am not able to deserialize my dev code in QA. So If used schema generated in Dev, the code won't work in QA.

I want to make sure that namespace is consistent across all the environments.

1 Answers
Related