Using Confluent Schema Registry with MSK

user_7027284 0
Viewed 1104

Is it possible to integrate Confluent Schema Registry with AWS MSK? If you have done this before, can you please provide some pointers / blogs you followed to achieve it?

aws-msk confluent-schema-registry apache-kafka
2 Answers

It is possible. My setup uses ec2 and docker.

  1. Download the IAM auth jar if you are using IAM based auth
mkdir -p /usr/share/java/aws
wget -P /usr/share/java/aws https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar
chmod -R 444 /usr/share/java/aws
  1. Use confluent official docker image for schema registry
...

  schema-registry:
    image: confluentinc/cp-schema-registry:5.4.6-1-ubi8
    hostname: schema-registry
    container_name: schema-registry
    ports:
      - "8081:8081"
    volumes:
      - /usr/share/java/aws/aws-msk-iam-auth-1.1.1-all.jar:/usr/share/java/cp-base-new/aws-msk-iam-auth-1.1.1-all.jar
      - /usr/share/java/aws/aws-msk-iam-auth-1.1.1-all.jar:/usr/share/java/rest-utils/aws-msk-iam-auth-1.1.1-all.jar
    environment: # https://docs.confluent.io/platform/current/schema-registry/installation/config.html#schemaregistry-config
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
      SCHEMA_REGISTRY_HOST_NAME: "${HOSTNAME}" # 
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: "${BOOTSTRAP_BROKERS_SASL_IAM}"
      SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL: "SASL_SSL"
      SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM: "AWS_MSK_IAM"
      SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG: "software.amazon.msk.auth.iam.IAMLoginModule required awsDebugCreds=true;"
      SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

...
  • HOSTNAME is your ec2 machine DNS name or IP, example ip-10-0-0-84.ec2.internal
  • BOOTSTRAP_BROKERS_SASL_IAM is comma separated host1:port,host2:port urls. For port information see this

If you are using PLAINTEXT or SSL auth, last 4 environment variable changes. And you don't have to download iam auth jar

  1. Configure source or sink connector with these property
...
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schemas.enable=true
value.converter.schema.registry.url=http://ip-10-0-0-84.ec2.internal:8081
value.converter.enhanced.avro.schema.support=true

Thats it.
Do open 8081 port in your security group of EC2 instance for MSK cluster

Resource:

Alternate option I have tried is AWS Glue Schema registry But we had to use KSQL, and KSQL does't have 3rd party schema registry integration or custom SerDe Github issue

user_78577010

It is possible and is no different than using it with regular Kafka installation.

You point its bootstrap server property at MSK and you point client applications at it.

user_23086830
Related