AWS MSK Public cluster secured with SASL/SCRAM

Viewed 204

I configured an AWS MSK cluster with public access. I created an AWS Secret via Secrets Manager and assigned it to the cluster.

Based on that secret, I managed to publish messages to MSK (I think). However, when I try to read messages from the topic, I get unauthorized errors.

I followed this documentation page, and I ended up with the following settings:

  • user_jaas.conf
KafkaClient {
   org.apache.kafka.common.security.scram.ScramLoginModule required
   username="<username>"
   password="<password>";
};
  • config.properties
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
  • export KAFKA_OPTS=-Djava.security.auth.login.config=/home/ec2-user/user_jaas.conf

Then, I'm using the consumer shell script as follows:

./kafka-console-consumer.sh --bootstrap-server <urls> --topic <topicName> --from-beginning --consumer.config client.properties

And I'm getting the following error:

[2022-06-21 13:34:17,566] WARN [Consumer clientId=consumer-console-consumer-13840-1, groupId=console-consumer-13840] Error while fetching metadata with correlation id 2 : {users=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient) [2022-06-21 13:34:17,568] ERROR [Consumer clientId=consumer-console-consumer-13840-1, groupId=console-consumer-13840] Topic authorization failed for topics [topicName] (org.apache.kafka.clients.Metadata) [2022-06-21 13:34:17,571] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$) org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [topicName] Processed a total of 0 messages

Am I missing an important aspect of the documentation?

1 Answers

You need to create acls to read from this topic.

Keep in mind that in order to do that you should Address to the zookkeeper due to limitations in aws msk.

For example:

./bin/kafka-acls.sh --authorizer-properties zookeeper.connect=ZooKeeper-Connection-String --add --allow-principal "User:CN=Distinguished-Name" --operation Read --group=* --topic Topic-Name

Related