I configured an AWS MSK cluster with public access. I created an AWS Secret via Secrets Manager and assigned it to the cluster.
Based on that secret, I managed to publish messages to MSK (I think). However, when I try to read messages from the topic, I get unauthorized errors.
I followed this documentation page, and I ended up with the following settings:
user_jaas.conf
KafkaClient {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="<username>"
password="<password>";
};
config.properties
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
export KAFKA_OPTS=-Djava.security.auth.login.config=/home/ec2-user/user_jaas.conf
Then, I'm using the consumer shell script as follows:
./kafka-console-consumer.sh --bootstrap-server <urls> --topic <topicName> --from-beginning --consumer.config client.properties
And I'm getting the following error:
[2022-06-21 13:34:17,566] WARN [Consumer clientId=consumer-console-consumer-13840-1, groupId=console-consumer-13840] Error while fetching metadata with correlation id 2 : {users=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient) [2022-06-21 13:34:17,568] ERROR [Consumer clientId=consumer-console-consumer-13840-1, groupId=console-consumer-13840] Topic authorization failed for topics [topicName] (org.apache.kafka.clients.Metadata) [2022-06-21 13:34:17,571] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$) org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [topicName] Processed a total of 0 messages
Am I missing an important aspect of the documentation?