Failing to connect to Kafka on MSK

user_4213730 0
Viewed 3019

I have created a Kafka cluster on MSK and now I'm trying to connect to the cluster with python.

I wrote this shortcode:

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers=['host1:9092', 'host2:9092'],
    value_serializer=lambda x: json.dumps(x).encode('utf-8'),
    api_version=(2, 4, 1)
)

producer.send('test', value={'hello':'world'})

The problem is that every time I running it I'm getting this error:

KafkaTimeoutError: Failed to update metadata after 60.0 secs.

I thought it might be related to Kafka creating topics so I added this line to the configuration.

auto.create.topics.enable=true

But I'm still getting the same error.

This is my full configuration file:

default.replication.factor=3
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=1
num.replica.fetchers=2
socket.request.max.bytes=104857600
unclean.leader.election.enable=true
auto.create.topics.enable=true
zookeeper.connection.timeout.ms=5000

What am I missing here? I read somewhere that is may relate to SSL authentication but at any step, there wasn't any .pem file, .ca file, or anything like that.

aws-msk kafka-producer-api kafka-consumer-api apache-kafka python
1 Answers

You probably fail to connect to the MSK. The error you experience is a Timeout error.

One wrong assumption when first starting to use MSK is, you can connect to it from outside AWS network. That is a wrong assumption. AWS has a detailed document on how can you access your MSK cluster.

If you're not running your client from an EC2 instance inside your MSK VPC, you won't be able to reach it. Even if you open the relevant security rules in the MSK security group policy.

I have spend time trying multiple proxies to access the MSK from outside AWS, with no success. Just follow the guide I referred above, and it will probably fix your connection issue.

Also, If you're new to MSK I highly suggest you to go through the Getting Started tutorial, or at least Steps 5 (Create a Topic) and 6 (Produce and Consume Data).

user_8405820
Related