I am having trouble connecting to AWS MSK Kafka (with with Access Control Method configured to None) from my laptop, error is: NoBrokersAvailable
I am using plaintext conn over port 9092:
from kafka import KafkaProducer
kafka_brokers = [ "xxx.kafka.ap-east-1.amazonaws.com:9092","xxx.kafka.ap-east-1.amazonaws.com:9092","xxx.kafka.ap-east-1.amazonaws.com:9092" ]
producer = KafkaProducer(bootstrap_servers=kafka_brokers)
Ref: AWS MSK - plaintext broker port 9092, TLS 9094 https://docs.aws.amazon.com/msk/latest/developerguide/client-access.html
I tried three things (Which did not help):
a. Under my VPC default security group added inbound+outbound source/destination over 9092+9094 source/dest 0.0.0.0/0
b. Tried run over AWS Client VPN following instructions below:
- https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-getting-started.html#cvpn-getting-started-certs
- Instructions to generate Certs: https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/client-authentication.html#mutual
- https://www.youtube.com/watch?v=Bv70DoHDDCY
c. Turn off Firewall on my laptop
I see some others asking similar questions: How to connect Python consumer to AWS MSK
Some folks suggesting it's simply not-supported for clients on local machines/laptop to connect to AWS MSK: How can I access my AWS MSK managed kafka queue from my local machine and EC2 instances in other regions Is this true/official?
I'm thinking create EIP (Elastic IP) https://repetitive.it/aws-msk-how-to-expose-the-cluster-on-the-public-network/?lang=en Then somehow associate with broker domain names xxx.kafka.ap-east-1.amazonaws.com (but not sure how) and configure inbound outbound rules.
But I haven't been able to find anything that helps.
I'm on my wits end, this problem already consumed whole day. Any help greatly appreciated.
Thanks