Connect to MongoDB atlas within AWS VPC using AWS MSK connector

Question

I am trying to use MongoDB change stream using Kafka. I chose AWS MSK since my entire infrastructure is within AWS and it will be easy to integrate with other AWS services.

I created an AWS MSK cluster within the VPC and created an AWS MSK connect using mongodb Kafka connector. I am getting the following error (Exception opening socket, connect timed out)

To test whether it has to do with VPC, I gave the public access to MongoDB by whitelisting 0.0.0.0, and then the connector is working fine. So I believe it has something to do with the VPC.

I tried multiple times with AWS MSK with VPC with public subnets, NAT gateway, etc, and it is still not working. Am getting the same error. I believe the AWS MSK connecter uses the same VPC configuration as AWS MSK since there's no separate configuration for AWS MSK Connect.

user_7378931 · Answer

The answer is to create a Private Endpoint in Atlas then a VPC endpoint in AWS.

There is a section in this guide that explains how to set up the private endpoint: https://www.mongodb.com/developer/languages/java/integrating-mongodb-amazon-apache-kafka/?bpass=true#create-mongodb-atlas-cluster

For me even though we had a peering connection set up between AWS and Atlas with all of the correct routes etc, we still had to go via this private endpoint option, it's a limitation of the MSK I believe

Connect to MongoDB atlas within AWS VPC using AWS MSK connector

1 Answers