Can I map multiple buckets with multiple topics in a single Kafka-Connector S3 sink connector?

Viewed 709

Google didn't help me, so I want to ask you. I have a lot of kafka topics, and I want to store the messages of a particular topic in a particular S3 bucket.

Do I need to create an S3 sink connector for each bucket or can I configure all the stuff in a single S3 json file (map a bucket with a topic)?

Thanks for your help! :)

1 Answers

The S3 bucket name is specified per-connector, so you will need to create one connector per bucket.

Note that you can specify multiple topics per connector (topics / topics.regex), so if you wanted multiple topics going to one bucket (but different paths), you could do this with a single connector. But for multiple buckets, you'll need multiple connectors.

Related