How to set produced json key as document _id for mongoDB Atlas

Viewed 25

I am building a data pipeline, where I am sending scraped data via Kafka to mongoDB Atlas (all in the cloud, not local).

I initialized a Producer p with my Confluent Kafka configs. Then I extract WKNs and ISINs of every company I have in my list from the internet. I am sending this data in JSON format to Kafka. See here:

p.produce(
            "wkns_and_isins",
            key=company,
            value=identification_numbers,
            callback=delivery_report,
        )

Now I want to set the JSON-key as the document "_id" when I am sending the JSON data to my cloud-managed mongoDB Atlas. Therefore I am using the MongoDB Atlas Sink Connector. In the mongoDB it should look like this:

{
_id: 'Adidas',
value: 'XXX / XXX' 
}

How will that work?

0 Answers
Related