I am building a data pipeline, where I am sending scraped data via Kafka to mongoDB Atlas (all in the cloud, not local).
I initialized a Producer p with my Confluent Kafka configs. Then I extract WKNs and ISINs of every company I have in my list from the internet. I am sending this data in JSON format to Kafka. See here:
p.produce(
"wkns_and_isins",
key=company,
value=identification_numbers,
callback=delivery_report,
)
Now I want to set the JSON-key as the document "_id" when I am sending the JSON data to my cloud-managed mongoDB Atlas. Therefore I am using the MongoDB Atlas Sink Connector. In the mongoDB it should look like this:
{
_id: 'Adidas',
value: 'XXX / XXX'
}
How will that work?