Has anyone used Kafka connect with one or more of the following cloud streaming services?
- AWS Kinesis
- AWS MSK
- Azure Event Hubs
FWIW we're looking to send data from Kafka to ElasticSearch without needing to use additional component such as Logstash or FileBeat.
At first I thought we could only do this using the Confluent platform, but then read that Kafka Connect is just an open-source Apache project. The only need for Confluent would be if we want/need to use one of the proprietary connectors, but given the ElasticSearch Sink connector is the only one we need (at least for now) and this is a community connector - see here (and here for licensing info), we might be able to do this using one of the AWS/Azure streaming services assuming this is supported (Note: AWS or Azure represents a path of less resistance as the company I work for already has vendor relationships with both AWS & Microsoft. Not saying we won't use Confluent or migrate to it at some stage, but for now Azure/AWS is going to be easier to get across the line).
I found a Microsoft document that implies we can use Azure Event Hubs with Kafka Connect, even though AEH is a bit different to open source Kafka... not sure about AWS Kinesis or MSK - I assume MSK would be fine, but not sure... any guidance/blogs/articles would be much appreciated....
Cheers,