Kafka using Docker for production clusters

user_11895146 0
Viewed 1782

We need to build a Kafka production cluster with 3-5 nodes in cluster ,

We have the following options:

  1. Kafka in Docker containers (Kafka cluster include zookeeper and schema registry on each node)

  2. Kafka cluster not using docker (Kafka cluster include zookeeper and schema registry on each node)

Since we are talking on production cluster we need good performance as we have high read/write to disks (disk size is 10T), good IO performance, etc.

So does Kafka using Docker meet the requirements for productions clusters?

more info - https://www.infoq.com/articles/apache-kafka-best-practices-to-optimize-your-deployment/

apache-kafka docker apache-zookeeper linux
1 Answers

It can be done, sure. I have no personal experience with it, but if you don't otherwise have experience managing other stateful containers, I'd suggest avoiding it.

As far as "getting started" with Kafka in containers, Kubernetes is the most documented way, and Strimzi (free, optional commercial support by Lightbend) or Confluent Operator (commercial support by Confluent) can make this easy when using Kubernetes or Openshift. Or DC/OS offers a Kafka service over Mesos/Marathon. If you don't already have any of these services, then I think it's apparent that you should favor not using containers.

Bare metal or virtualized deployments would be much easier to maintain than hand-deployed containerized ones, from what I have experienced. Particularly for logging, metric gathering, and statically assigned Kafka listener mappings over the network. Confluent provides Ansible scripts for doing deployments to such environments

That isn't to say there's companies that have been successful at it, or at least tried. IBM, RedHat, and Shopify immediately pop up in my searches, for example

Here's a few talk about things to consider when Kafka is in containers https://www.confluent.io/kafka-summit-london18/kafka-in-containers-in-docker-in-kubernetes-in-the-cloud

https://kafka-summit.org/sessions/running-kafka-kubernetes-practical-guide/

user_23086830
Related