We have Kafka confluent cluster , cluster include 3 Kafka brokers ,
Version details:
Kafka machines are installed on rhel version 7.2
Kafka confluent version is 0.1x
Zookeeper version: 3.4.10
schema-registry version: 4.0.0
Each Kafka broker machine include the following services
Kafka broker
Zookeeper server
Schema registry
Now we want to add a additionally new 3 Kafka brokers machines to the current Kafka cluster ( the Additional Kafka machines are – kafka04/05/06 with the same Kafka version – 0.1X , )
So cluster should be finally with:
6 Kafka brokers machines - kafka01, kafka02 , kafka03 , kafka04 , kakfa05 , kafka05
3 zookeeper servers - kafka01, kafka02 , kafka03
3 schema registry services - kafka01, kafka02 , kafka03
In order to connect the new 3 Kafka brokers to the existing Kafka cluster we need to change the Configuration on all Kafka Machines ( the old Kafka machines and the new Kafka machines )
We are not sure what are exactly the configuration files in Kafka brokers that should be changed, But From my understanding We should change the Kafka and zookeeper settings as the following
I will happy to get remarks/notes about the following procedure
Edit the
server.propertiesfile on the new Kafka brokers -kafka04/05/06, and change thebroker.idparameter as the followingOn kafka04 -
broker.id=4On kafka05 -broker.id=5On kafka06 -broker.id=6Edit the server.properties on all Kafka machines -
kafka01/02/03/04/05/06And change the following parameters to the number of total nodes in the clusteroffsets.topic.replication.factor=6 transaction.state.log.replication.factor=6On the new Kafka machines -
Kafka 04/05/06edit theserver.propertiesand update the parameter --zookeeper.connectwith the zookeeper server ip’s that located onkafka01,kafka02,kafka03Example
zookeeper.connect=10.10.10.1:2181,10.10.10.2:2181,10.10.10.3:2181On Kafka machines –
kafka 04/05/06, edit the file -zookeeper.propertiesas the
Followingserver.1=10.10.10.1:2888:3888 server.2=10.10.10.2:2888:3888 server.3=10.10.10.3:2888:3888Edit the file -
myidfile onKafka 04/05/06, and change the parameterbroker.idas the Followingon
kafka04set:broker.id=4on
kafka05set:broker.id=5on
kafka06set:broker.id=6After settings as above , restart all Kafka brokers services on
kafka01/02/03/04/05/06and Restart the zookeeper servers onkafka01/02/03And verify all Kafka services and zookeeper service started successfully
Reference info - https://www2.microstrategy.com/producthelp/current/PlatformAnalytics/en-us/Content/Add_kafka_node_to_kafka_cluster.htm