How to decrease number partitions Kafka topic?

user_7806540 0
Viewed 26492

I created a topic with 4 partitions on Kafka. (set default number.partition=4). Now I want to change number partition of this topic to 3. I've tried running 

./bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic my-topic --partitions 3

but there is no change. It still has 4 partitions. Anyone know about this?

apache-kafka
6 Answers

I don't buy all the above answers. "Remove a partition causes data loss" is a vague answer. Decreasing partition numbers is not a new thing in the distributed system and in fact many systems support it. If you can afford the overhead of rebalancing the entire storage system while keeping the consistency of the data, decreasing partition is not an impossible thing to do.

In my opinion, the true reason Kafka doesn't support decreasing the partition number is due to an important property of Kafka: Kafka guarantees the order of the message within each partition but the order of the message between the partition is not guaranteed (but it's possible). This ordering property is crucial in many use cases. In the cause of removing one of the partitions, redistributing messages in the removed partition to other partitions while preserving the order is impossible because ordering between partitions is not guaranteed. No matter how you distribute the data in the removed partition, you will break the order guarantee properties of any partition you distribute into. If Kafka doesn't care about the order of messages within each partition, decreasing the partition number can easily be supported.

user_81555970

You can use create standalone java program to achieve the same , i.e increase and decrease the partition and replication using AdminUtils.

import org.I0Itec.zkclient.ZkClient;

import kafka.admin.AdminUtils;

import kafka.utils.ZKStringSerializer$;

import kafka.utils.ZkUtils;

import scala.collection.Seq;

import scala.collection.Map;

public PartitionCreator(String zkhost, String topicName, int partitions, int replifactor) {
    ZkClient zkClient = new ZkClient(zkhost, 30000, 30000, ZKStringSerializer$.MODULE$);
    zkUtils = ZkUtils.apply(zkClient, false);

    this.topicName = topicName;
    this.partitions = partitions;
    this.replifactor = replifactor;
}

public void createPartion() {

    AdminUtils.createTopic(zkUtils, topicName, partitions, replifactor, new Properties());
    System.out.println("created/updated topic..");
}

Note: createTopic() internally updates the topic if topic not available.

user_10536970

Apache Kafka provides us with alter command to change Topic behavior and add/modify configurations. We will be using alter command to add more partitions to an existing Topic. Note: While Kafka allows us to add more partitions, it is NOT possible to decrease the number of partitions of a Topic.

user_45372680
Related