How to spread existing kafka topic partitions into more directories?

user_5873780 0
Viewed 4067

By default, kafka uses one directory to keep the log. To increase performance, it is advised to mount more disks to the broker, and assign each disk to one directory then in server.properties enter the log.dirs= as a coma separated list of directories. The documentation says, that partitions will be distributed among the directories round-robin style. As I understand now, this is true for new topics.

I would like to distribute half of the partitions of my already created topic to a newly created log.dir while keeping the other half where they are - is there a supported way to do that ?

apache-kafka
2 Answers

The Admin api has been updated since the original answer was provided in 2016. Now (mid-2022), the kafka-reassign-partitions.sh script found in your kafka/bin directory supports moving partitions between log dirs (it also still moves partitions among brokers). None of the 3 approaches in the original answer are necessary now. kafka-reassign-partitions.sh allows you to move partitions without taking brokers offline.

Moving partitions over log dirs is supported on kafka 2.8+ and 3.x. (I believe the earliest supported version is 2.6 - but check yourself for versions before 2.8: kafka kip-455 describes the goals of the feature)

Here is the help for the --reassignment-json-file options from the script

--reassignment-json-file <String:       The JSON file with the partition
  manual assignment json file path>       reassignment configurationThe format
                                          to use is -
                                        {"partitions":
                                                [{"topic": "foo",
                                                  "partition": 1,
                                                  "replicas": [1,2,3],
                                                  "log_dirs": ["dir1","dir2","dir3"]
                                          }],
                                        "version":1
                                        }
                                        Note that "log_dirs" is optional. When
                                          it is specified, its length must
                                          equal the length of the replicas
                                          list. The value in this list can be
                                          either "any" or the absolution path
                                          of the log directory on the broker.
                                          If absolute log directory path is
                                          specified, the replica will be moved
                                          to the specified log directory on
                                          the broker.

Don't forget to set a reasonable --throttle option in MB/s.

This Cloudera documentation explains in prose, but nothing that can't be figured out from the reassign script's help.

user_35160340
Related