a framework for testing the performance of Kafka cluster

user_7838169 0
Viewed 577

Is there any comprehensive test framework for testing the performance of Kafka 2.12 cluster in which test done using a predefined dataset?

I looked at Jmeter and Pepper-box but it seems they don't fit my requirement

thanks in advance

apache-kafka testing
1 Answers

Actually, both tools are used for performance testing of Kafka but Kafka also have the executable script to test the performance of Producer and Consumer.

kafka-producer-perf-test.sh

kafka-consumer-perf-test.sh

Setup

bin/kafka-topics.sh \
  --zookeeper localhost:2181 \
  --create \
  --topic test-rep-one \
  --partitions 6 \
  --replication-factor 1
bin/kafka-topics.sh \
  --zookeeper localhost:2181 \
  --create \
  --topic test \
  --partitions 6 --replication-factor 3

Producer

A single thread, no replication

bin/kafka-producer-perf-test.sh \
  --topic test \
  --num-records 50000000 \
  --record-size 100 \
  --throughput -1 \
  --producer-props acks=1 \
  bootstrap.servers=localhost:9092 \
  buffer.memory=67108864 \
  batch.size=8196

Single-thread, async 3x replication

    bin/kafk-topics.sh \
  --zookeeper zookeeper.example.com:2181 \
  --create \
  --topic test \
  --partitions 6 \
  --replication-factor 3
bin/kafka-producer-perf-test.sh \
  --topic test \
  --num-records 50000000 \
  --record-size 100 \
  --throughput -1 \
  --producer-props acks=1 \
  bootstrap.servers=localhost:9092 \
  buffer.memory=67108864 \
  batch.size=8196

Single-thread, sync 3x replication

bin/kafka-producer-perf-test.sh \
  --topic test \
  --num-records 50000000 \
  --record-size 100 \
  --throughput -1 \
  --producer-props acks=1 \
  bootstrap.servers=localhost:9092 \
  buffer.memory=67108864 batch.size=64000

Three Producers, 3x async replication

bin/kafka-producer-perf-test.sh \
  --topic test \
  --num-records 50000000 \
  --record-size 100 \
  --throughput -1 \
  --producer-props acks=1 \
  bootstrap.servers=localhost:9092 \
  buffer.memory=67108864 \
  batch.size=8196

Throughput Versus Stored Data

bin/kafka-producer-perf-test.sh \
  --topic test \
  --num-records 50000000 \
  --record-size 100 \
  --throughput -1 \
  --producer-props acks=1 \
  bootstrap.servers=localhost:9092 \
  buffer.memory=67108864 batch.size=8196

Effect of message size

for i in 10 100 1000 10000 100000; do
  echo ""
  echo $i
  bin/kafka-producer-perf-test.sh \
    --topic test \
    --num-records $((1000*1024*1024/$i))\
    --record-size $i\
    --throughput -1 \
    --producer-props acks=1 \
    bootstrap.servers=localhost:9092 \
    buffer.memory=67108864 \
    batch.size=128000

Consumer

Consumer throughput

bin/kafka-consumer-perf-test.sh \
  --zookeeper localhost:2181 \
  --messages 50000000 \
  --topic test \
  --threads 1

3 Consumers On three servers, run:

bin/kafka-consumer-perf-test.sh \
  --zookeeper localhost:2181 \
  --messages 50000000 \
  --topic test \
  --threads 1

End-to-end Latency

bin/kafka-run-class.sh \
  kafka.tools.TestEndToEndLatency \
  localhost:9092 \
  localhost:2181 \
  test 5000

Producer and consumer

bin/kafka-run-class.sh \
  org.apache.kafka.tools.ProducerPerformance \

bin/kafka-producer-perf-test.sh \
  --topic test \
  --num-records 50000000 \
  --record-size 100 \
  --throughput -1 \
  --producer-props acks=1 \
  bootstrap.servers=localhost:9092 \
  buffer.memory=67108864 \
  batch.size=8196

bin/kafka-consumer-perf-test.sh \
  --zookeeper localhost:2181 \
  --messages 50000000 \
  --topic test \
  --threads 1
user_44224790
Related