env KAFKA_LOG_RETENTION_BYTES not recognized by Kafka, "log.retentions.bytes" is still -1

user_3659012 0
Viewed 340

We currently run Kafka container on ARM based machine (AWS Graviton).

We follow https://docs.confluent.io/platform/current/installation/docker/config-reference.html to pass configuration to Kafka:

enter image description here

Our docker command to start Kafka is:

$ docker run -d --network mynetwork --name=kafka -p9092:9092 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092  -e KAFKA_LOG_RETENTION_BYTES=500000000 linuxforhealth/kafka-alpine:2.5.0

However, we find that certain configurations via environment variable work, while others don't.

E.g. for -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092, the configuratio is effective:

$ docker logs kafka | grep "advertised.listeners"
        advertised.listeners = PLAINTEXT://kafka:9092

E.g. for -e KAFKA_LOG_RETENTION_BYTES=500000000, the configuration is not effective:

$ docker logs kafka | grep "log.retention.bytes"
        log.retention.bytes = -1

Because of this problem, the disk ran out of space.

Please help. Thanks!

apache-kafka docker
1 Answers

Don't know where you get the guide of KAFKA_LOG_RETENTION_BYTES, search the doc you mentioned, not found it.

But, from the entrypoint of this image, I can see just KAFKA_ADVERTISED_LISTENERS been used, no KAFKA_LOG_RETENTION_BYTES:

$ docker run --rm -it linuxforhealth/kafka-alpine:2.5.0 cat /opt/lfh/kafka/bin/lfh-kafka-start.sh
#!/bin/sh
# lfh-kafka-start.sh
# runs the Kafka Standalone server

${APP_ROOT}/kafka/bin/kafka-server-start.sh ${APP_ROOT}/kafka/config/server.properties \
    --override zookeeper.connect=${KAFKA_ZOOKEEPER_CONNECT} \
    --override listeners=${KAFKA_LISTENERS} \
    --override advertised.listeners=${KAFKA_ADVERTISED_LISTENERS} \
    --override listener.security.protocol.map=${KAFKA_LISTENER_SECURITY_PROTOCOL_MAP} \
    --override inter.broker.listener.name=${KAFKA_INTER_BROKER_LISTENER_NAME}

In fact, the entrypoint uses server.properties to configure kafka. If I search log.retention.bytes in that configure, I could find next:

$ docker run --rm -it linuxforhealth/kafka-alpine:2.5.0 cat /opt/lfh/kafka/config/server.properties | grep -B2 log.retention.bytes

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

This implies you could define your own server.properties to set your own value for log.retention.bytes, then mount it to container when start your container, something like next:

-v `pwd`/server.properties:/opt/lfh/kafka/config/server.properties linuxforhealth/kafka-alpine:2.5.0
user_63947220
Related