Bitnami Kafka Property max.request.size always null

user_7209380 0
Viewed 829

I have a spring boot application as kafka producer and consumer and want to send and receive large messages, but I get the following error:

org.apache.kafka.common.errors.RecordTooLargeException: The message is 1717955 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.

I am using the image bitnami/kafka:2.8.0 within a docker-compose file and try to increase the limit by setting the kafka property max.request.size with the following entry in the environment section:

  • KAFKA_CFG_MAX_REQUEST_SIZE=11534336

The problem is, that it does not work. The error appears anyway. The config output of the broker with kafka-configs.sh shows the property, but with the value null:

max.request.size=null sensitive=true synonyms={STATIC_BROKER_CONFIG:max.request.size=null}

Other config properties are set correctly, e.g.:

  • KAFKA_CFG_ZOOKEEPER_CONNECT=172.16.0.25:2181

corresponds to

zookeeper.connect=172.16.0.25:2181 sensitive=false synonyms={STATIC_BROKER_CONFIG:zookeeper.connect=172.16.0.25:2181}

or

  • KAFKA_CFG_LOG_RETENTION_HOURS=672

corresponds to

log.retention.hours=672 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.retention.hours=672, DEFAULT_CONFIG:log.retention.hours=168}

I tried to set other params like:

  • KAFKA_CFG_MESSAGE_MAX_BYTES=11534336
  • KAFKA_CFG_PRODUCER_MAX_REQUEST_SIZE=11534336
  • KAFKA_CFG_CONNECT_PRODUCER_MAX_REQUEST_SIZE=11534336
  • KAFKA_CFG_FETCH_MESSAGE_MAX_BYTES=11534336
  • KAFKA_CFG_SOCKET_REQUEST_MAX_BYTES=11534336

These properties are adopted, but the error occurs anyway and max.request.size is still null. What am I doing wrong?

bitnami docker-compose apache-kafka docker
1 Answers

I am Bitnami engineer. You are not doing anything wrong. Setting up the KAFKA_CFG_MAX_REQUEST_SIZE environment variable, you can verify that the value is properly included inside the broker configuration (/opt/bitnami/kafka/config/server.properties):

max.request.size=11534336 (using the values defined in your env var)

I noticed that the key (max.request.size) are not included properly inside the producer configuration (/opt/bitnami/kafka/config/producer.properties). I included the support to set up properly as we do in server.properties.

The new version is already released. Using the max request size environment variable (KAFKA_CFG_MAX_REQUEST_SIZE), you will see the server.properties and producer.properties files with the values that you are setting up.

Let me know if I can help you with something else

Thanks a lot

Álvaro

user_158607900
Related