unable to send large files to Kafka broker | Error message: The request included a message larger than the max message size the server will accept

Viewed 526

I am unable to send large file to the broker, I have edited the configuration files and inside my client program but it didn't work.

  • Error Message

    The request included a message larger than the max message size the server will accept.

  • Server properties

    message.max.bytes=900000000
    replica.fetch.max.bytes=900000000
    
  • Producer

    max.request.size=100000000 
    
  • Consumer

    max.partition.fetch.bytes=100000000
    fetch.message.max.bytes=100000000
    
  • Client app

    props.put("max.request.size",96214400);
    props.put("message.max.bytes",96214400);
    
2 Answers

Moving comment to answer...

The message.max.bytes is a cluster-wide setting and you need to make sure that the corresponding max.message.bytes configuration is also set at topic-level. As the topic-level configuration defaults to 1048588 you are seeing this error.

kafka-topics.bat --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic test --config max.message.bytes=9000000
Related