KafkaProducer for Python sending message if done on command line but not if done through a Python script

user_6446511 0
Viewed 3597

I've got my zookeeper and broker running.

When I send a message from the command line like so:

from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('topic', b'some_message_bytes')

I'm able to receive it on the other end in my consumer running this:

from kafka import KafkaConsumer
consumer = KafkaConsumer('topic')
for msg in consumer:
    print msg

However, when I include the exact same code for the producer inside of a python script and run it, the consumer doesn't print the message and furthermore, it does not even get sent. I confirmed this by running this on my command line:

./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
localhost:9092 --topic topic --time -1 --offsets 1 | awk -F  ":" '{sum +=
 $3} END {print sum}'

This shows the number of messages in the current topic. It goes up as I send more messages from the command line-version of the python producer, but the number of messages in the topic does not get incremented after running the python script.

Does anyone know what's going on here?

apache-kafka python
1 Answers

Running a program in shell vs running it in a file is different. In shell when you send the message it's sent with some fixed flush time in Kafka and Kafka producer instance is still alive in the shell. When running the code in a text file, even before the message can be flushed the program ends and the producer object is garbage collected.

To fix this in producer code use producer.flush()

from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('topic', b'some_message_bytes')
# This is important
producer.flush()
user_44406750
Related