python kafka producer produces duplicated records

k31 0
Viewed 27

i am faced with very strange behavior of python kafka producer:

i have a code like this:

self.producer = KafkaProducer(bootstrap_servers=['kafka:9092'],
                                value_serializer=lambda x: xml_serializer.render(x).encode()
                             )

and i send xml messages

my_time: int = 12345 ... 
resp: SomeObject = SomeObject(1,2,3)
self.producer.send(topic='xxx-name', value=resp, timestamp_ms=my_time)

but unexpectedly if I observe the content of kafka topic using any ui client (i tried couple of them). then I see that topic contains duplicated records with exactly the same content.

what is also strange is timestamp info of such duplicates

in example above "my_time" variable takes values like 12:00:00, 12:00:05,

obviously such message sending happens not exactly at :00 seconds every 5 secs but with some small delay (say 100-500 ms)

but then in ui kafka client I see

timestamp   partition   offset  value   key
1662403605522   0   881 "<?xml version=\"1.0\" ...>"    null
1662403605000   0   882 "<?xml version=\"1.0\" ...>"    null

you may also notice that offset value grows but timestamp of the records is smaller, what I think comes from the fact that kafka producer first sends message without timestamp (current time value) and then send the same message (duplicate) using now timestamp that I provided (from variable my_time)

apache-kafka python
1 Answers

well as wisdom has it: sleep on it
in the morning things got more clear: obviously kafka-python library was not guilty: I had two python processes running in parallel - one in pycharm (debug mode) and second in docker container which i didn't see and forgot about

k310
Related