i am trying to setup a simple foreach based continuous streaming using pyspark on kafka and i get in the following log:
21/03/07 15:44:00 INFO AppInfoParser: Kafka version: 2.4.1
21/03/07 15:44:00 INFO AppInfoParser: Kafka commitId: c57222ae8cd7866b
21/03/07 15:44:00 INFO AppInfoParser: Kafka startTimeMs: 1615131840856
21/03/07 15:44:00 INFO KafkaConsumer: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Subscribed to partition(s): telegram-test-0
21/03/07 15:44:00 INFO KafkaConsumer: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Seeking to offset 2 for partition telegram-test-0
21/03/07 15:44:01 INFO ContinuousWriteRDD: Writer for partition 0 in epoch 0 is committing.
21/03/07 15:44:01 INFO ContinuousWriteRDD: Writer for partition 0 in epoch 0 committed.
21/03/07 15:44:01 INFO PythonRunner: Times: total = 582, boot = 394, init = 188, finish = 0
21/03/07 15:44:01 INFO ContinuousWriteRDD: Writer for partition 0 in epoch 1 is committing.
21/03/07 15:44:01 INFO ContinuousWriteRDD: Writer for partition 0 in epoch 1 committed.
21/03/07 15:44:01 INFO Metadata: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Cluster ID: 22a3Ek5jSY2Ja1P8zcdcLQ
21/03/07 15:44:01 INFO SubscriptionState: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Seeking to EARLIEST offset of partition telegram-test-0
21/03/07 15:44:02 INFO SubscriptionState: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Resetting offset for partition telegram-test-0 to offset 0.
21/03/07 15:44:02 INFO SubscriptionState: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Seeking to LATEST offset of partition telegram-test-0
21/03/07 15:44:02 INFO SubscriptionState: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Resetting offset for partition telegram-test-0 to offset 2.
21/03/07 15:44:02 WARN KafkaDataConsumer: KafkaDataConsumer is not running in UninterruptibleThread. It may hang when KafkaDataConsumer's methods are interrupted because of KAFKA-1894
21/03/07 15:44:02 INFO SubscriptionState: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Seeking to EARLIEST offset of partition telegram-test-0
21/03/07 15:44:02 INFO SubscriptionState: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Resetting offset for partition telegram-test-0 to offset 0.
21/03/07 15:44:02 INFO SubscriptionState: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Seeking to LATEST offset of partition telegram-test-0
21/03/07 15:44:02 INFO SubscriptionState: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Resetting offset for partition telegram-test-0 to offset 2.
21/03/07 15:44:02 WARN KafkaDataConsumer: KafkaDataConsumer is not running in UninterruptibleThread. It may hang when KafkaDataConsumer's methods are interrupted because of KAFKA-1894
21/03/07 15:44:02 INFO KafkaConsumer: [Consumer clientId=consumer-spark-group-1-1, groupId=spark-group-1] Seeking to offset 2 for partition telegram-test-0
it seems that the kafka readerStream is resetting the offset bouncing from the latest to the earliset ... and this unfortunately makes the messages never be properly read in the foreach() script.
Note: the exact same program is working fine in micro-batch mode ...