I am running spark 2.4.4 on a virtual machine and reading kafka topic in batch mode. What kafka consumer configuration can cause this ?
Getting numerous messages like below and spark job is running forever.
20/06/14 09:00:43 INFO SubscriptionState: [Consumer clientId=consumer-spark-kafka-relation-de56e437-85a0-4431-bb02-faca7b3487ab-executor-2, groupId=spark-kafka-relation-de56e437-85a0-4431-bb02-faca7b3487ab-executor]
Seeking to EARLIEST offset of partition topic-12
20/06/14 09:00:44 INFO SubscriptionState: [Consumer clientId=consumer-spark-kafka-relation-de56e437-85a0-4431-bb02-faca7b3487ab-executor-3, groupId=spark-kafka-relation-de56e437-85a0-4431-bb02-faca7b3487ab-executor]
Resetting offset for partition topic-38 to offset 0.
20/06/14 09:00:44 INFO SubscriptionState: [Consumer clientId=consumer-spark-kafka-relation-de56e437-85a0-4431-bb02-faca7b3487ab-executor-3, groupId=spark-kafka-relation-de56e437-85a0-4431-bb02-faca7b3487ab-executor]
Seeking to LATEST offset of partition topic-38
20/06/14 09:00:44 INFO SubscriptionState: [Consumer clientId=consumer-spark-kafka-relation-de56e437-85a0-4431-bb02-faca7b3487ab-executor-2, groupId=spark-kafka-relation-de56e437-85a0-4431-bb02-faca7b3487ab-executor]
Resetting offset for partition topic-12 to offset 0.
20/06/14 09:00:44 INFO SubscriptionState: [Consumer clientId=consumer-spark-kafka-relation-de56e437-85a0-4431-bb02-faca7b3487ab-executor-2, groupId=spark-kafka-relation-de56e437-85a0-4431-bb02-faca7b3487ab-executor]
Seeking to LATEST offset of partition topic-12