Why planInputPartitions is called multiple times in a micro-batch?

Viewed 120

I'm developing a new Spark connector using data source v2 API (spark 3.1.1). I noticed that the planInputPartitions method (in MicroBatchStream) is called twice every micro-batch.

What the motivation/reason is?

0 Answers
Related