I have noticed that the output to the following code spark.read.format("csv").option("header",True).schema(schema).load(path).limit(nrows).rdd.getNumPartitions() is always 1 notwithstanding the argument nrows. Does using limit clause always result in only 1 partition?
I haven't been able to find anything mentioned about such constraint
I don't understand the background of this. But for this statement, the result will always be 1 as you tried to collect result.
>>> df3.limit(1000000).explain()
== Physical Plan ==
CollectLimit 1000000
+- FileScan csv
But for LimitExec which is physical leaf node for limit operation. It has several implementation such as GlobalLimitExec, LocalLimitExec. For some implementations, it could use same partition num to parrellel fetch enough records, or pick subsets of partitions iteratively. Generally, you don't need to care how is doing.