I've read some resources claiming that Spark read operations are generally lazy. But I've run some jobs that took a long time on the csv read step. Then I read this article saying csv read is an eager operation[1]. Do you have a more definitive answer with reference? Thank you!
1.https://towardsdatascience.com/a-brief-introduction-to-pyspark-ff4284701873
Try minimizing eager operations: In order for your pipeline to be as scalable as possible, it’s good to avoid eager operations that pull full dataframes into memory. I’ve noticed that reading in CSVs is an eager operation, and my work around is to save the dataframe as parquet and then reload it from parquet to build more scalable pipelines.