Can someone explain to me which method is most efficient when checkpointing very large data frames in pyspark?
Here it says that Non Eager checkpointing keeps the lineage and will only trigger the checkpoint when it comes to it in the code. Eager checkpointing triggers it immediately so one would have to wait a long time before it is used.
For a very large dataset where checkpointing needs to be done iteratively, would it make sense to use Non Eager?