I was asked a question in an interview that what if while writing a 100GB dataset to a disc, the application crashes after writing 99 GB of data. How can we avoid writing the entire 100GB of data again on restart? I gave an answer of checkpointing but the interviewer wasn't happy with it. Creating RDD and comparing for delta for what is already written, doesn't seem optimized either. Is there a better way?
0 Answers
Related