We have spark streaming from kafka creating checkpoints in the HDFS server and its not getting cleaned up , Now we have millions of checkpoints in the HDFS. Is there a way to clean it automatically from spark ?
Spark Version 1.6 HDFS 2.70
1 Answers
Related