Delete files older than 10days on HDFS

user_2142994 0
Viewed 33346

Is there a way to delete files older than 10 days on HDFS?

In Linux I would use:

find /path/to/directory/ -type f -mtime +10 -name '*.txt' -execdir rm -- {} \;

Is there a way to do this on HDFS? (Deletion to be done based on file creation date)

hdfs hadoop
7 Answers

I attempted to implement the accepted solution above.

Unfortunately, it only partially worked for me. I ran into 3 real world problems.

First, hdfs didn't have enough RAM to load up and print all the files.

Second, even when hdfs could print all the files awk could only handle ~8300 records before it broke.

Third, the performance was abysmal. When implemented it was deleting ~10 files per minute. This wasn't useful because I was generating ~240 files per minute.

So my final solution was this:

tmpfile=$(mktemp)
HADOOP_CLIENT_OPTS="-Xmx2g" hdfs dfs -ls /path/to/directory    |   tr -s " "    |    cut -d' ' -f6-8    |     grep "^[0-9]"    |    awk 'BEGIN{ MIN=35*24*60; LAST=60*MIN; "date +%s" | getline NOW } { cmd="date -d'\''"$1" "$2"'\'' +%s"; cmd | getline WHEN; DIFF=NOW-WHEN; if(DIFF > LAST){ print $3}}; close(cmd);' > $tmpfile
hdfs dfs -rm -r $(cat $tmpfile)
rm "$tmpfile"

I don't know if there are additional limits on this solution but it handles 50,000+ records in a timely fashion.

EDIT: Interestingly, I ran into this issue again and on the remove, I had to batch my deletes as hdfs rm statement couldn't take more than ~32,000 inputs.

user_128157780 
hdfs dfs -ls -t /file/Path|awk -v dateA="$date" '{if ($6" "$7 < {target_date}) {print ($8)}}'|xargs -I% hdfs dfs -rm "%" /file/Path
user_83742860 
today=`date +'%s'`
days_to_keep=10

# Loop through files
hdfs dfs -ls -R /file/Path/ | while read f; do
  # Get File Date and File Name
  file_date=`echo $f | awk '{print $6}'`
  file_name=`echo $f | awk '{print $8}'`

  # Calculate Days Difference
  difference=$(( ($today - $(date -d $file_date +%s)) / (24 * 60 * 60) ))
  if [ $difference -gt $days_to_keep ]; then
    echo "Deleting $file_name it is older than $days_to_keep and is dated $file_date."
    hdfs dfs -rm -r $file_name
  fi
done
user_86375760

Just to add another variation of previously submitted answers (I do not try to pretend to be original). The script can be modified to remove sub-folders or files recursively

#!/bin/bash
function hdfs-list-older-than () {
# list all content | filter out sub-folders | filter by creation datetime and isolate filepath
  hdfs dfs -ls  $1 | grep ^- | awk -v d=$2 -v t=$3 '{if($6 < d || ($6 == d && $7 < t) ){print $8}}'
}
hdfs-list-older-than $1 `date -d "-10 days"  +'%Y-%m-%d %H:%M'` | xargs hdfs dfs -rm {}
user_105496620
Related