Grep across multiple files in Hadoop Filesystem

user_663148 0
Viewed 44553

I am working with Hadoop and I need to find which of ~100 files in my Hadoop filesystem contain a certain string.

I can see the files I wish to search like this:

bash-3.00$ hadoop fs -ls /apps/mdhi-technology/b_dps/real-time

..which returns several entries like this:

-rw-r--r--   3 b_dps mdhi-technology 1073741824 2012-07-18 22:50 /apps/mdhi-technology/b_dps/HADOOP_consolidated_RT_v1x0_20120716_aa
-rw-r--r--   3 b_dps mdhi-technology 1073741824 2012-07-18 22:50 /apps/mdhi-technology/b_dps/HADOOP_consolidated_RT_v1x0_20120716_ab

How do I find which of these contains the string bcd4bc3e1380a56108f486a4fffbc8dc? Once I know, I can edit them manually.

hadoop grep shell unix bash
5 Answers

You are looking to applying grep command on hdfs folder

hdfs dfs -cat /user/coupons/input/201807160000/* | grep -c null

here cat recursively goes through all files in the folder and I have applied grep to find count.

user_63630230

To find all files with any extension recursively inside hdfs location:

hadoop fs -find  hdfs_loc_path  -name ".log"
user_116822700 
hadoop fs -find /apps/mdhi-technology/b_dps/real-time  -name "*bcd4bc3e1380a56108f486a4fffbc8dc*"

hadoop fs -find /apps/mdhi-technology/b_dps/real-time  -name "bcd4bc3e1380a56108f486a4fffbc8dc"
user_61729790
Related