Is there any way to read the last 3 hours of files in a Spark dataframe?
On HDFS I have a directory structure like:
/user/hdfs/test/partition_date=2022-09-14/hour=01/fil1.csv
/user/hdfs/test/partition_date=2022-09-14/hour=01/fil2.csv
/user/hdfs/test/partition_date=2022-09-14/hour=01/fil3.csv
/user/hdfs/test/partition_date=2022-09-14/hour=02/fil1.csv
/user/hdfs/test/partition_date=2022-09-14/hour=02/fil2.csv
/user/hdfs/test/partition_date=2022-09-14/hour=02/fil3.csv
/user/hdfs/test/partition_date=2022-09-14/hour=03/file1.csv
/user/hdfs/test/partition_date=2022-09-14/hour=03/file2.csv
/user/hdfs/test/partition_date=2022-09-14/hour=04/file1.csv
Now I want to read the data only Last 3 hrs. So basically below 3 files should be loaded in spark.
/user/hdfs/test/partition_date=2022-09-14/hour=03/file1.csv
/user/hdfs/test/partition_date=2022-09-14/hour=03/file2.csv
/user/hdfs/test/partition_date=2022-09-14/hour=04/file1.csv
Can Someone Pla guide me to read these folders data.