read multiple parquet file at once in pyspark

user_13126794 0
Viewed 3457

I have multiple parquet files categorised by id something like this:

/user/desktop/id=1x/year=2020/month=8/day=12/file1.parquet 
/user/desktop/id=2x/year=2020/month=8/day=15/file2.parquet 
/user/desktop/id=3x/year=2020/month=9/day=11/file3.parquet 
/user/desktop/id=4x/year=2020/month=8/day=22/file4.parquet

I have a python list which holds all the id value something like this:

id_list = ['1x','2x','3x']

I want to read all the files at once for ids present inside in id_list and also I want to read files which corresponds to month=8 So, for this example only file1 and file2 should be read.

I am doing like this:

sub_path = '/*/*/*/*.parquet'
input_df = sqlContext.read.parquet('/user/desktop/' + 'id={}'.format(*id_list) + sub_path)

This is picking only the file inside first id of the id_list which is id='1x'. Can anyone please help me what I am missing here?

pyspark apache-spark-sql apache-spark
2 Answers

You can do this by :

id_list = ['1x','2x','3x']
input_df = sqlContext.read.parquet('/user/desktop/').filter(col('id').isin(id_list))

While using the filter operation, since Spark does lazy evaluation you should have no problems with the size of the data set. The filter will be applied before any actions and only the data you are interested in will be kept in memory, thus reading only required all data or files into the memory for the IDs specified.

user_82715490

This will exactly load those path which have id specified in id_list and month=8

id_list = ['1x','2x','3x']
path = "/user/desktop/id={id}/year=2020/month=8/*/*"
paths = list(map(lambda idx: path.format(idx), id_list))
spark =  SparkSession.builder.appName("sample_app").getOrCreate()
df = spark.read.format("parquet").load(paths)
user_52516650
Related