I have a question about performance with pushdown filters. I have data stored in parquet S3 under
s3://mybucket/data
it has 'lat', 'lon' and 'qk' columns. There are about 750 files. Sorted by qk7 column (min value = 0230001 max value = 0233333).
When I do a simple query with spark:
path = "s3://mybucket/data"
prod_df = spark.read.load(path)
# not existed value in column
prod_df.where("""qk='0000000'""").count()
I expect a very quick spark lookup for statistics in each parquet header and 0 results since this value does not exist. However, it takes more time than scanning "real values":
path = "s3://mybucket/data"
prod_df = spark.read.load(path)
# real value that occures ~75000 times
prod_df.where("""qk='0230101'""").count()
What could be the root cause for this? I am not expecting such results