How Spark stores Parquet Table?

Viewed 82

I have a fact table which is 10Tb (Parquet) which contains 100+ columns. When I have created another table with just 10 columns from the fact table and size is 2TB.

  1. I was expecting the size should be in some GBs because I am storing just few (10) columns?
  2. My question is when we have more columns does Parquet format stores in more efficient manner?
1 Answers

Parquet is a column based storage. Say if I have a table with fields userId, name, address, state, phone no. In non-parquet storage If I do a select * where state = "TN" it will go through every record in my table (i.e all the columns of each row) and output the records that match my where condition. However in parquet format all the columns are stored together so I don't need to go through all the other columns. The same select query will directly go to column 'state' and output records that match the where condition. Parquet is good for faster retrieval (to get results faster). It doesn't matter how many columns are present in total.

Parquet uses snappy compression. Since all the columns are stored together it makes compression very effective.

Related