During my first tests with BigQuery, I've observed that the size of a table imported in BigQuery is much bigger than its original representation in Hadoop.
Here are the numbers that I get:
- ORC original Hadoop table: 2GB
- Avro compressed representation to load the data into BigQuery: 6.4 GB
- (test: Avro uncompressed: 45.8 GB)
- size in BigQuery (Capacitor format): 47.1 GB
This table has 11 million rows, with 366 columns (most of them being "strings").
Is it a normal behavior from BigQuery? I thought that Capacitor optimized the data in a very efficient way.
Is there any way to see the internal structures of my data in BigQuery to understand what is going wrong and what causes such amount of space being generated?