I'm fairly new to using Hadoop in production. I used scoop to bring in a large table from a database into Hive. Scoop created a comma delimited text file and created the corresponding table in Hive.
I then executed a create table new_table_orc stored as orc as select * from old_table_csv
Since a text file is as about as inefficient as can be compared to ORC (binary data, column wise data storage for fat tables, compression, etc.), I expected a huge, orders of magnitude improvement but the query execution time doesn't seem to have changed at all!
I used the same simple query on both version (text, ORC and even parquet) and did the same thin when several of these tables were used in a join.
Additional info: The main table I'm testing has around 430 million rows and around 50 columns.
I'm running a couple of queries:
select sum(col1) from my_table; <= 40 sec
select sum(col1) from my_table_orc; <= 31 sec
And
select distinct col2 from my_table where col3 = someval; <= 53 sec
select distinct col2 from my_table_orc where col3 = someval; <= 35 sec
I also enabled vectorization, as @sahil desai suggested but does seem to have made a huge different (it did reduce the time by a couple of seconds).
What is going on here, why am I not seeing orders of magnitude speedup? What more detail do you need?