As we all know HDFS is basically an Object storage system where in data once written into it can't be updated but the entire file has to be overwritten as a whole. If this is the case then what is the need of introducing ORC format to HDFS which allows us to modify / update the data stored in hive tables whose underlying storage system is HDFS.
This makes HDFS a block storage where in the data can be updated / modified which is contradictory to the entire Hadoop framework
ORC's (limited) support for ACID transactions does not change the fact that HDFS is write-only. See https://orc.apache.org/docs/acid.html for an explanation. Updates in ORC are simply written to new files in HDFS, and so writing individual transactions quickly will result in many new files which will result in severe performance issues.
First we should be familiar with the history of ORC and columnar data formats on HDFS. Originally HDFS was designed to be a file system as the name indicates. Meaning the people wanted to store "files" in a scalable way.
Originally ORC and Parquet did not support updates because it does not makes sense in a long term read efficient storage solution to do modifications like updates. The data should be immutable and stored very efficiently. As usual some enterprise companies decided that the customer is always right and introduced modifications. They call this process maturing on the ORC page. I see it as a problematic feature that I do not recommend to my clients.
This makes HDFS a block storage where in the data can be updated / modified which is contradictory to the entire Hadoop framework
This is not the case because the actual implementation is having extra files instead of doing in-place updates.
