What's the advantage of using multiple columns than just a single column in Apache Arrow?

hexiecs 0
Viewed 31

In Apache Arrow, for example, I have a data which looks like:

struct TabularData {
   int A;
   int B;
   int C;
}

One way is to set three corresponding columns and store the data in A column, B column and C column. Another way is to set just a column and set this struct TabularData as a whole data into this column.
Since Arrow is zero-copy in memory, we don't need any memory copy when we need to read data of A column no matter whether it's in a single column or in a nested message.
My question is: What's the advantage of having multiple columns than just setting all data into one column?

apache-arrow
1 Answers

There isn't exactly a right answer and I don't think anyone can say one is superior to the other. Use what makes the most sense for your problem.

From a storage and I/O perspective there isn't going to be a whole lot of difference between the two representations. They will take up the same amount of space for the data. The struct representation typically has a few more bytes of implementation-specific metadata to keep track of everything.

From a functionality perspective there are probably currently a few advantages to storing things as multiple columns because new functionality doesn't always fully support nested structures but these advantages will hopefully shrink over time as implementations round out their support for nested structures. For example, parquet is (I'm fairly certain) capable of storing leaf statistics (e.g. statistics for A, B, and C, even if stored as a struct) but Arrow (at least Arrow-C++) only uses the top-level statistics at the moment for predicate pushdown so you wouldn't be able to use this feature if you had a filter on A or B or C and the data was stored as a struct (though in this case this is just a (often minor) performance optimization and you can live without it).

Pace0
Related