I'm trying to open a hive-partitioned parquet, which is essentially a nested directory with many little parquet fragments at the bottom level. When working with Python (using pyarrow), I can just use read_table on the top directory that has a .parquet at the end of the name, and everything is handled automatically. If I'm working with just a single file in C++, I can use a std::shared_ptr<arrow::io::ReadableFile> instance to read from it, but of course it does not work on a directory.
Basically, I want to find a set of tools with the starting point marked as ? below:
Python:
open single file -> pyarrow.parquet.read_table
open directory (partitioned dataset) -> pyarrow.parquet.ParquetDataset
C++:
open single file -> std::shared_ptr<arrow::io::ReadableFile>
open directory (partitioned dataset) -> ?