Specifying Apache Arrow partitioning from file names in R

user_5558188 0
Viewed 83

I am learning Apache Arrow with R and I am trying to get a better understanding of the partitioning mechanisms.

I have a folder with more than 5 000 CSV files that have that structure: cci_v4_2004106_ppz_takuvik_above_45n.csv

library(arrow)
#> 
#> Attaching package: 'arrow'
#> The following object is masked from 'package:utils':
#> 
#>     timestamp

ds <- open_dataset("~/Desktop/ppz/", format = "csv")

ds
#> FileSystemDataset with 5824 csv files
#> longitude: double
#> latitude: double
#> primary_production: double

Is it possible to infer the partitioning from the filenames? For instance, I would like to use something like:

# ds <- open_dataset("~/Desktop/ppz/", format = "csv", partitioning = c("year", "yday"))

Can I define a schema such as partitioning = c("year", "yday") takes the values from the filename (here: year = 2004, yday = 106).

Created on 2022-03-25 by the reprex package (v2.0.1)

apache-arrow r
1 Answers

No, this feature does not exist today. Partitioning is currently limited to directories:

/2014/106/ppz_takuvik_above_45n.csv           // "directory" flavor
/year=2014/yday=106/ppz_takuvik_above_45n.csv // "hive" flavor

There is an open JIRA ticket for filename based partitioning and there is a PR under review for this so I expect it will become available in 8.0.0.

However, that ticket won't do exactly what you are describing. It will allow you to treat each underscore-separated element as a column but it won't split the 2014106 column into year/yday. If you could change your filenames to 2014_106 on the other hand I think you would be ok.

user_2026940
Related