Apache arrow, R. FileSystemDataset, how to tag which file a record came from?

Question

library(arrow, warn.conflicts = FALSE) # 6.0.1
library(dplyr, warn.conflicts = FALSE) # 0.8.5
# Build data
ytd_snapshot <- data.frame(id = 1:5, date = seq.Date(from = as.Date("2022-05-01"), as.Date("2022-05-05"), by = "1 day"), value = 1:5)
tdy_snapshot <- data.frame(id = 1:6, date = seq.Date(from = as.Date("2022-05-01"), as.Date("2022-05-06"), by = "1 day"), value = c(2,2:6))

# Caching it in yyyy/mm structure
two_level_tree <- tempfile()
dir.create(file.path(two_level_tree, 2022, 05), recursive = TRUE)
arrow::write_feather(ytd_snapshot, sink = file.path(two_level_tree, 2022, 05, "snapshot_20220506.feather"))
arrow::write_feather(tdy_snapshot, sink = file.path(two_level_tree, 2022, 05, "snapshot_20220507.feather"))
list.files(two_level_tree, recursive = TRUE)

[1] "2022/5/snapshot_20220506.feather" "2022/5/snapshot_20220507.feather"

ds <- open_dataset(sources = c(file.path(two_level_tree, 2022, 05, "snapshot_20220506.feather"), 
                               file.path(two_level_tree, 2022, 05, "snapshot_20220507.feather")), unify_schemas = FALSE, format = "feather")
# Actual
ds %>% collect()

   id       date value
1   1 2022-05-01     1
2   2 2022-05-02     2
3   3 2022-05-03     3
4   4 2022-05-04     4
5   5 2022-05-05     5
6   1 2022-05-01     2
7   2 2022-05-02     2
8   3 2022-05-03     3
9   4 2022-05-04     4
10  5 2022-05-05     5
11  6 2022-05-06     6

# Desired
rbind(cbind(ytd_snapshot, data.frame(file=file.path(two_level_tree, 2022, 05, "snapshot_20220506.feather"))),
      cbind(tdy_snapshot, data.frame(file=file.path(two_level_tree, 2022, 05, "snapshot_20220507.feather"))))

   id       date value                                                                                                        file
1   1 2022-05-01     1 \AppData\Local\Temp\1\RtmpU1FX5C\file54c85b3c1513/2022/5/snapshot_20220506.feather
2   2 2022-05-02     2 \AppData\Local\Temp\1\RtmpU1FX5C\file54c85b3c1513/2022/5/snapshot_20220506.feather
3   3 2022-05-03     3 \AppData\Local\Temp\1\RtmpU1FX5C\file54c85b3c1513/2022/5/snapshot_20220506.feather
4   4 2022-05-04     4 \AppData\Local\Temp\1\RtmpU1FX5C\file54c85b3c1513/2022/5/snapshot_20220506.feather
5   5 2022-05-05     5 \AppData\Local\Temp\1\RtmpU1FX5C\file54c85b3c1513/2022/5/snapshot_20220506.feather
6   1 2022-05-01     2 \AppData\Local\Temp\1\RtmpU1FX5C\file54c85b3c1513/2022/5/snapshot_20220507.feather
7   2 2022-05-02     2 \AppData\Local\Temp\1\RtmpU1FX5C\file54c85b3c1513/2022/5/snapshot_20220507.feather
8   3 2022-05-03     3 \AppData\Local\Temp\1\RtmpU1FX5C\file54c85b3c1513/2022/5/snapshot_20220507.feather
9   4 2022-05-04     4 \AppData\Local\Temp\1\RtmpU1FX5C\file54c85b3c1513/2022/5/snapshot_20220507.feather
10  5 2022-05-05     5 \AppData\Local\Temp\1\RtmpU1FX5C\file54c85b3c1513/2022/5/snapshot_20220507.feather
11  6 2022-05-06     6 \AppData\Local\Temp\1\RtmpU1FX5C\file54c85b3c1513/2022/5/snapshot_20220507.feather

I'm trying to identifying a record to it's original file source. I know if we added what we want into the data.frame before we save it, life would be much easier. That said, this is more of a what-if scenario.

user_1141050 · Answer

This isn't currently a feature implemented in the Arrow R package, though it exists in the C++ implementation on which the R package is based and there is ongoing work looking at implementing it the R package - see JIRA ticket ARROW-15260.

In the meantime, as a workaround, you could use this example (from that ticket) which is a very off-label (i.e. not recommended as a long-term solution as the APIs might change) usage of the Scanner API:

library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)

tf <- tempfile()
write_dataset(mtcars, tf, partitioning = "cyl")
ds <- open_dataset(tf)

# works!
scanner <- Scanner$create(
  open_dataset(tf), 
  projection = c("__filename", names(ds))
)

as_tibble(scanner$ToTable())

Apache arrow, R. FileSystemDataset, how to tag which file a record came from?

1 Answers