How do count unique entities with disk.frame in R?

user_11117255 0
Viewed 171

I'd like to convert a data frame to a disk frame and then count the first column. It's not counting the number of unique values of the column when I try it. It appears to be counting the number of workers.

library(disk.frame)
options(future.globals.maxSize = Inf)
setup_disk.frame(workers = 8)

This is an example dataset

    bigint <- sample(123901239804:901283455390, 3*10^5)
    df <- data.frame(bigint)
    df %>% 
      summarize(ints = length(unique(bigint)))
    
    df %>% 
      as.disk.frame %>%
      summarize(ints = length(bigint)) %>% 
      collect

In the first query, it gets me this output

    ints
1 300000

In the second query, it gets me this output

    ints
1      8
disk.frame r
1 Answers

{disk.frame} only supports some group-by functions. You can use dplyr::n_distinct

df %>% 
  as.disk.frame %>%
  summarize(ints = n_distinct(bigint)) %>% 
  collect

which yields

    ints
1 300000

See the list of supported group-by verbs here https://diskframe.com/articles/10-group-by.html#list-of-supported-group-by-functions

You can define more customised group-by verbs by following this guide

https://diskframe.com/articles/11-custom-group-by.html

Certain groups-by verbs are not possible to be done in an exact way (and has to rely on estimates) due to the chunking-nature of disk.frame. But that is true of all "big" data systems.

user_2399230
Related