For repeating inputs, does group + nest + mutate + unnest work faster than group+mutate?

user_6704562 0
Viewed 169

Let's consider a tibble with repeated arguments like this:

data <- tibble(
        ~x, ~y,
       "A", "17:145:01800",
       "B", "17:145:01800",
       "C", "17:145:01800",
       "A", "17:146:01800",
       "B", "17:146:01800",
       "C", "17:146:01800",
    )

Now I want to create a column z which is the output of a function of y. The "slow" method would be:

data %>% mutate(z=myfunc(y))

However, as the input repeats 3 times in this case (one for each value of x), I consider a grouping approach to avoid calculate the same twice:

# Version 1
data %>%
 group_by(y) %>%
 nest() %>% # now the tibble has only 2 rows
 mutate(z=myfunc(y)) %>%
 unnest(c("data"))
# Version 2
data %>%
 group_by(y) %>%
 mutate(z=myfunc(y))

What should be faster? Version 1 or version 2?

I wonder if grouping would reduce the number of times myfunc is called, or if nesting is necessary for this. I tried to do some testing, but with small number of rows maybe the nesting and unnesting makes results different...I particularly want to apply this to a tibble with a lot of different x's (~350) and different y's (~9600)

Thanks in advance.

Solution

I found a way to improve a lot the performance. First, run the function only on unique values: dict <- myfunc(unique(y)) . Then, name the vector names(dict) <- unique(y). Then you can get the value by looking at the vector for each input data$z <- dict[data$y]. Or use a mutate() alternative to this. I think it is also possible to use a left_join() approach to this but I havent tried. Results of a microbenchmark I tried on it (oldthing is just using data$z <- myfunc(data$y) and newthing is the approach I just described).

Unit: milliseconds
        expr        min        lq      mean    median        uq        max neval
     nothing  36.434995  57.39253  93.38216  82.91927 109.70960  588.06789   100
       group 156.039489 338.40348 457.83076 426.54913 562.26887 1807.16725   100
 groupunique 112.453367 248.53734 365.68583 327.62761 442.73981 1494.96353   100
    oldthing  34.265350  56.41572  85.71731  73.83113 105.02589  493.63245   100
    newthing   5.147881  10.93754  14.69213  12.67236  14.66174   89.80508   100
dplyr tidyverse r performance
2 Answers

Try to install microbenchmark package:

library(microbenchmark)
library(tidyverse)

# example with your data
microbenchmark(
  data %>%
    group_by(y) %>%
    nest() %>%
    mutate(z=myfunc(y)) %>%
    unnest(c("data")),
  data %>%
    group_by(y) %>%
    mutate(z=myfunc(y))
)

It's input and output is simple like this:

# simple example
microbenchmark(
  iris %>% select(Species),
  iris["Species"],
  iris$Species)
#> Unit: nanoseconds
#>                      expr     min      lq    mean  median      uq     max neval
#>  iris %>% select(Species) 1463000 1484550 1568863 1538400 1598950 2206400   100
#>           iris["Species"]   12700   14700   17865   18550   19450   66100   100
#>              iris$Species     600     850    1725    1800    2000   19700   100

It allows you to test your approaches and kindly teaches how to use R simply and efficiently. Sometimes the simpliest solution can be the fastest, like e.g.

data$z <- my_func(data$y)
Admin1

group_by only provides benefits if you reduce the calculations from running on the full vector, such as by using unique. Your formulation just splits up the calculation to run on a vector y with 3 entries each time, providing no inherent speed up.

However, if we use unique, group_by could provide benefits, but it depends on what my_func is. If it's naturally vectorized, it's likely the performance will be similar to just straight running mutate. However, if it's a complex function on each individual value that's vectorized manually (or some other computationally intensive function), then you could see speed improvements using group_by.

Using nest is never going to improve your performance, because it provides the same reduction as group_by but is computationally expensive in splitting up the data frame. See below a simple example using a vectorized function (paste0), where we essentially see the same performance using a raw mutate and group_by, but where nest is significantly slower.

library(tidyverse)
library(microbenchmark)

set.seed(1)

df <- tibble(x = rep(stringi::stri_rand_strings(300,3), 30),
             y = rep(1:30, each = 300))

my_func <- function(x) {
  paste0("test", x, "_result")
}

microbenchmark(
  "nothing" = {
    mutate(df, z = my_func(y))
  },
  "group" = {
    group_by(df, y) %>%
      mutate(z = my_func(unique(y)))
  },
  "nest" = {
    group_by(df, y) %>%
      nest() %>%
      mutate(z = my_func(y)) %>%
      unnest(c("data"))
  }
)
#> Unit: milliseconds
#>     expr       min        lq      mean    median        uq      max neval
#>  nothing  3.305522  3.502466  4.099286  4.157768  4.358949  6.74135   100
#>    group  3.259249  3.581619  4.486733  4.048356  4.392127 35.09225   100
#>     nest 19.929131 21.896518 25.293476 24.772588 27.267238 40.86496   100
user_103573830
Related