R: fastest way to summarize by group?

user_11825717 0
Viewed 549

What is the absolute fastest way to summarize by group in R? I've used data.table to optimize this step as much as I can but it's still the main bottleneck in my code as it has to run thousands of times.

library(data.table)
data <- matrix(rnorm(5e6 * 16), ncol = 16)
colnames(data) <- paste0("mark", 1:16)
group <- gl(10, 5e5, labels = paste0("sample", 1:10)) 
DT <- data.table(group, data) # 1/10 actual row #
out <- DT[, lapply(.SD, function(x) {mean(x^3)}), by = group]
data.table r optimization performance
1 Answers

As r2evans mentions, the mean function is not the slowest part. It is the power function x^3 for all data.

We can see this if we separate the calls and measure time.

system.time(x <- lapply(seq_along(DT)[-1], function(i) DT[[i]]^3)) # 4.7
system.time(setDT(x)) # 0
system.time(x[, lapply(.SD, mean), by = DT$group]) # 0.41

In this specific case I can propose:

v2 <- function() {
  x <- lapply(seq_along(DT)[-1], function(i) DT[[i]]*DT[[i]]*DT[[i]])
  setDT(x)
  x[, lapply(.SD, mean), by = DT$group]
}

timing:

v1 <- function() {
  DT[, lapply(.SD, function(x) {mean(x^3)}), by = group]
}
system.time(v1()) # 4.92 
system.time(v2()) # 0.84

Also,

x[, lapply(.SD, mean), by = DT$group]
x[, lapply(.SD, function(i) mean(i)), by = DT$group]

are different. First one invokes data.tables gmean, but the second call doesn't. Depending on the size of your data one could be faster then the other approach.

user_70633750
Related