In R, why do operations using dplyr take different amounts of time depending on the format of the dataframe (long vs wide)?

Viewed 36

I've noticed depending on the number of variables/features a data set has it might be faster to first pivot_wider (less variables) or pivot_longer (more variables). I'm wondering what this is primarily due to? My guess is a trade off between the grouping operation (longer format) and the binding (wider format) but I was wondering if anyone had more insight. Example below of raising a power to a random value for different groups:

Code (many features):

# 5000 features , 100 observations

df_long = data.frame(grp=rep(1:100, each=5e3),
                     feat=paste0("V",rep(1:5e3, times=100)), 
                     val=rnorm(5e5))

df_wide = df_long %>% group_by(feat) %>% 
  pivot_wider(names_from = feat, 
              values_from = val) %>% ungroup()

system.time({
  df_long %>% 
    group_by(grp) %>% mutate(val=grp^val) 
})

system.time({
  df_wide %>% mutate(across(paste0("V",1:5000), ~grp^.))
})

Results:

   user  system elapsed 
  0.028   0.000   0.028 

   user  system elapsed 
  0.158   0.000   0.158

Code (less features):

# 100 features , 5000 observations

df_long = data.frame(grp=rep(1:5e3, each=100),
                     feat=paste0("V",rep(1:100, times=5e3)), 
                     val=rnorm(5e5))

df_wide = df_long %>% group_by(feat) %>% 
  pivot_wider(names_from = feat, 
              values_from = val) %>% ungroup()

system.time({
  df_long %>% 
    group_by(grp) %>% mutate(val=grp^val) 
})

system.time({
  df_wide %>% mutate(across(paste0("V",1:100), ~grp^.))
})

Results:

   user  system elapsed 
  0.051   0.000   0.050 

  user  system elapsed 
 0.025   0.000   0.025 
0 Answers
Related