I've noticed depending on the number of variables/features a data set has it might be faster to first pivot_wider (less variables) or pivot_longer (more variables). I'm wondering what this is primarily due to? My guess is a trade off between the grouping operation (longer format) and the binding (wider format) but I was wondering if anyone had more insight. Example below of raising a power to a random value for different groups:
Code (many features):
# 5000 features , 100 observations
df_long = data.frame(grp=rep(1:100, each=5e3),
feat=paste0("V",rep(1:5e3, times=100)),
val=rnorm(5e5))
df_wide = df_long %>% group_by(feat) %>%
pivot_wider(names_from = feat,
values_from = val) %>% ungroup()
system.time({
df_long %>%
group_by(grp) %>% mutate(val=grp^val)
})
system.time({
df_wide %>% mutate(across(paste0("V",1:5000), ~grp^.))
})
Results:
user system elapsed
0.028 0.000 0.028
user system elapsed
0.158 0.000 0.158
Code (less features):
# 100 features , 5000 observations
df_long = data.frame(grp=rep(1:5e3, each=100),
feat=paste0("V",rep(1:100, times=5e3)),
val=rnorm(5e5))
df_wide = df_long %>% group_by(feat) %>%
pivot_wider(names_from = feat,
values_from = val) %>% ungroup()
system.time({
df_long %>%
group_by(grp) %>% mutate(val=grp^val)
})
system.time({
df_wide %>% mutate(across(paste0("V",1:100), ~grp^.))
})
Results:
user system elapsed
0.051 0.000 0.050
user system elapsed
0.025 0.000 0.025