Iterate through columns to sum the previous 2 numbers of each row

Viewed 757

In R, I have a dataframe, with columns 'A', 'B', 'C', 'D'. The columns have 100 rows.

I need to iterate through the columns to perform a calculation for all rows in the dataframe which sums the previous 2 rows of that column, and then set in new columns ('AA', 'AB', etc) what that sum is:

A  B  C  D 
1  2  3  4
2  3  4  5
3  4  5  6
4  5  6  7 
5  6  7  8
6  7  8  9

to

A   B   C   D   AA   AB   AC   AD
1   2   3   4   NA   NA   NA   NA
2   3   4   5   3    5    7    9
3   4   5   6   5    7    9    11
4   5   6   7   7    9    11   13
5   6   7   8   9    11   13   15
6   7   8   9   11   13   15   17

Can someone explain how to create a function/loop that allows me to set the columns I want to iterate over (selected columns, not all columns) and the columns I want to set?

5 Answers

A base one-liner:

cbind(df, setNames(df + df[c(NA, 1:(nrow(df)-1)), ], paste0("A", names(df))))

If your data is large, this one might be the fastest because it manipulates the entire data.frame.


A dplyr solution using mutate() with across().

library(dplyr)

df %>%
  mutate(across(A:D,
                ~ .x + lag(.x),
                .names = "A{col}"))

#   A B C D AA AB AC AD
# 1 1 2 3 4 NA NA NA NA
# 2 2 3 4 5  3  5  7  9
# 3 3 4 5 6  5  7  9 11
# 4 4 5 6 7  7  9 11 13
# 5 5 6 7 8  9 11 13 15
# 6 6 7 8 9 11 13 15 17

If you want to sum the previous 3 rows, the second argument of across(), i.e. .fns, should be

~ .x + lag(.x) + lag(.x, 2)

which is equivalent to the use of rollsum() in zoo:

~ zoo::rollsum(.x, k = 3, fill = NA, align = 'right')

Benchmark

A benchmark test with microbenchmark package on a new data.frame with 10000 rows and 100 columns and evaluate each expression for 10 times.

# Unit: milliseconds
#                     expr        min         lq       mean     median         uq        max neval
#              darren_base   18.58418   20.88498   35.51341   33.64953   39.31909   80.24725    10
#         darren_dplyr_lag   39.49278   40.27038   47.26449   42.89170   43.20267   76.72435    10
# arg0naut91_dplyr_rollsum  436.22503  482.03199  524.54800  516.81706  534.94317  677.64242    10
#    Grothendieck_rollsumr 3423.92097 3611.01573 3650.16656 3622.50895 3689.26404 4060.98054    10

You can use dplyr's across (and set optional names) with rolling sum (as implemented e.g. in zoo):

library(dplyr)
library(zoo)

df %>%
  mutate(
    across(
      A:D,
      ~ rollsum(., k = 2, fill = NA, align = 'right'), 
      .names = 'A{col}'
    )
  )

Output:

  A B C D AA AB AC AD
1 1 2 3 4 NA NA NA NA
2 2 3 4 5  3  5  7  9
3 3 4 5 6  5  7  9 11
4 4 5 6 7  7  9 11 13
5 5 6 7 8  9 11 13 15
6 6 7 8 9 11 13 15 17

With A:D we've specified the range of column names we want to apply the function to. The assumption above in .names argument is that you want to paste together A as prefix and the column name ({col}).

That is a naive approach with nested for loops. Beware it is damn slow if you gonna iterate over hundreds thousand rows.

i <- 1
n <- 5
df <- data.frame(A=i:(i+n), B=(i+1):(i+n+1), C=(i+2):(i+n+2), D=(i+3):(i+n+3))
for (col in colnames(df)) {
  for (ind in 1:nrow(df)) {
    if (ind-1==0) {next}
    s <- sum(df[c(ind-1, ind), col])
    df[ind, paste0('S', col)] <- s
  }
}  

That is a cumsum method:

na.df <- data.frame(matrix(NA, 2, ncol(df)))
colnames(na.df) <- colnames(df)
cs1 <- cumsum(df)
cs2 <- rbind(cs1[-1:-2,], na.df)
sum.diff <- cs2-cs1
cbind(df, rbind(na.df[1,], cs1[2,], sum.diff[1:(nrow(sum.diff)-2),]))  

Benchmark:

#    Unit: milliseconds  
#                      expr     min       lq     mean   median       uq     max neval  
#          darrentsai.rbind 11.5623 12.28025 23.38038 16.78240 20.83420 91.9135   100  
#     darrentsai.rbind.rev1  8.8267  9.10945 15.63652  9.54215 14.25090 62.6949   100  
#             pseudopsin.dt  7.2696  7.52080 20.26473 12.61465 17.61465 69.0110   100  
#            ivan866.cumsum 25.3706 30.98860 43.11623 33.78775 37.36950 91.6032   100  

I believe, most of the time the cumsum method wastes on df allocations. If correctly adapted to data.table backend, it could be the fastest.

Here's a data.table solution. As you ask for, it allows you to select which columns you want to apply it to rather than just for all columns.

library(data.table)
x <- data.table(A=1:6, B=2:7, C=3:8, D=4:9)
selected_cols <- c('A','B','D')
new_cols <- paste0("A",selected_cols)
x[, (new_cols) := lapply(.SD, function(col) col+shift(col, 1)), .SDcols = selected_cols]
x[]

NB This is 2 or 3 times faster than the fastest other answer.

Specify the columns we want. We show several different ways to do that. Then use rollsumr to get the desired columns, set the column names and cbind DF with it.

library(zoo)

# jx <- names(DF)  # if all columns wanted
# jx <- sapply(DF, is.numeric)  # if all numeric columns
# jx <- c("A", "B", "C", "D")  # specify columns by name
jx <- 1:4   # specify columns by position

r <- rollsumr(DF[jx], 2, fill = NA)
colnames(r) <- paste0("A", colnames(r))
cbind(DF, r)

giving:

  A B C D AA AB AC AD
1 1 2 3 4 NA NA NA NA
2 2 3 4 5  3  5  7  9
3 3 4 5 6  5  7  9 11
4 4 5 6 7  7  9 11 13
5 5 6 7 8  9 11 13 15
6 6 7 8 9 11 13 15 17

Note

The input in reproducible form:

DF <- structure(list(A = 1:6, B = 2:7, C = 3:8, D = 4:9), 
  class = "data.frame", row.names = c(NA, -6L))
Related