Speed up/alternative to for-loop that refers to prior calculated row

user_15319040 0
Viewed 44

I have a bit of code that needs to build a column that looks at the previous row and sums until hitting a certain point at which it returns to 0 and starts again.

Currently this is done with a for-loop, however one of the datasets that I want to use this to process is 3 million rows and takes days to process with this method. To be useable for my purposes it needs to process in seconds.

Is there a way to speed up or replace the for-loop so this can process faster? (lag() and the like can't be used as they use the pre-calculation data)

DD <- c("Yes", "No", "No", "Yes", "No", "No", "No", "No", "No", "No", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "No", "Yes", "Yes", "Yes", "Yes", "Yes", "No", "No", "No")

dataOut <- data.frame(DD)

alarm.upper.limit <- 7.26
alarm.lower.limit <- -7.26

s <- 0.11

dataOut$weight <- if_else(dataOut$DD=="Yes", 1-s, 0-s)

dataOut$cusum[1] <- dataOut$weight[1]

for (j in 2:nrow(dataOut)) {
  dataOut$cusum[j] <- if(dataOut$cusum[j-1]<alarm.lower.limit){dataOut$cusum[j]=0}
  else if(dataOut$cusum[j-1]>alarm.upper.limit){dataOut$cusum[j]=0}
  else{(dataOut$weight[j])+(dataOut$cusum[j-1])}
}
r for-loop performance
1 Answers

We could use accumulate and it should be faster than the for loop operation

library(dplyr)
library(purrr)
dataOut %>%
   mutate(cusum2 = accumulate(weight, ~
     if(between(.x, alarm.lower.limit, alarm.upper.limit)) .x + .y else 0))

Or use Reduce in base R

dataOut$cusum3 <- with(dataOut, Reduce(function(x, y)
  if(x >= alarm.lower.limit & x <= alarm.upper.limit) x + y 
       else 0, weight, accumulate = TRUE))

On a slightly bigger data

dataOut1 <- dataOut[rep(seq_len(nrow(dataOut)), 1e3), ]
 
 
system.time({
 dataOut1$cusum[1] <- dataOut1$weight[1]

 for (j in 2:nrow(dataOut1)) {
   dataOut1$cusum[j] <- if(dataOut1$cusum[j-1]<alarm.lower.limit){
          dataOut1$cusum[j]=0}
   else if(dataOut1$cusum[j-1]>alarm.upper.limit){
         dataOut1$cusum[j]=0}
   else{
         (dataOut1$weight[j])+(dataOut1$cusum[j-1])}
 }
 
 })
#  user  system elapsed 
#  0.902   0.758   1.617 

system.time({
   dataOut1 <-  dataOut1 %>%
              mutate(cusum2 = accumulate(weight, 
   ~ if(between(.x, alarm.lower.limit, alarm.upper.limit)) .x + .y else 0))

})

#user  system elapsed 
#  0.110   0.005   0.116
user_37322710
Related