Vectorizing a for loop that changes columns of a matrix

user_19334210 0
Viewed 99

Say I have a vector of ages of 100 trees. Then I age those trees up for 5, 10, 15, and 20 years into the future to create a matrix of tree ages for this year and four 5-year planning periods in the future.

But then, I decide to cut some of those trees (only 10 per planning period), documented in a matrix of T/F values where T is harvested and F is not (trees can't be harvested twice).

age.vec <- sample(x = 1:150, size = 100, replace = T) # create our trees
age.mat <- cbind(age.vec, age.vec+5, age.vec + 10, age.vec + 15, age.vec + 20) # grow them up
x.mat <- matrix(data = F, nrow = 100, ncol = 5) # create the empty harvest matrix
x.mat[cbind(sample(1:100, size = 50), rep(1:5, each = 10))] <-  T # 10 trees/year harvested

So then, the ages of trees that are harvested become zero in that year: age.mat[x.mat] <- 0

I then would like to age the harvested trees up again for the following periods. E.g. if a tree were harvested in the first planning period, in the second planning period (5 years later), I want the age of the tree to be 5, then in the third planning period (10 years later), I want the age of the tree to be 10. I have successfully implemented this in the following for loop:

for (i in 2:5){ # we don't need to calculate over the first year
  age.mat[,i]<-age.mat[,i-1]+5L # add 5 to previous year
  age.mat[x.mat[,i],i] <- 0L # reset age of harvested trees to zero
}

This works, however, it is clunky and slow. Is there a way to implement this faster (i.e. without the for loop)? It also is implemented within a function, which means that using "apply" actually slows things down, so it needs to be vectorized directly. This is something I'm iterating over thousands of times so speed is of the essence!

Thank you!

r for-loop matrix loops performance
5 Answers

An alternative to the t(apply in @Jon Spring's answer is matrixStats::rowCumsums.

library(matrixStats)

n <- 1e4L
n10 <- n/10L
age.mat <- outer(sample(150, n, TRUE), seq(0, 20, 5), "+")
x.mat <- matrix(FALSE, n, 5) # create the empty harvest matrix
# sample harvests so that no tree is harvested twice
x.mat[matrix(c(sample(n, n/2L), sample(n10:(6L*n10 - 1L)) %/% n10), n/2L)] <- TRUE

f1 <- function(age, x) {
  age[x[,1],] <- 0
  for (i in 2:5){ # we don't need to calculate over the first year
    age[,i] <- age[,i - 1] + 5L # add 5 to previous year
    age[x[,i], i] <- 0L # reset age of harvested trees to zero
  }
  age
}

f2 <- function(age, x) {
  age - rowCumsums(x*age)
}

microbenchmark::microbenchmark(f1 = f1(age.mat, x.mat),
                               f2 = f2(age.mat, x.mat),
                               check = "equal")
#> Unit: microseconds
#>  expr   min    lq     mean median     uq     max neval
#>    f1 294.4 530.2 1023.450  566.6 629.35 33222.8   100
#>    f2 135.2 263.6  334.622  284.2 307.15  4343.6   100
user_94634890

This looks to be about 12x faster, based on testing with rbenchmark.

Here's an approach relying on the fact that harvesting a tree doesn't stop the passage of time, it just resets the clock. So we can think of a harvest as subtracting the harvest age from all future ages.

x.die <- x.mat * age.mat 
x.dif <- t(apply(x.die, 1, cumsum))
age.mat2 <- age.mat - x.dif

x.die, by multiplying the harvests by the ages, we get the age at each harvest. The next line calculates the cumulative sum of these across each row, and finally we subtract those from the original ages.

I assume your "trees can't be harvested twice" means we won't ever see two TRUEs in one row of x.mat? My code won't work right if there were more than one harvest per tree location.

user_68518250

I found a way to do it! I implemented the idea of going backwards from @john-spring, where I created a matrix with the age of the stand at the harvested year filled in for the harvested year and all subsequent years, then subtracted that from my pre-made aged-up matrix. I built a function similar to what "fill" from tidyr or "na.locf" from zoo did (because they were too slow).

First I used arrayInd to determine the positions in the matrix of trees that were changed. I then used that to make another matrix that combined a repeat of each index row a number of times equal to the number of periods minus the period the tree was harvested in plus one, and a sequence vector of the same length that sequences from the period of the index number to the number of periods.

x.ind <- arrayInd(which(x.mat), dim(x.mat)) # gets index of row that was changed
x.new.ind <- cbind(rep(x.ind[,1], times = nper-x.ind[,2]+1), sequence(nvec = nper-x.ind[,2]+1, from = x.ind[,2]))

For example, if there was a tree harvested at position [4, 2], meaning the fourth tree was harvested in the second period, and we had 5 periods total, it would create a matrix:

    [,1] [,2] 
[1,]  4    2
[2,]  4    3
[3,]  4    4
[4,]  4    5

Then I made a vector with the ages of the trees that were harvested in the correct positions, and zeros in the rest of the positions (e.g. for our example, if the tree harvested was 100 years old, we would have a vector of 0 0 0 100 0 (if we had 5 trees)).

ages.vec <- vector(mode = "integer", length = nrow(age.mat))
ages.vec[x.ind[,1]]<- age.mat[x.ind]

I then multiplied this vector by a logical matrix with "T" at the row, column positions in the matrix above.

Continuing with the above example, we get:

    [,1] [,2] [,3] [,4] [,5]
[1,]  0    0    0    0    0
[2,]  0    0    0    0    0
[3,]  0    0    0    0    0
[4,]  0  100  100  100  100
[5,]  0    0    0    0    0

I then subtracted it from our current (already aged-up) ages matrix. So tree four was 95 100 105 110 115 and now it is 95 0 5 10 15.

new.ages.mat<- age.mat - replace(x.mat, x.new.ind, TRUE)*ages.vec

Though this might not be the most elegant solution, using microbenchmark, it is 90x faster than our for loop, and 3x faster than the lovely apply function that John created. I would put in the microbenchmark calls and results, but this post is long enough already! I know there's a better way to create the ages.vec and incorporate it, and am going to continue working on that, and will update this answer with my results!

user_193342100

This approach builds on the use of which used with arr.ind=TRUE to create a two column matrix the encodes the starting locations (in first column) and times (in second column) for new tree planting. It does violate the functional programming paradigm by using <<- to assign new values to age.mat` "in place".

fiveseq <- seq(0,20, by=5) # this way one only needs to call `seq` once
apply(which(x.mat, arr.ind=TRUE) , 1, 
          function(r) {age.mat[ r[1], r[2]:5] <<- fiveseq[  1:(6-r[2])] } )

In summary, it locates the new locations and intervals and replaces the rest of that row with the right number of items from the sequence {0, 5, 10, 15, 20}

(I would be interested in seeing how this compares with the benchmarking framework that you have already established.)

user_18556770

You can use apply to work on each vector rowwise, then use some logic within the function to adjust the values.

Should be about 4 times faster

  age.mat |> 
  apply(1, \(x) { 
    
    if(any(x == 0 & (which(x == 0) != length(x)))) { 
      
   x[which(x == 0):length(x)] <- (0:(length(x) - which(x == 0))) * 5
   
   x
    
      } else x
  }) |> t()

     [,1] [,2] [,3] [,4] [,5]
  [1,]  101    0    5   10   15
  [2,]   55   60   65   70   75
  [3,]   23   28   33    0    5
  [4,]    0    5   10   15   20
  [5,]   23   28   33    0    5
  [6,]   84    0    5   10   15
  [7,]   52   57   62    0    5
  [8,]   26   31   36   41    0
  [9,]  114  119  124  129    0
 [10,]   33   38   43   48   53
 [11,]  144  149  154  159  164
 [12,]   19   24   29   34   39
 [13,]   43   48   53   58   63
 [14,]   69   74   79   84   89
 [15,]   98  103  108  113  118
 [16,]  110  115  120  125  130
 [17,]    8   13   18   23   28
 [18,]   16   21   26   31   36
 [19,]    1    6   11   16   21
 [20,]   60   65    0    5   10
Admin1
Related