performance considerations get() in data.table

user_7326560 0
Viewed 53

I've been using get() in a loop to manipulate a column j by i with reference to multiple other columns.

I wonder if there is a faster/more efficient way? Any performance considerations?

Here a minimal example of the type of operation I have in mind:

require(data.table) # version 1.12.8
dt = data.table(v1=c(1,2,NA),v2=c(0,0,1),v3=c(0,0,0))
for (i in 1:2){
     dt[ is.na(get(paste0('v',i))), (paste0('v',i)):= get(paste0('v',i+1))+2 ][]
}

The actual tables I do this with are much larger (~5 mio rows, ~300 columns).

I'd highly appreciate any thoughts.

data.table r loops
2 Answers

We can use set which would assign in place

library(data.table)
for(j in 1:2) {
    i1 <- which(is.na(dt[[j]]))
    set(dt, i = i1, j = j, value = dt[[j+1]][i1]+ 2)
 }

dt
#   v1 v2 v3
#1:  1  0  0
#2:  2  0  0
#3:  3  1  0

There is not much difference between a for loop or lapplyif both are using the get. For performace improvement, it is better to use set

In base R, we can also do

setDF(dt)
i1 <- is.na(dt[-length(dt)])
dt[-length(dt)][i1] <- dt[-1][i1] + 2
dt
#  v1 v2 v3
#1  1  0  0
#2  2  0  0
#3  3  1  0
user_37322710

Yes your for loop slows you down considerably. Even a simple lapply (and there's probably more elegant ways to do), brings you significant performance gains:

library(data.table)
dt <- data.table(v1 = rnorm(100), v2 = sample(c(NA,1:5)), v3 = sample(c(NA,1:5)), v4 = sample(c(NA,1:5)))
dt2 <- copy(dt)
dt3 <- copy(dt)
dt4 <- copy(dt)

microbenchmark::microbenchmark(
  for (i in 1:2){
    dt[ is.na(get(paste0('v',i))), (paste0('v',i)):= get(paste0('v',i+1))+2 ]
  },
  for (i in 1:2){ 
    dt2[ is.na(get(paste0('v',i))), (paste0('v',i)):= get(paste0('v',i+1))+2 ][]
  },
  lapply(1:2, function(i) dt3[ is.na(get(paste0('v',i))), (paste0('v',i)):= get(paste0('v',i+1))+2 ]),
  for(j in 1:2) {
    i1 <- which(is.na(dt4[[j]]))
    set(dt4, i = i1, j = j, value = dt[[j+1]][i1]+ 2)
  }  
)

Unit: milliseconds
                                                                                                                   expr      min       lq      mean   median
    for (i in 1:2) {     dt[is.na(get(paste0("v", i))), `:=`((paste0("v", i)), get(paste0("v",          i + 1)) + 2)] } 8.439924 8.651308 10.257670 8.900500
 for (i in 1:2) {     dt2[is.na(get(paste0("v", i))), `:=`((paste0("v", i)), get(paste0("v",          i + 1)) + 2)][] } 8.902435 9.098875 10.469305 9.262659
     lapply(1:2, function(i) dt3[is.na(get(paste0("v", i))), `:=`((paste0("v",      i)), get(paste0("v", i + 1)) + 2)]) 1.032788 1.144117  1.561741 1.224858
           for (j in 1:2) {     i1 <- which(is.na(dt4[[j]]))     set(dt4, i = i1, j = j, value = dt[[j + 1]][i1] + 2) } 6.216452 6.392754  7.970074 6.502356
       uq       max neval
 9.588571 35.259060   100
 9.729876 23.245224   100
 1.349337  9.467026   100
 7.046646 30.857044   100

Checking results are equivalent:

identical(dt,dt2)
# [1] TRUE
identical(dt,dt3)
# [1] TRUE
identical(dt,dt4)
# [1] TRUE

There's probably more elegant way to do that but a division by 10 of mean computation time for something that only took a few seconds to program is a good yield ;)

user_91977260
Related