Getting Error: Could not find function "." for only one data.table using .I

user_1818713 0
Viewed 1093

I've been through several similarly titled questions and I believe my case is different. To be sure, I've already stopped my Rstudio server, uninstalled data.table, then reinstalled it from source before restarting Rstudio server.

I have a data.table that looks like:

wind<-    structure(list(pricedate = structure(c(1538629200, 1538629200, 
                                       1538629200, 1538629200, 1538629200), class = c("POSIXct", "POSIXt"
                                       ), tzone = "America/Chicago"), hour = c(1L, 1L, 1L, 1L, 1L), 
               type = c("cop_hsl", "stwpf", "wgrpp", "cop_hsl", "stwpf"), 
               zone = c("coastal", "coastal", "coastal", "north", "north"
               ), as_of = structure(c(1538199804, 1538199804, 1538199804, 
                                      1538199804, 1538199804), class = c("POSIXct", "POSIXt"), tzone = "America/Chicago"), 
               wind = c(712, 751.5, 548.2, 843, 846), age = c("4day", "4day", 
                                                              "4day", "4day", "4day"), daysold = c(4L, 4L, 4L, 4L, 4L)), row.names = c(NA, 
                                                                                                                                       -5L), class = c("data.table", "data.frame"))

The full table is about 20M rows and takes up 1.1GB of ram as reported by tables()

The following commands work:

windindx<-wind[,.I[as_of==max(as_of)], by=.(pricedate, hour)][,V1]
wind[windindx]

Combining those into:

wind[wind[,.I[as_of==max(as_of)], by=.(pricedate, hour)][,V1]]

Results in Error: could not find function "."

If I subset that data.table then it works, like so:

windsm<-wind[round(runif(10000000,0,20676204))]
windsm[windsm[,.I[as_of==max(as_of)], by=.(pricedate, hour)][,V1]]

Here is my sessionInfo()

R version 4.1.1 (2021-08-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.2 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3

locale:
  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8      
[8] LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
  [1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
  [1] yaml_2.2.1         R.utils_2.10.1     R.oo_1.24.0        R.methodsS3_1.8.1  nanotime_0.3.3     xts_0.12.1         zoo_1.8-9          bit64_4.0.5        bit_4.0.4         
[10] glue_1.4.2         magrittr_2.0.1     future_1.21.0      lubridate_1.7.10   data.table_1.14.0  ggplot2_3.3.5      DALEX_2.3.0        mlr3tuning_0.8.0   paradox_0.7.1     
[19] mlr3viz_0.5.5      mlr3learners_0.4.5 mlr3_0.12.0        RPostgres_1.3.3   

loaded via a namespace (and not attached):
  [1] tidyselect_1.1.1     xfun_0.25            purrr_0.3.4          listenv_0.8.0        lattice_0.20-44      colorspace_2.0-2     vctrs_0.3.8          generics_0.1.0      
[9] htmltools_0.5.1.1    bbotk_0.3.2          utf8_1.2.2           blob_1.2.2           rlang_0.4.11         pillar_1.6.2         withr_2.4.2          DBI_1.1.1           
[17] palmerpenguins_0.1.0 uuid_0.1-4           lifecycle_1.0.0      munsell_0.5.0        gtable_0.3.0         codetools_0.2-18     evaluate_0.14        knitr_1.33          
[25] parallel_4.1.1       fansi_0.5.0          Rcpp_1.0.7           scales_1.1.1         backports_1.2.1      checkmate_2.0.0      RcppCCTZ_0.2.9       parallelly_1.27.0   
[33] hms_1.1.0            digest_0.6.27        dplyr_1.0.7          grid_4.1.1           tools_4.1.1          tibble_3.1.3         mlr3misc_0.9.3       crayon_1.4.1        
[41] pkgconfig_2.0.3      ellipsis_0.3.2       rmarkdown_2.10       lgr_0.4.2            R6_2.5.0             globals_0.14.0       compiler_4.1.1

Is there something about (relatively) big data.tables that prevents it from working? The machine I'm using is a 64GB VM on the cloud. htop is only reporting about 3.5GB of memory used so still about 60GB of free memory. My work around isn't too onerous so I'm more curious of the answer than anything.

EDIT: for the bounty, I'd like to know why eval is only required sometimes.

data.table r
2 Answers

You could use eval to force evaluation of i argument in base environment:

wind[eval(wind[,.I[as_of==max(as_of)], by=.(pricedate, hour)][,V1])]

    pricedate hour    type    zone               as_of  wind  age daysold
1: 2018-10-04    1 cop_hsl coastal 2018-09-29 00:43:24 712.0 4day       4
2: 2018-10-04    1   stwpf coastal 2018-09-29 00:43:24 751.5 4day       4
3: 2018-10-04    1   wgrpp coastal 2018-09-29 00:43:24 548.2 4day       4
4: 2018-10-04    1 cop_hsl   north 2018-09-29 00:43:24 843.0 4day       4
5: 2018-10-04    1   stwpf   north 2018-09-29 00:43:24 846.0 4day       4

help('data.table)` tells that :

Advanced: When i is a single variable name, it is not considered an expression of column names and is instead evaluated in calling scope.

This is why the first solution with i being a single variable windindx works, but not the combination which is evaluated in the wrong scope.

user_135133280

Elaborating on @Waldi's answer:

> wind[browser()]
Called from: eval(.massagei(isub), x, ienv)
Browse[1]> wind
[1] 712.0 751.5 548.2 843.0 846.0
Browse[1]> wind[,.I[as_of==max(as_of)], by=.(pricedate, hour)][,V1]
Error in .(pricedate, hour) : could not find function "."
Browse[2]> ls()
[1] "age"       "as_of"     "daysold"   "hour"      "pricedate" "type"      "wind"      "zone"

For the syntax x[i], this is the environment we are operating in when i is not a single symbol. This environment includes columns of x, and looks among them first when looking up a symbol.

If we instead pass a single symbol, like wind[wind, on=.(hour)], then it makes the lookup in the parent environment and does not need to evaluate i in the columns-of-x environment.

I think the documentation quoted in @Waldi's answer is sufficient to tell us how to avoid this issue, but for what it's worth this seems to be the relevant part of the code:

    else if (!is.name(isub)) {
      ienv = new.env(parent=parent.frame())
      if (getOption("datatable.optimize")>=1L) assign("order", forder, ienv)
      i = tryCatch(eval(.massagei(isub), x, ienv), error=function(e) {
        if (grepl(":=.*defined for use in j.*only", e$message))
          stopf("Operator := detected in i, the first argument inside DT[...], but is only valid in the second argument, j. Most often, this happens when forgetting the first comma (e.g. DT[newvar := 5] instead of DT[ , new_var := 5]). Please double-check the syntax. Run traceback(), and debugger() to get a line number.")
        else
          .checkTypos(e, names_x)
      })
    } else {
      # isub is a single symbol name such as B in DT[B]
      i = try(eval(isub, parent.frame(), parent.frame()), silent=TRUE)
      ...
    }
user_11912590
Related