R: How to make a data.table read-only without copying it (e. g. for data validation)

Viewed 599

I do data wrangling (ETL) to load new data into a data base and want to let users implement their own data validation functions based on my existing data.table object that contains the prepared data.

How can I avoid that the user can change (modify) the data of my data.table within the validation function (intentionally or unintentionally) without making a copy (since it slows down the validation processing and therefore the whole ETL process)?

Locking/write-protecting the data is only required temporarily within the validation function...

Simplified example:

library(data.table)

DT <- as.data.table(mtcars)
DT[, row.number := 1:(.N)]  # add a row number column to allow identification of invalid rows
DT[c(3,6,7), cyl := 100]    # create some data errors (100 cylinders in the car)

# correctly implemented validate function (does not change the data)
validate <- function(data) {
  data[cyl > 10, .(row.number = row.number, col.name = "cyl", col.value = cyl,
                   severity = "ERROR", msg = "More than 10 cylinders")]
}

validate(DT)

The output is OK and the data.table passed to the validate() function is not changed:

   row.number col.name col.value severity                    msg
1:          3      cyl       100    ERROR More than 10 cylinders
2:          6      cyl       100    ERROR More than 10 cylinders
3:          7      cyl       100    ERROR More than 10 cylinders

What I want to avoid is that a user implements validate() like this and modifies the original data:

validate.with.side.effects <- function(data) {
  data[, max.cyl := 10]  # this adds a new column into the original data.table!
  data[cyl > max.cyl, .(row.number = row.number, col.name = "cyl", col.value = cyl, severity = "ERROR", msg = "More than 10 cylinders")]
}

This implementation would modify the original data.table in DT!

Update:

There is an open feature request for that: https://github.com/Rdatatable/data.table/issues/1086

The feature request is based on a similar requirement in a different context (immutable data in package): Lock or protect a data.table in R

Update 2

There is another similar feature request for that: https://github.com/Rdatatable/data.table/issues/778

And another similar SO question: How does one return a 'const' data.table from an R function?

Update 3: Could an immutable data.frame be used?

Generally yes (to make a data.frame read-only) but this solution does not meet the requirements of the question:

library(plyr)
idf <- idata.frame(DT)
idf[, max.cly := 10]
# Error in `:=`(max.cly, 10) :
#  Check that is.data.table(DT) == TRUE. Otherwise, := and `:=`(...) are defined for use in j, once only and in particular ways. See help(":=").

validate(idf)
# Error in `[.idf`(data, cyl > 10, .(row.number = row.number, col.name = "cyl",  :
#                                     object 'cyl' not found

# it is no longer a data.table...
class(idata.frame(DT))
# [1] "idf"         "environment"

# so that the data.table syntax does NOT work anymore...
idf[cyl > 10, .(row.number = row.number, col.name = "cyl", col.value = cyl,
                severity = "ERROR", msg = "More than 10 cylinders")]
# Error in `[.idf`(idf, cyl > 10, .(row.number = row.number, col.name = "cyl",  :
#                                    object 'cyl' not found
1 Answers
Related