I do data wrangling (ETL) to load new data into a data base and want to let users implement their own data validation functions based on my existing data.table object that contains the prepared data.
How can I avoid that the user can change (modify) the data of my data.table within the validation function (intentionally or unintentionally) without making a copy (since it slows down the validation processing and therefore the whole ETL process)?
Locking/write-protecting the data is only required temporarily within the validation function...
Simplified example:
library(data.table)
DT <- as.data.table(mtcars)
DT[, row.number := 1:(.N)] # add a row number column to allow identification of invalid rows
DT[c(3,6,7), cyl := 100] # create some data errors (100 cylinders in the car)
# correctly implemented validate function (does not change the data)
validate <- function(data) {
data[cyl > 10, .(row.number = row.number, col.name = "cyl", col.value = cyl,
severity = "ERROR", msg = "More than 10 cylinders")]
}
validate(DT)
The output is OK and the data.table passed to the validate() function is not changed:
row.number col.name col.value severity msg
1: 3 cyl 100 ERROR More than 10 cylinders
2: 6 cyl 100 ERROR More than 10 cylinders
3: 7 cyl 100 ERROR More than 10 cylinders
What I want to avoid is that a user implements validate() like this and modifies the original data:
validate.with.side.effects <- function(data) {
data[, max.cyl := 10] # this adds a new column into the original data.table!
data[cyl > max.cyl, .(row.number = row.number, col.name = "cyl", col.value = cyl, severity = "ERROR", msg = "More than 10 cylinders")]
}
This implementation would modify the original data.table in DT!
Update:
There is an open feature request for that: https://github.com/Rdatatable/data.table/issues/1086
The feature request is based on a similar requirement in a different context (immutable data in package): Lock or protect a data.table in R
Update 2
There is another similar feature request for that: https://github.com/Rdatatable/data.table/issues/778
And another similar SO question: How does one return a 'const' data.table from an R function?
Update 3: Could an immutable data.frame be used?
Generally yes (to make a data.frame read-only) but this solution does not meet the requirements of the question:
library(plyr)
idf <- idata.frame(DT)
idf[, max.cly := 10]
# Error in `:=`(max.cly, 10) :
# Check that is.data.table(DT) == TRUE. Otherwise, := and `:=`(...) are defined for use in j, once only and in particular ways. See help(":=").
validate(idf)
# Error in `[.idf`(data, cyl > 10, .(row.number = row.number, col.name = "cyl", :
# object 'cyl' not found
# it is no longer a data.table...
class(idata.frame(DT))
# [1] "idf" "environment"
# so that the data.table syntax does NOT work anymore...
idf[cyl > 10, .(row.number = row.number, col.name = "cyl", col.value = cyl,
severity = "ERROR", msg = "More than 10 cylinders")]
# Error in `[.idf`(idf, cyl > 10, .(row.number = row.number, col.name = "cyl", :
# object 'cyl' not found