Error: BoxCox error during preprocess imputation R language

user_12891582 0
Viewed 53

I am looking at the answers to the Chapter 6 exercise 3 questions from the Applied Predictive Modeling book Max Kuhn and I am getting an error in the imputation prediction step (despite following their answer exactly). The reproducible code and problem is as follows:

    library(AppliedPredictiveModeling)
    library(caret)
    library(RANN)

    data(ChemicalManufacturingProcess)
    predictors <- subset(ChemicalManufacturingProcess,select= -Yield)
    yield <- subset(ChemicalManufacturingProcess,select="Yield")
    # Impute
    #Split data into training and test sets
    set.seed(517)
    trainingRows <- createDataPartition(yield$Yield,
                                          p = 0.7,
                                          list = FALSE)
    
    trainPredictors <- predictors[trainingRows,]
    trainYield <- yield[trainingRows,]
    testPredictors <- predictors[-trainingRows,]
    testYield <- yield[-trainingRows,]
    
    #Pre-process trainPredictors and apply to trainPredictors and testPredictors
    pp <- preProcess(trainPredictors,method=c("BoxCox","center","scale","knnImpute"))
    ppTrainPredictors <- predict(pp,newdata=trainPredictors)
    ppTestPredictors <- predict(pp,newdata=testPredictors) # This results in an error

The error it gives is: Error in RANN::nn2(old[, non_missing_cols, drop = FALSE], new[, non_missing_cols, : NA/NaN/Inf in foreign function call (arg 2)

When I use YeoJohnson transformation instead it seems to work (which i read is able to handle non positive numbers)

However, I dont understand why it isn't working on the test data seeing as its just a different subset of the training data? and it's just used for the imputation step of the question?

I was unable to find any answers to this which seems strange as surely other people who have followed the book would have noticed? or am I being thick?

Thank you

r-caret imputation predict transformation r
1 Answers

You get that error because the boxcox transformation does not take in zero. If you check the help page for BoxCoxTrans it writes:

If any(y <= 0) or if length(unique(y)) < numUnique, lambda is not estimated and no transformation is applied.

So if your preProcess() is run on a train set with no zeros in the columns, the boxcox transformation is applied, but it will not work on a test set with zeros.

In the book example above, most likely the seed was set with an older R version, so it works. If you are using a newer version of R, then it doesn't work. So if I check with your example:

cbind(colSums(trainPredictors==0,na.rm=TRUE),colSums(testPredictors==0,na.rm=TRUE)) 
                       [,1] [,2]
BiologicalMaterial01      0    0
BiologicalMaterial02      0    0
BiologicalMaterial03      0    0
BiologicalMaterial04      0    0
BiologicalMaterial05      0    0
BiologicalMaterial06      0    0
BiologicalMaterial07      0    0
BiologicalMaterial08      0    0
BiologicalMaterial09      0    0
BiologicalMaterial10      0    0
BiologicalMaterial11      0    0
BiologicalMaterial12      0    0
ManufacturingProcess01    1    2
ManufacturingProcess02   29    6
ManufacturingProcess03    0    0
ManufacturingProcess04    0    0
ManufacturingProcess05    0    0
ManufacturingProcess06    0    0
ManufacturingProcess07    0    0
ManufacturingProcess08    0    0
ManufacturingProcess09    0    0
ManufacturingProcess10    0    0
ManufacturingProcess11    0    0
ManufacturingProcess12  104   38
ManufacturingProcess13    0    0
ManufacturingProcess14    0    0
ManufacturingProcess15    0    0
ManufacturingProcess16    1    0
ManufacturingProcess17    0    0
ManufacturingProcess18    1    0

You can see that ManufacturingProcess16, ManufacturingProcess18 will give you problems.

Yeo-Johnson transformation can deal with zeros or negative values, so it is not a problem.

If you would like to carry on with the work example, you can try to use another seed:

set.seed(517)
trainingRows <- createDataPartition(yield$Yield,
                                          p = 0.7,
                                          list = FALSE)
    
trainPredictors <- predictors[trainingRows,]
trainYield <- yield[trainingRows,]
testPredictors <- predictors[-trainingRows,]
testYield <- yield[-trainingRows,]
user_122584590
Related