How can I use a machine learning model to predict on data whose features differ slightly?

user_10734443 0
Viewed 80

I have a randomForest model trained on a bunch of NLP data (tf-idf values for each word). I want to use it to predict on a new dataset. The features in the model overlap with but don't quite match the features in the new data, such that when I predict on the new data I get:

Error in predict.randomForest(object = model, newdata = new_data) : 
  variables in the training data missing in newdata

I thought to get around this error by excluding all the features from the model which do not appear in the new data, and all the features in the new data which do not appear in the model. Putting aside for the moment the impact on model accuracy (this would significantly pare down the number of features, but there would still be plenty to predict with), I did something like this:

model$forest$xlevels <- model$forest$xlevels[colnames(new_data)]
# and vice versa
new_data <- new_data[names(model$forest$xlevels)]

This worked, insofar as names(model$forest$xlevels) == colnames(new_data) returned TRUE for each feature name.

However, when I try to predict on the resulting new_data I still get the variables in the training data missing in newdata error. I am fairly certain that I'm amending the correct part of the model (model$forest$xlevels), so why isn't it working?

machine-learning nlp random-forest r
1 Answers

i think you should go the other way around. That is add the missing columns to the newdata.

When you are working with bags of words, it is common to have words that are not present in some batch of new data. These missing words should just be encoded as a columns of zeros.

# do something like this (also exclude the target variable, obviously)
names_missing <- names(traindata)[!names(traindata) %in% names(new_data)]
new_data[,names_missing] <- 0L

and then you should be able to predict

user_90257150
Related