I'm struggling big-time to get my XGBoost model to predict an article's engagement time from its text. First, I get a dataframe representing the features I extracted from the article like this:
article_features = pd.concat([tfidf_df, numeric_df_normalized], axis=1)
I then train my model and get the relevant correct columns (features):
with open('correct_columns') as fp:
correct_columns = pickle.load(fp)
Then I go through all of the required features and set them to 0.0 if they're not already in article_features:
for col in correct_columns:
if col not in article_features.columns:
article_features[col] = 0.0
Finally, I delete features that were extracted from this article that don't exist in the training data:
for col in article_features:
if col not in correct_columns:
del article_features[col]
So now article_features has the correct number of features. I try to run:
model.predict(article_features)
And I get:
ValueError: feature_names mismatch:...
So I Google around and try converting my dataframe to :
model.predict(article_features.as_matrix())
But I get the same error.
I was then worried about order of columns in article_features not being the same as correct_columns so I did:
article_features.sort_index(axis=1, inplace=True)
But got the same error.
Any idea how to fix?
Thanks!