I'm relatively new to sk-learn & machine learning here so forgive any possible ignorance. I'm making a model to classify assets based on a text description (in python). There is only one predictor (the text) and one predicted (the category) variable. For the labels, I am factorizing the categories, there are about 30, so each is represented by a number from 0 to 29. For the features, I'm using a tf-idf score. The modeling and accuracy are fine and I'm saving the model using a pickle dump.
However, the model needs to be reusable, so it must be able to load again at some point in time in order to label a new set of data. Please see the code for the saving/loading of the model below.
## Save the linear SVC model
filename = 'SVM_model.sav'
pickle.dump(model, open(filename, 'wb'))
## Load model and test prediction accuracy
tfidf = TfidfVectorizer(analyzer='word',sublinear_tf=True, min_df=3,
norm='l2', ngram_range=(1, 2), encoding='latin-1', stop_words='english')
testdf=pd.read_excel('learning_df.xlsx').tail(54068)
testdf['input']=testdf['item_description'].astype(str)+'-
'+testdf['category_name_client'].astype(str)
testdf=testdf[['input','category_id_D&P']].dropna()
testdf['factor']=testdf['category_id_D&P'].apply(lambda x: cat_dict[x])
loaded_model = pickle.load(open(filename, 'rb'))
X_test1=tfidf.fit_transform(testdf.input).toarray()
y_test1=testdf['factor']
result = loaded_model.score(X_test1, y_test1)
print(result)
Please note that the tfidf vectorizer settings are exactly the same as what the model training/validation was done with. The cat_dict is the initial factorization of the categories and here I'm just making sure that the text categories are converted to the same numbers as what the model was trained/validated on.
When I attempted this, I came to this error:
ValueError: X has 24008 features per sample; expecting 20012
which is understandable because the tf-idf of the new dataset does not produce the same number of features as the training/validation dataset initially used.
So I was wondering if there a workaround for this? Should I not use tf-idf in the first place when training the model? If not, what are alternative ways for feature selection that would not lead to this problem later on?
Thanks in advance and again sorry if I'm missing something obvious.