When doing text classification do the train and test text need to have the same shape after being tokenized

user_17171766 0
Viewed 14

I am revisiting a project I did with the reuters dataset and while my model has some slight overfitting the training accuracy being 99 and validation being around 96. When I evaluate the model on the test data my accuracy is around 27%. So I was wondering if this is because the training and test data have a different shape.

print(one_hot_train_results.shape)
print(one_hot_test_results.shape)

returned

(5485, 10000) (2189, 10000)

nlp
1 Answers

Usually if you have the wrong shape you should get an error rather than just having bad performance.

Whether the shape needs to be the same depends on what kind of model you're using. Some models can take inputs of arbitrary length, some will only handle things with a fixed length and will need to use padding or have some way of combining over-long documents.

user_3557150
Related