Larger Goal:
Classify comment threads as spam or not spam.
Specific Goal:
Prepare comment threads for text classification
Example of a spam comment:
Joey Dyson: "I am so glad I was able to make $9000 in the last 30 days with Mrs Felicitas Harrison"
|-> Caron Jackson: "Can you tell me how you did this?"
|-> aaron1: "I'm also interested"
|-> John Ellison: "I'm also interested"
|-> Lucy Carlson: "Mrs Harrison can be contacted on +44 123..."
|-> Lucy Carlson: "456 789"
There is only one layer of comments. Put another way, a child comment cannot have another child comment.
How I plan to use the prepared comment threads
x = np.array(data["PREPARED_COMMENT_THREAD"])
cv = CountVectorizer()
x = cv.fit_transform(x)
xtrain, xtest, ytrain, ytest = train_test_split(x, y,
test_size=0.2,
random_state=42)
model.fit(xtrain, ytrain)
print(model.score(xtest, ytest))