How do we use a Random Forest for sentence-classification using word-embedding

user_6224975 0
Viewed 645

When we have a random forest, we have n-inputs and m-features e.g for 3 observations and 2 features we have

X = [[1,23],[0,-12],[-0.5,29]]
y = [1,0,1]

and we can train a RandomForest with

from sklearn.ensemble import RandomForestClassifier
model = RandomForest()
model.fit(X,y)

If I have made a word-embedding using, say, a 100-dimensional vector, how do we create the X matrice, where each input is a sentence?

Say we have the following 3-dimensional embedding of the words ["I","like","dogs","cats"]:

I = [-0.5,0,1]
like = [5,2,3]
dogs = [1,2,3]
cats = [3,2,1]

then the dataset ["I like dogs","I like cats"] would be

X = [
[[-0.5,0,1], [5,2,3], [1,2,3]],
[[-0.5,0,1], [5,2,3], [3,2,1]]
]
y = ["dog-lover","cat-lover"]

which a RF naturally cannot train thus giving the erropr ValueError: Found array with dim 3. Estimator expected <= 2.

Apart from RF might not be suitable for NLP - is there a way to do so?

word-embedding nlp random-forest python
1 Answers

I don't think performing Random Forest classifier on the 3-dimensional input will be possible, but as an alternative way, you can use sentence embedding instead of word embedding. Therefore your input data will be 2-dimensional ((n_samples, n_features)) as this classifier expected.
There are many ways to get the sentence embedding vector, including Doc2Vec and SentenceBERT, but the most simple and commonly used method is to make an element-wise average over all the word embedding vectors.
In your provided example, the embedding length was considered as 3. Suppose that the sentence is "I like dogs". So the sentence embedding vector will be computed as follow:

I = [-0.5,0,1]
like = [5,2,3]
dogs = [1,2,3]
cats = [3,2,1]

# sentence: 'I like dogs'
sentence = [-0.5+5+1, 0+2+2, 1+3+3] / 3
         = [5.5, 4, 7] / 3
         = [1.8333, 1.3333, 2.3333]
user_93469420
Related