I am using scikit-learn for text processing, but my CountVectorizer isn't giving the output I expect.
My CSV file looks like:
"Text";"label"
"Here is sentence 1";"label1"
"I am sentence two";"label2"
...
and so on.
I want to use Bag-of-Words first in order to understand how SVM in python works:
import pandas as pd
from sklearn import svm
from sklearn.feature_extraction.text import CountVectorizer
data = pd.read_csv(open('myfile.csv'),sep=';')
target = data["label"]
del data["label"]
# Creating Bag of Words
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(data)
X_train_counts.shape
count_vect.vocabulary_.get(u'algorithm')
But when I do print(X_train_counts.shape) I see the output is only (1,1), whereas I have 1048 rows with sentences.
What I am doing wrong? I am following this tutorial.
(Also the output of count_vect.vocabulary_.get(u'algorithm') is None.)