I'm trying to use scikit-learn's CountVectorizer to count character 2-grams, ignoring spaces. In the docs it mentions the parameter analyzer which states
Whether the feature should be made of word or character n-grams. Option ‘char_wb’ creates character n-grams only from text inside word boundaries.
However, "char_wb" doesn't appear to work as I expected. For example:
corpus = [
"The blue dog Blue",
"Green the green cat",
"The green mouse",
]
# CountVectorizer character 2-grams with word boundaries
vectorizer = CountVectorizer(analyzer='char_wb', ngram_range=(2, 2), min_df=1)
X = vectorizer.fit_transform(corpus)
vectorizer.get_feature_names()
[' b',
' c',
' d',
' g',
' m',
' t',
'at',
'bl',
'ca', ....
Notice the examples like ' b' which include a space. What gives?