How to handle unknown number of values for a categorical feature?

Question

I have a pandas dataframe that looks like this

Text                  | Label

Some text             |   0
hellow bye what       |   1
...

Each row is a data point. Label is 0/1 binary. The only feature is Text which contains a set of words. I want to use the presence or absence of each word as features. For example, the features could be contains_some contains_what contains_hello contains_bye, etc. This is typical one hot encoding.

However I don't want to manually create so many features, one for every single word in the vocabulary (the vocabulary is not huge, so I am not worried about the feature set exploding). But I just want to supply a list of words as a single column to tensorflow and I want it to create a binary feature for each word in the vocabulary.

Does tensorflow/keras have an API to do this?

user_15529253 · Answer

You can use sklearn for this , try this:

from sklearn.feature_extraction.text import CountVectorizer
    
vectorizer = CountVectorizer(binary=True)
    
X = vectorizer.fit_transform(old_df['Text'])

new_df = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names())
    
new_df['Label'] = old_df['label']

and this should give you :

bye hellow  some    text    what    target
0     0       1     1         0         0
1     1       0     0         1         1

CountVectorizer convert a collection of text documents to a matrix of token counts. This implementation produces a sparse representation of the counts using scipy.sparse.csr_matrix and if binary = True then all non zero counts are set to 1. This is useful for discrete probabilistic models that model binary events rather than integer counts.

user_7605543 · Answer

What you're looking for is a (binary) bag of words which you can get from scikit-learn using their CountVectorizer here.

You can do something like:

from sklearn.feature_extraction.text import CountVectorizer

bow = CountVectorizer(ngram_range=(1, 1), binary=True)

X_train = bow.fit_transform(df_train['text'].values)

This will create an array of binary values indicating the presence of a word in each text. Use binary=True to output a 1 or 0 if the word is present. Without this field you will get counts of occurrences per word, either method works fine.

In order to inspect the counts you could use the below:

# Create sample dataframe of BoW outputs
count_vect_df = pd.DataFrame(X_train[:1].todense(),columns=bow.get_feature_names())

# Re-order counts in descending order. Keep top 10 counts for demo purposes
count_vect_df= count_vect_df[count_vect_df.iloc[-1,:].sort_values(ascending=False).index[:10]]

# print combination of original train dataframe with BoW counts
pd.concat([df_train['text'][:1].reset_index(drop=True), count_vect_df], axis=1)

Update

If your features include categorical data you could try using to_categorical from tf.keras. See the docs for more information.

How to handle unknown number of values for a categorical feature?

2 Answers