sklearn FeatureHasher output has lots of collisions and unused columns

Question

I am using sklearn.feature_extraction.FeatureHasher to encode categorical variables for machine learning. My categorical variables are numeric IDs, like 1, 2, 3, 6, 18, 19, 20, ... . In total I have 18,000 unique IDs and the highest ID is 28,000.

I want to hash them to encode them as categories since One-hot encoding is out of the picture as it would create 18,000 columns and my dataset already has 4,000,000 rows and that would be painful.

I can't show my dataframe so I illustrate on a sample dataframe.

from sklearn.feature_extraction import FeatureHasher
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

tmpdf = pd.DataFrame(np.arange(10000), columns=["test"])

produces

      test
0        0
1        1
2        2
3        3
4        4
...    ...
9995  9995
9996  9996
9997  9997
9998  9998
9999  9999

[10000 rows x 1 columns]

Now if I hash with a dictionary:

H = FeatureHasher(n_features=50, input_type="dict")

cdict = [{str(i) : 1} for i in range(10000)]
arr = H.transform(cdict)

plt.matshow(arr.toarray()[::100], cmap="tab10", vmin=-5, vmax=4)
plt.colorbar(shrink=0.4)
plt.show()

this produces:

If instead I hash with strings:

H = FeatureHasher(n_features=50, input_type="string")

arr = H.transform(tmpdf["test"].astype(str))

plt.matshow(arr.toarray()[::100], cmap="tab10", vmin=-5, vmax=4)
plt.colorbar(shrink=0.4)
plt.show()

I get:

Question:

Why does the output of the string based hashing look so weird? looks like a lot of collisions are happening as a lot of columns remain completely unused... Am I using it wrong? For my task of encoding user IDs, is there a better way to do it?

user_9439097 · Answer

As commented by Ben Reiniger, the issue is that giving a list of strings makes the hasher iteratre over the strings, so e.g. the string "12345" is not hashed as is, but instead the substrings "1", "2", "3", "4", "5" are hashed instead. Given I have only numbers, I only have 10 unique strings (The numbers "0" to "9"), and this thus results in a lot of collisions

Feature Hashing of zip codes with Scikit in machine learning

A potential solution is to put the strings into lists, so the Hasher won't iterate over the string itself, but instead over the list, and then hash the complete string.

obj = tmpdf["test"].astype(str).to_numpy()
obj = obj.reshape(*obj.shape, 1)

results in

array([['0'],
       ['1'],
       ['2'],
       ...,
       ['9997'],
       ['9998'],
       ['9999']], dtype=object)

Which when hashed:

H = FeatureHasher(n_features=50, input_type="string")

arr = H.transform(obj)

plt.matshow(arr.toarray()[::100], cmap="tab10", vmin=-5, vmax=4)
plt.colorbar(shrink=0.4)
plt.show()

produces an output that is more to my liking.

Thanks!

sklearn FeatureHasher output has lots of collisions and unused columns

Question:

1 Answers