pandas using qcut on series with fewer values than quantiles

user_291877 0
Viewed 19692

I have thousands of series (rows of a DataFrame) that I need to apply qcut on. Periodically there will be a series (row) that has fewer values than the desired quantile (say, 1 value vs 2 quantiles):

>>> s = pd.Series([5, np.nan, np.nan])

When I apply .quantile() to it, it has no problem breaking into 2 quantiles (of the same boundary value)

>>> s.quantile([0.5, 1])
0.5    5.0
1.0    5.0
dtype: float64

But when I apply .qcut() with an integer value for number of quantiles an error is thrown:

>>> pd.qcut(s, 2)
...
ValueError: Bin edges must be unique: array([ 5.,  5.,  5.]).
You can drop duplicate edges by setting the 'duplicates' kwarg

Even after I set the duplicates argument, it still fails:

>>> pd.qcut(s, 2, duplicates='drop')
....
IndexError: index 0 is out of bounds for axis 0 with size 0

How do I make this work? (And equivalently, pd.qcut(s, [0, 0.5, 1], duplicates='drop') also doesn't work.)

The desired output is to have the 5.0 assigned to a single bin and the NaN are preserved:

0     (4.999, 5.000]
1                NaN
2                NaN
quantile pandas
3 Answers

You can try filling your object/number cols with the appropriate filling ('null' for string and 0 for numeric)

#fill numeric cols with 0
numeric_columns = df.select_dtypes(include=['number']).columns
df[numeric_columns] = df[numeric_columns].fillna(0)

#fill object cols with null
string_columns = df.select_dtypes(include=['object']).columns
df[string_columns] = df[string_columns].fillna('null')
user_93924460

Use python 3.5 instead of python 2.7 . This worked for me

user_64562900
Related