Pandas Sample Based on Two Criteria

user_13071340 0
Viewed 87

I have a dataframe that looks like this:

     feature     target
0       2        0
1       0        0
2       0        0
3       0        0
4       1        0
...    ...      ...
33208   1        0
33209   0        0
33210   2        0
33211   2        0
33212   1        0

In the feature column there are 3 classes (0, 1, 2) and in the target column there are two classes (0, 1). If I group the dataframe by this two columns, I get:

df.groupby(['feature', 'target']).size()

feature  target
0         0             4282
          1               81
1         0             8537
          1               37
2         0            20161
          1              115
dtype: int64

Each feature class have 0s and 1s as target values, I need to find a way of sampling this values, my intention is to have something like this at the end:

new_df.groupby(['feature', 'target']).size()

feature  target
0         0             4282
          1               81
1         0             4282
          1               37
2         0             4282
          1              115
dtype: int64

I need to sample the amount of target values for each feature class, any suggestions?

pandas-groupby pandas python
1 Answers

You have different distributions, depending on the value of feature. You need to sample n value from a distribution, provided the value of feature: given that there are 2 possible outcomes, that is a binomial distribution problem.
The approach shown below should facilitate situation when target is not necessarily (0, 1) - could be anything (win vs lose, team A vs team B, as so forth) as far as I can see:

import numpy as np
import pandas as pd

# this is just reproducting your grouped end stated
df = pd.DataFrame({"feature":[0, 0, 1, 1, 2, 2], "target":[0, 1, 0, 1, 0, 1], "number":[4282, 81, 4282, 37, 4282, 115]})
df = df.set_index(["feature", "target"])

def sample_values(feature, sample_size):
    # select one of the distribution by feature
    df_sub = df.loc[feature]

    (event1, number1), (event2, number2) = zip(df_sub.index,df_sub["number"].tolist())

    return [event2 if np.random.binomial(1, number2/(number1+number2))==1 else event1 for _ in range(sample_size)]

print(sample_values(2, 100))

OUTPUT

[1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
user_76942630
Related