python pandas callable in mask/where recalc for every entry

user_10016858 0
Viewed 125

I am using pandas mask/where function with random() as the callable - is it possible to get this to calculate a different value for each entry so that i dont get the same random number in each mask cell or can I only do this using apply?

e.g.

df_factor_history = pd.DataFrame([
    ['2020-10-01', 'A', 1, 5, 3], 
    ['2020-10-01', 'B', 1, 5, 3], 
    ['2020-11-01', 'A', 1, 5, 3], 
    ['2020-11-01', 'B', 1, 5, 3], 
    ['2020-12-01', 'A', 1, 5, 3], 
    ['2020-12-01', 'B', 1, 5, 3], 
],
    columns=['as_at_date',
             'factor_name', 
             'observation_1', 
             'observation_2', 
             'observation_3'])

df_factor_history.set_index(['as_at_date', 'factor_name'], inplace=True)

df_factor_history

enter image description here

df_valid = pd.DataFrame([
    ['2020-09-01', True, True],
    ['2020-10-01', False, True],
    ['2020-11-01', True, False],
    ['2020-12-01', True, True],
    ['2021-01-01', True, True]], 
    columns = ['as_at_date', 'A', 'B'])

df_valid.set_index(['as_at_date'], inplace=True)

df_valid

enter image description here

from random import seed
from random import random
# seed random number generator
seed(1)

df_factor_history.where(df_valid.T.unstack(), random())

This generates the same random number (in this case 0.134364) and i would like, in this example, six different random numbers enter image description here

pandas callable mask python
1 Answers

You can generate 2d numpy array by DataFrame.shape, here * means unpack tuple to integers:

np.random.seed(1)

arr = np.random.rand(*df_factor_history.shape)
print (arr)
[[4.17022005e-01 7.20324493e-01 1.14374817e-04]
 [3.02332573e-01 1.46755891e-01 9.23385948e-02]
 [1.86260211e-01 3.45560727e-01 3.96767474e-01]
 [5.38816734e-01 4.19194514e-01 6.85219500e-01]
 [2.04452250e-01 8.78117436e-01 2.73875932e-02]
 [6.70467510e-01 4.17304802e-01 5.58689828e-01]]

df = df_factor_history.where(df_valid.T.unstack(), arr)
print (df)
                        observation_1  observation_2  observation_3
as_at_date factor_name                                             
2020-10-01 A                 0.417022       0.720324       0.000114
           B                 1.000000       5.000000       3.000000
2020-11-01 A                 1.000000       5.000000       3.000000
           B                 0.538817       0.419195       0.685220
2020-12-01 A                 1.000000       5.000000       3.000000
           B                 1.000000       5.000000       3.000000
user_29010020
Related