I have the dataframe ('data') which looks like this:
| index | competitor | region | sku | date | price |
|---|---|---|---|---|---|
| 000 | A | M | 01 | 2022-01-01 | 100 |
| 001 | A | M | 01 | 2022-01-02 | 099 |
| 002 | A | M | 01 | 2022-01-03 | 099 |
| 003 | A | B | 02 | 2022-01-01 | 101 |
| 004 | A | B | 02 | 2022-01-02 | 100 |
| 005 | A | B | 02 | 2022-01-03 | 101 |
Columns 'competitor', 'region', 'sku', 'date' DO NOT contain nans, but 'price' does.
I want to do the following:
- ffill prices for every competitor, for every region, for every sku;
- bfill prices for every competitor, for every region, for every sku;
- if all prices for competitor, for region, for sku are nans for all dates, then do nothing.
For loops/ apply is obviously too slow, so I decided to go with np.select:
prev_comp = data['competitor'].shift(1)
prev_reg = data['region'].shift(1)
prev_art = data['sku'].shift(1)
conditions = [
(data['price'].isna()) & (data['price'].shift(1).notna()) & (data['competitor'].values == prev_comp) & (data['region'].values == prev_reg) & (data['sku'].values == prev_art),
(data['price'].isna()) & (data['price'].shift(-1).notna()) & (data['competitor'].values != prev_comp) & (data['region'].values != prev_reg) & (data['sku'].values != prev_art),
(data['price'].shift(1).notna()) & (data['price'].shift(-1).notna())
]
choices = [
data.fillna(method='ffill'),
data.fillna(method='bfill'),
data
]
data = np.select(conditions, choices)
I get the following error:
ValueError: shape mismatch: objects cannot be broadcast to a single shape. Mismatch is between arg 0 with shape (3930229,) and arg 1 with shape (3930229, 10).
The error referres to the shapes of conditions (3930229,) and choices (3930229, 10), but I have no idea what to do with it.