I have the following table and I want to take one random sample for each id where value is equal to 1.
| id | date | value |
|---|---|---|
| A | 2022-01-01 | 0 |
| A | 2022-01-02 | 1 |
| A | 2022-01-04 | 0 |
| B | 2022-01-01 | 0 |
| B | 2022-01-02 | 0 |
| C | 2022-01-02 | 1 |
| C | 2022-01-03 | 1 |
| C | 2022-01-05 | 1 |
| C | 2022-01-06 | 0 |
In pandas I would do the below, but how do I do it in spark?
df[df.value==1].groupby('id').apply(lambda df: df.sample(1)).reset_index(drop=True)
This is what the final dataset could look like:
| id | date | value |
|---|---|---|
| A | 2022-01-02 | 1 |
| C | 2022-01-03 | 1 |