upsampling entire groups in python

user_11400815 0
Viewed 85

Suppose I have a dataframe like this

import pandas as pd

df = pd.DataFrame({'ID':[1,1,1,1,1,2,2,2,2], 'Order':[1,2,3,4,5,1,2,3,4]})

df
    ID  Order
0   1   1
1   1   2
2   1   3
3   1   4
4   1   5
5   2   1
6   2   2
7   2   3
8   2   4

I need to upsample by entire blocks, so essentially creating new copies of the block for ID == 1, and ID == 2. So the upsampled df (for n = 2 two samples with replacement) might look something like


df_upsampled = pd.DataFrame({'ID':[1,1,1,1,1,2,2,2,2,1,1,1,1,1,2,2,2,2], 'Order':[1,2,3,4,5,1,2,3,4,1,2,3,4,5,1,2,3,4]})

df_upsampled

    ID  Order
0   1   1
1   1   2
2   1   3
3   1   4
4   1   5
5   2   1
6   2   2
7   2   3
8   2   4
9   1   1
10  1   2
11  1   3
12  1   4
13  1   5
14  2   1
15  2   2
16  2   3
17  2   4

I thought I could handle this quick with resample() but haven't figured out how to copy entire blocks (per ID)

resampling python
2 Answers

So basically all you need to do is to get a copy of your data frame and concat it to the original one:

df = pd.concat([df, df], ignore_index=True)
user_52961060

The repeat function on series will be a better generalized solution. It only works on an Series so you have to combine it with apply to apply over all the columns.

df_upsampled = df.apply(pd.Series.repeat, axis=0, repeats=500)
# %timeit 646 µs ± 12.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

df = pd.concat([df for x in range(500)], ignore_index=True)
# %timeit 8.41 ms ± 623 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
user_104511260
Related