Skip first n rows in each group

user_14585190 0
Viewed 458

Let's say I have a Pandas Dataframe:

df = pd.DataFrame({'Company': ['company A']*5 + ['company B']*5, 
         'Date': ['01.01.2020', '01.02.2020', '01.03.2020', '01.04.2020', '01.05.2020'] + 
                 ['01.04.2020', '01.05.2020', '01.06.2020', '01.07.2020', '01.08.2020'], 
         'Revenue': np.random.rand(1, 10)[0]*10000})

     Company        Date      Revenue
0  company A  01.01.2020  5033.243098
1  company A  01.02.2020  5967.112256
2  company A  01.03.2020  6328.425874
3  company A  01.04.2020  7289.514777
4  company A  01.05.2020  9642.728016
5  company B  01.04.2020   805.708717
6  company B  01.05.2020   162.177508
7  company B  01.06.2020  7549.296095
8  company B  01.07.2020  4398.211089
9  company B  01.08.2020  1651.938946

The goal is to get a DF where the first N months for each company are excluded:

     Company        Date      Revenue
2  company A  01.03.2020  5731.949686
3  company A  01.04.2020  4300.537741
4  company A  01.05.2020  4283.022397
7  company B  01.06.2020  8011.727731
8  company B  01.07.2020  1935.579432
9  company B  01.08.2020  3866.649045

For example like this:

for company in df['Company'].unique():
    company_df = df[df['Company'] == company].sort_values(by='Date')
    ind_to_drop = company_df.iloc[:2].index
    df = df.drop(ind_to_drop)

I'm looking for a more efficient way.

pandas group-by numpy python
3 Answers

You can use:

(df.sort_values(['Date']) # sort values by 'Date'
    .groupby('Company', as_index=False) # group by 'Company'
    .apply(lambda x: x.iloc[2:]) # skip first two rows
    .droplevel(0)) # drop first index level

Output:

     Company        Date      Revenue
2  company A  01.03.2020   559.525103
3  company A  01.04.2020  4692.250518
4  company A  01.05.2020  8206.546659
7  company B  01.06.2020  3519.014808
8  company B  01.07.2020  4902.521804
9  company B  01.08.2020  6533.685687
user_89736200

I would use groupby to get rid of repetitive filters by company. Also, I think that dropping all the indexes at one time slightly improves the performance -- again, a single scan through the database will do.

ind_to_drop = list()
for _, data in df.groupby(by=['Company']):
    data = data.sort_values(by='Date')
    ind_to_drop += list(data.iloc[:2].index)

df = df.drop(ind_to_drop)
user_131186320

You can use groupby().head() to extract the index then drop:

df.drop(df.sort_values(['Date']).groupby('Company').head(1).index)

Output:

     Company        Date      Revenue
1  company A  01.02.2020  8354.050677
2  company A  01.03.2020  9867.805507
3  company A  01.04.2020  4072.178342
4  company A  01.05.2020  9626.621319
6  company B  01.05.2020  8712.769956
7  company B  01.06.2020  6751.648895
8  company B  01.07.2020   492.769737
9  company B  01.08.2020  1709.737424
user_42384080
Related