Remove first and last group from python pandas groupby

user_10439888 0
Viewed 190

I have some timeseries data

time x   y
1s   34  8017  
1s   43  5019
1s   1   8017
2s   64  8870
2s   34  8305
2s   11  8305
3s   343 8221
3s   3   8221
3s   143 8221

that I grouped by df.groupby(data.index.second) using python pandas groupby. Producing 3 groups where group 1 looks like this which corresponds to the first second

time x   y
1s   34  8017  
1s   43  5019
1s   1   8017

How can I remove the first group (1th second) and the last group (3th second)?

I only want this group (group 2)

time x   y
2s   64  8870
2s   34  8305
2s   11  8305

I have tried this without success and maybe the groupby function is not the way to go.

pandas-groupby pandas dataframe python
5 Answers

You can do something like this

df2 = df[df['time']=='2s']

This will remove both '1s' and '3s' from your main df and then we can store it in new variable which we can all df2

user_159261360
user_104398880

I note that you answered your own question, but perhaps this is of some use: using filter,

df.groupby('time').filter(lambda g: g.name not in ['1s','3s'])

produces

    time    x   y
3   2s     64   8870
4   2s     34   8305
5   2s     11   8305
user_145514260

You can filter all unique times without first and last unique values in Series.isin with boolean indexing:

df = df[df['time'].isin(df['time'].unique()[1:-1])]
print (df)
  time   x     y
3   2s  64  8870
4   2s  34  8305
5   2s  11  8305
user_29010020

Or a better solution:

data = data.drop(data[data.timestamp.dt.second == data.iloc[0][0].second].index)
data = data.drop(data[data.timestamp.dt.second == data.iloc[-1][0].second].index)
user_104398880
Related