Python DataFrame : Seperate rows based on custom condition?

Viewed 80

I have dataframe with three columns name, content and day

df

        content          day           name
    0     first_day      01-01-2017      marcus
    1     present        10-01-2017      marcus
    2     first_day      01-02-2017      marcus
    3     first_day      01-03-2017      marcus
    4     absent         05-03-2017      marcus
    5     present        20-03-2017      marcus
    6     first_day      01-04-2017      bruno
    7     present        11-04-2017      bruno
    8     first_day      01-05-2017      bruno
    9     absent         02-05-2017      bruno
    10    first_day      01-06-2017      bruno
    11    absent         02-06-2017      bruno
    12    payment        09-06-2017      bruno

I'm trying to find out users month wise whose rows has first_day,absent and present continuously.

Sample output:

        content          day           name         absent_after_present
    0     first_day      01-01-2017      marcus         False
    1     first_day      01-02-2017      marcus         False
    2     first_day      01-03-2017      marcus         True
    3     first_day      01-04-2017      bruno          False
    4     first_day      01-05-2017      bruno          False
    5     first_day      01-06-2017      bruno          True

Ex: marcus has first_day,absent and present continuously from 01-03-2017,05-03-2017 and 20-03-2017 in the same month. So marcus status should be True

3 Answers

Try the following:

def pattern_in_group(s):
    s_list = s.to_list()
    for i in range(0, len(s_list)-2):
        if ['first_day', 'absent', 'present' ] == s_list[i:i+3]:
            return True
    return False

df['day1'] = pd.to_datetime(df['day'], dayfirst=True)
df['absent_after_present'] = df.groupby(['name', df['day1'].dt.year, df['day1'].dt.month])['content'].transform(pattern_in_group)

df2 = df.groupby(['name', df['day1'].dt.year, df['day1'].dt.month], as_index=False).first().drop(columns='day1')

print(df2)



     name    content         day  absent_after_present
0   bruno  first_day  01-04-2017                 False
1   bruno  first_day  01-05-2017                 False
2   bruno  first_day  01-06-2017                  True
3  marcus  first_day  01-01-2017                 False
4  marcus  first_day  01-02-2017                 False
5  marcus  first_day  01-03-2017                  True

As you have a typo in the sample data in the last row, I have rectified it as follows:

Test Data Construction

data = {'content': ['first_day', 'present', 'first_day', 'first_day', 'absent', 'present', 'first_day', 'present', 'first_day', 'absent', 'first_day', 'absent', 'present'], 
 'day': ['01-01-2017', '10-01-2017', '01-02-2017', '01-03-2017', '05-03-2017', '20-03-2017', '01-04-2017', '11-04-2017', '01-05-2017', '02-05-2017', '01-06-2017', '02-06-2017', '09-06-2017'],
 'name': ['marcus', 'marcus', 'marcus', 'marcus', 'marcus', 'marcus', 'bruno', 'bruno', 'bruno', 'bruno', 'bruno', 'bruno', 'bruno']}   

df = pd.DataFrame(data)

print(df)

      content         day    name
0   first_day  01-01-2017  marcus
1     present  10-01-2017  marcus
2   first_day  01-02-2017  marcus
3   first_day  01-03-2017  marcus
4      absent  05-03-2017  marcus
5     present  20-03-2017  marcus
6   first_day  01-04-2017   bruno
7     present  11-04-2017   bruno
8   first_day  01-05-2017   bruno
9      absent  02-05-2017   bruno
10  first_day  01-06-2017   bruno
11     absent  02-06-2017   bruno
12    present  09-06-2017   bruno

May be you can try extracting the monthly content and then grouping by name and month as below.

import pandas as pd

data = pd.DataFrame({'content' : ['first_day','present', 'first_day', 'first_day', 'absent', 
'present', 'first_day', 'present', 'first_day', 'absent', 'first_day', 'absent', 'present'],
'day' : ['2017-01-01', '2017-01-10', '2017-02-01', '2017-03-01', '2017-03-05', '2017-03-20',
'2017-04-01', '2017-04-11', '2017-05-01', '2017-05-02', '2017-06-01', '2017-06-02', '2017-06-09'],
'name' : ['marcus', 'marcus', 'marcus', 'marcus', 'marcus', 'marcus', 'bruno', 'bruno', 'bruno',
'bruno', 'bruno', 'bruno', 'bruno']})

data['day'] = pd.to_datetime(data['day'])

data['month'] = data.day.dt.month

data_new = pd.DataFrame(data.groupby(['name', 'month'])['content'].unique()).join(pd.DataFrame(data.groupby(['name', 'month'])['day'].unique()), on=['name', 'month'])

data_new['absent_after_present'] = data_new['content'].apply(lambda x : True if len(x) == 3 and len(set(x)) == 3 else False)
data_new['day'] = data_new['day'].apply(lambda x : x[0])
data_new['content'] = data_new['content'].apply(lambda x : x[0])

data_new = data_new.droplevel(1)



data_new


name    content        day  absent_after_present

bruno   first_day   2017-04-01  False
bruno   first_day   2017-05-01  False
bruno   first_day   2017-06-01  True
marcus  first_day   2017-01-01  False
marcus  first_day   2017-02-01  False
marcus  first_day   2017-03-01  True

Your expected output contains True if some group contains 'absent' before 'present' (not after).

So I defined the source DataFrame as:

      content         day    name
0   first_day  01-01-2017  marcus
1     present  10-01-2017  marcus
2   first_day  01-02-2017  marcus
3   first_day  01-03-2017  marcus
4      absent  05-03-2017  marcus
5     present  20-03-2017  marcus
6   first_day  01-04-2017   bruno
7     present  11-04-2017   bruno
8   first_day  01-05-2017   bruno
9      absent  02-05-2017   bruno
10  first_day  01-06-2017   bruno
11     absent  02-06-2017   bruno
12    present  09-06-2017   bruno

(Note a change in the last row).

Start from:

import itertools

Then define a function, which returns the first row from a source group (grp), with added value for the last (new) column:

def getRow(grp):
    lst = [k for k, g in itertools.groupby(grp.content)]
    isAbs = lst[-2] == 'absent' and lst[-1] == 'present' if len(lst) > 1 else False
    return grp.iloc[0].append(pd.Series([isAbs], index=['absent_before_present']))

And to get the expected result, run:

result = df.groupby([pd.to_datetime(df.day, dayfirst=True)
    .apply(lambda x: x.strftime('%Y-%m')), 'name']).apply(getRow)\
    .reset_index(drop=True)

The result is:

     content         day    name  absent_before_present
0  first_day  01-01-2017  marcus                  False
1  first_day  01-02-2017  marcus                  False
2  first_day  01-03-2017  marcus                   True
3  first_day  01-04-2017   bruno                  False
4  first_day  01-05-2017   bruno                  False
5  first_day  01-06-2017   bruno                   True

Note that the above code uses actually 2 various groupby methods:

  • from Pandas (group df by year, month and name),
  • from itertools, where each new value (other than the present) creates a new output group.
Related