Calculating handling time out of overlapping intervals

user_16233135 0
Viewed 129

I have a raw data exported and transformed a bit from salesforce below;

df = pd.DataFrame(columns=['contact_start','name', 'aht'], 
                  data=[['2021-09-27 09:58:00','Venus','180'],
                        ['2021-09-27 10:00:00','Venus','240'],
                        ['2021-09-27 11:05:00','Venus','60'],
                        ['2021-09-27 10:55:00','Mars','30'],
                        ['2021-09-27 10:56:00','Mars','30']])

original data

using these codes below

df["contact_start"] = pd.to_datetime(df["contact_start"], format = "%Y-%m-%d %H:%M:%S",errors='coerce')
df["date"] = df["contact_start"].dt.strftime('%Y-%m-%d')
df['aht']=pd.to_datetime(df["aht"], unit='s').dt.strftime("%H:%M:%S")
df['contact_finish'] = pd.to_timedelta(df['aht']) + pd.to_datetime(df['contact_start'])
df['contact_finish'] = df['contact_finish'].astype('datetime64[s]')

I transform this into :

transform

but my final goal is to deal with overlapping and I ran out of ideas how to make it happen.

the outcome should be like this in below:

df = pd.DataFrame(columns=['date','name', 'total_duration_sec'], 
                  data=[['2021-09-27','Venus','420'], 
                        ['2021-09-27','Mars','60']])

outcome

I guess it looks simple but in fact it is really not. I would appreciate any help.

Edit : I do not know how to put a more meaningful data in python so i uploaded a sample data file (3kb csv)

sample data

pandas intervals overlap datetime python
3 Answers

I think you could create a time difference in seconds between successive contact_start per name

upper_seconds = (
    df.sort_values(['name','contact_start'])
      .groupby('name')['contact_start'].diff(-1)
      .dt.total_seconds().abs())

print(upper_seconds.sort_index())
# 0     120.0
# 1    3900.0
# 2       NaN
# 3      60.0
# 4       NaN
# Name: contact_start, dtype: float64

Now you can use this as a upper clip on aht then groupby name and date and sum.

res = (
    df['aht'].astype(int)
      .clip(upper=upper_seconds)
      .groupby([df['name'], df['date']]).sum()
      .reset_index(name='total_duration_sec')
)
print(res)
    name        date  total_duration_sec
0   Mars  2021-09-27                  60
1  Venus  2021-09-27                 420

Note that I used first two lines you already wrote to have the good type.

df["contact_start"] = pd.to_datetime(df["contact_start"], 
                                     format = "%Y-%m-%d %H:%M:%S",errors='coerce')
df["date"] = df["contact_start"].dt.strftime('%Y-%m-%d')
user_92747320

You can make your existing code work by adding these lines to your code:

overlapped = pd.Series(df.groupby(['name']).apply(lambda x: (x['contact_finish'] - x['contact_start'].shift(-1)).dt.total_seconds().shift()).droplevel(0), name='overlapped')
overlapped = overlapped.mask(overlapped<0, 0).fillna(0)

df['date'] = df['contact_start'].dt.date
df = df.groupby(['date', 'name']).apply(lambda x: (((x['contact_finish'] - x['contact_start']).dt.seconds) - overlapped).sum()).reset_index(name='total_duration_sec')

OUTPUT:

         date   name  total_duration_sec
0  2021-09-27   Mars                60.0
1  2021-09-27  Venus               420.0
user_107207230

There is a solution involving step functions which can handle overlaps over day boundaries (in case a more general approach is required)

import pandas as pd
import staircase as sc

def create_union_stepfunction(dframe):
   return sc.Stairs(dframe, "contact_start", "contact_finish").make_boolean()

step_functions = df.groupby("name").apply(create_union_stepfunction)

This gives you a series called step_functions, indexed by planet name, and the values are staircase.Stairs objects which represent step functions.

name
Mars     <staircase.Stairs, id=1956311648200>
Venus    <staircase.Stairs, id=1956311120648>
dtype: object

These step functions have value 1 during contact, and 0 otherwise. We can then slice the step functions up with bins and calculate the integral, to get the total time per bin that contact was made. For daily bins use

def calc_seconds_per_bin(sf, bins):
    return sf.slice(bins).integral()/pd.Timedelta("1 second")


step_functions.apply(calc_seconds_per_bin, pd.date_range("2021-9-27", "2021-9-29"))

You'll get a pandas.DataFrame

        [2021-09-27, 2021-09-28)    [2021-09-28, 2021-09-29)
name        
Mars                        60.0                         0.0
Venus                      420.0                         0.0

note: I am the creator of staircase. Please feel free to reach out with feedback or questions if you have any.

user_114727610
Related