How to infer frequency from an index where a few observations are missing?

user_3437787 0
Viewed 879

Using pd.date_range like dr = pd.date_range('2020', freq='15min', periods=n_obs) will produce this DateTimeIndex with a 15 minute interval or frequency:

DatetimeIndex(['2020-01-01 00:00:00', '2020-01-01 00:15:00',
           '2020-01-01 00:30:00', '2020-01-01 00:45:00',
           '2020-01-01 01:00:00'],
          dtype='datetime64[ns]', freq='15T')

You can use this to set up a pandas dataframe like:

import pandas as pd
import numpy as np

# data
np.random.seed(10)
n_obs = 10
daterange = pd.date_range('2020', freq='15min', periods=n_obs)
values = np.random.uniform(low=-1, high=1, size=n_obs).tolist()
df = pd.DataFrame({'time':daterange, 'value':values})
df = df.set_index('time')

And now you can use pd.infer_freq(df.index) to retreive the frequency '15T' again for further calculations. Taking a closer look at help(pd.infer_freq()) lets us know that pd.infer_freq will:

Infer the most likely frequency given the input index. If the frequency is
uncertain, a warning will be printed.

My understanding of this would be that it would be possible to retrieve '15T' if a few observations were missing, leading to an irregular time index. But when I remove a few of the observations using:

dropped = df.index[[1,3]]
df =  df.drop(dropped)

Then pd.infer_freq(df.index) returns None. This also happens if we set n_obs = 100. So it would seem that I was hoping for a bit too much when I thought that [...] infer the most likely frequency [...] meant that pd.infer_freq() could infer that this was in fact an index with a 15 minute frequency with only a few missing values. Is there any other approach I could use to programmatically infer an index frequency from a somewhat irregular time series using pandas?

pandas python
1 Answers

You could compute the minimum time difference of values in the index (here min_delta), try to find 3 consecutive values in the index, each with this minimum time difference between them, and then call infer_freq on these consecutive values of the index:

diffs = (df.index[1:] - df.index[:-1])
min_delta = diffs.min()
mask = (diffs == min_delta)[:-1] & (diffs[:-1] == diffs[1:])
pos = np.where(mask)[0][0]
print(pd.infer_freq(idx[pos: pos + 3]))

This retrieves "15T".

user_3953500
Related