Pandas Resample-Sum without Zero filling

Viewed 626

When resampling Series with mean aggregation (daily to monthly) -> missing datetimes are filled with NaNs which is okay since we can simply remove them using .dropna() function, however, with sum/total aggregation -> missing datetimes are filled with 0s (zeros) which is technically correct, but a bit bothersome as masks are needed to remove them. The question is if there is a more efficient way on resampling with aggregate sum without zero-filling or using masks? Preferrably similar to dropna() but for dropping 0s.

For example:

ser = pd.Series([1]*6)
ser.index = pd.to_datetime(['2000-01-01', '2000-01-02', '2000-03-01', '2000-03-02', '2000-05-01', '2000-05-02'])
# wanted output
# 2000-01-31    2.0
# 2000-03-31    2.0
# 2000-05-31    2.0

# ideal output but for aggregate sum.
ser.resample('M').mean().dropna()
# 2000-01-31    1.0
# 2000-03-31    1.0
# 2000-05-31    1.0

# not ideal
ser.resample('M').sum()
# 2000-01-31    2
# 2000-02-29    0
# 2000-03-31    2
# 2000-04-30    0
# 2000-05-31    2

using .groupby() with .grouper() seems to have the exact behavior from resampling.

# not ideal
ser.groupby(pd.Grouper(freq='M')).sum()
# 2000-01-31    2
# 2000-02-29    0
# 2000-03-31    2
# 2000-04-30    0
# 2000-05-31    2

using .groupby() with index.year is also doable, however, there does not seem to be an 'identity' for calendar month. Noting that .index.month is not what we are after.

ser = pd.Series([1]*6)
ser.index = pd.to_datetime(['2000-01-01', '2000-01-02', '2002-03-01', '2002-03-02', '2005-05-01', '2005-05-02'])
ser.groupby(ser.index.year).sum()
# 2000    2
# 2002    2
# 2005    2

1 Answers

Use pd.offsets.MonthEnd and add this with the DatetimeIndex of ser to create a month end grouper, then use Series.groupby with this grouper and aggregate using sum or mean:

grp = ser.groupby(ser.index + pd.offsets.MonthEnd()) 
s1, s2 = grp.sum(), grp.mean()

Result:

print(s1)
2000-01-31    2
2002-03-31    2
2005-05-31    2
dtype: int64

print(s2)
2000-01-31    1
2002-03-31    1
2005-05-31    1
dtype: int64
Related