95% confidence interval of single proportion in pandas timeseries using statsmodel

user_14251610 0
Viewed 285

I have a time-series dataframe:

df = pd.DataFrame({'year':['2010','2011','2012','2013','2014','2015','2016','2017','2018','2019'],
                       'total_count': [545,779,706,547,626,530,766,1235,1260,947], 
                       'rand_count':[96,184,148,154,160,149,124,274,322,301],
                       'rand_perc':[17.61,23.62,20.96,28.15,25.56,28.11,16.19,22.19,25.56,31.78]
                       })

Here;

df['rand_perc'] = (df['rand_count']/df['total_count'])*100

Problem:

I want to compute confidence interval for single proportion of df['rand_count'] from df['total_count'] ,in each row of df and plot df['year'] vs df['rand_perc'] with CI as errorbars. I tried using statsmodel to compute CI for each row using the following code:

import statsmodels.api as sm

df['CI'] =  df[['total_count', 'rand_count']].apply(lambda row: sm.stats.proportion_confint(count = 
df['rand_count'], nobs = df['total_count'], alpha = 0.05), axis = 1)

But the result df['CI']looks very annoying with the tuple of all CIs in each row as;

0    ([0.14416430990026746, 0.2063732756491498, 0.1...
1    ([0.14416430990026746, 0.2063732756491498, 0.1...
2    ([0.14416430990026746, 0.2063732756491498, 0.1...
3    ([0.14416430990026746, 0.2063732756491498, 0.1...
4    ([0.14416430990026746, 0.2063732756491498, 0.1...
5    ([0.14416430990026746, 0.2063732756491498, 0.1...
6    ([0.14416430990026746, 0.2063732756491498, 0.1...
7    ([0.14416430990026746, 0.2063732756491498, 0.1...
8    ([0.14416430990026746, 0.2063732756491498, 0.1...
9    ([0.14416430990026746, 0.2063732756491498, 0.1...
Name: CI, dtype: object

Desired result

respective tuple of two elements in each row of df['CI'] like:

(0.144164, 0.206373)
(0.179606, 0.243846)
(0.221421, 0.242859)
...................

And two separate columns df[upper] and df[lower] for upper and lower limit of the df['CI'] respectively.

Your kind help would be highly appreciable.

Thanks a lot!

statsmodels pandas time-series python
1 Answers

Consider assigning multiple columns which should line up by index since per docs:

When a pandas object is returned, then the index is taken from the count.

df['lower_CI'], df['upper_CI'] =  sm.stats.proportion_confint(
                                      count = df['rand_count'],
                                      nobs = df['total_count'],
                                      alpha = 0.05
                                  )
user_14224510
Related