How to access data most effectively in pandas

Question

I have a tiny test dataset for students university applications in different majors It looks like this

0   35377       female  Chemistry   False
1   56105       male    Physics     True
2   31441       female  Chemistry   False
3   51765       male    Physics     True
4   53714       female  Physics     True

shape is 500,4

I need to get the admission rate for females and solved it now in three different ways. Each of them return the correct result.

DONE

Using group by

female_admitted_rate = df.groupby('gender').get_group('female')[df['admitted'] == True].count()/len(df.groupby('gender').get_group('female'))```

```python
[OUT]
student_id    0.287938
gender        0.287938
major         0.287938
admitted      0.287938
dtype: float64

Using plain pandas

len(df[(df['gender']=='female') & (df['admitted'])])/(len(df[df['gender']=='female']))

[Out] 0.28793774319066145

Using query

len(df.query("gender == 'female' & admitted"))/len(df.query("gender == 'female'"))

[Out] 0.28793774319066145

QUESTIONS

What would you use to get this information?
Is there a special advantage for one of the shown approaches?
Is there one approach what makes absolutely not sense to you guys?
Is there an perticular calculation performance benefit by using one of the three above the others when it comes to big data sets?

user_11884237 · Answer

I think you only need DataFrame.loc[] + Series.mean():

df.loc[df['gender'].eq('female'), 'admitted'].mean()

True is interpreted as 1 and False as 0 by Series.mean

You can check with timeit

%%timeit
df.loc[df['gender'].eq('female'), 'admitted'].mean()
1.16 ms ± 66.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
len(df[(df['gender']=='female') & (df['admitted'])])/(len(df[df['gender']=='female']))
3.45 ms ± 428 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit
df.groupby('gender').get_group('female')[df['admitted'] == True].count()/len(df.groupby('gender').get_group('female'))
10.3 ms ± 718 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit
len(df.query("gender == 'female' & admitted"))/len(df.query("gender == 'female'"))
11.1 ms ± 604 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

These times are for the sample DataFrame I think performance could vary greatly depending on the shape of your DataFrame. Although I honestly believe that the method I propose will be the fastest in most cases, in addition to providing a clean and simple syntax.

How to access data most effectively in pandas

1 Answers