Graphing the cumulative average per top k% value

Question

I have a DataFrame df sorted by value in a descending order:

value       gender       age
3015        male         10
2519        male         30
2397        male         15
...
1           male         12
1           female       10
1           male         9

value consists of int larger than 0.
gender consists of str data: male or female.
age consists of int larger than 0.

I have two objectives:

Graph the proportion of female per top k% value. (Hence, the graph should have the k% value for the x-axis and the proportion of female for the y-axis.)
Graph the average age cumulatively for female per top k% value. (Hence, the graph should have the k% value for the x-axis and the average age of female who qualify for that value for the y-axis.)

A more thorough explanation on Task 2:

For the top 20% value, for instance, I would first of all determine which value corresponds to the top 20%. Then, I would count all data points with value either equal to or greater than the top 20% value with gender == 'female', as well as cumulating their age. Finally, I would plot the average age, calculated by the cumulated age divided by the number of counted female data points.

I have completed the first task using np.arange() and np.cumsum():

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df_gender = df['gender'].to_numpy()
cumulate_df_gender = np.cumsum(df_gender == "female")

plt.plot((np.arange(len(df))*100)/len(df),
         cumulate_df_gender/np.arange(1, len(df)+1), color='black', lw=3)

I tried replicating my method for my second task, but I was unable to do so as np.cumsum() only takes one column cumulatively and I cannot take the average of a different column simultaneously.

Any insights on how to tackle this would be much appreciated.

user_12750353 · Answer

First let's figure out how to plot top-k mean.

The cumsum divided by [1,2,3,4, ..., n], is equivalent to [mean(x[:k+1]) for k in range(n)]. Let's define this as a function

def cummean(x):
    return x.cumsum() / (np.arange(1, len(x) + 1))

Then we if we are given a sorted vector we can plot the cummean of the flipped vector cummean(x[::-1]). The x axis will be simply the k value (1 + np.arange(n) converted to percentage by multiplying by 100% / n.

def plot_topk_mean(x):
    n = len(x)
    plt.plot(100 * (1 + np.arange(n)) / n, cummean(x[::-1]))

Your first query is achieved by sorting by values transforming the gender to 1 where it is female and 0 where it is male, by using ['gender'] == 'female'.

plot_topk_mean(df.sort_values(by='value')['gender'] == 'female')

For the second query we have to take the rows of df where df['gender'] == 'female', we state it as df[df['gender'] == 'female']. Just to be save we sort_values(by='value'), and finally take the age column, by ['age'].

plot_topk_mean(df[df['gender'] == 'female'].sort_values(by='value')['age'])

About the data I used:

Notice that I synthesized some data to create the plots, basically I ran this code before the lines before plotting.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
df = pd.DataFrame();
df['value'] = np.random.rand(1000) * 3000;
df['gender'] = np.where(np.random.rand(1000) < 0.1, 'female', 'male')
df['age'] = np.random.rand(1000) * 30

Graphing the cumulative average per top k% value

1 Answers

About the data I used: