Check if random variables are independent Python

user_9531937 0
Viewed 3552

Given a table below

X Y pr
0 1 0.30
0 2 0.25
1 1 0.15
1 2 0.30

I intended to create a function to check the independence between the two variables X and Y. Note that the third column pr in the table is probability. For example P(X=0 ^ Y=1) = 0.3. Similarly, P(Y=1) = 0.3+0.15 = 0.45.

Two random variables are independent if for each possible value of x for X and for each possible value y for Y P(X =x ^ Y = y) = P(X = x)*P(Y = y).

I understand that we can use iterrows() or itertuples() to iterate over the DataFrame. But I am getting issues to get the marginal probabilities within the for loop.

Note: Marginal probabilities are P(X = x) and P(Y = y) .

Here is my basic code

import pandas as pd

#you can use this table as an example
distr_table = pd.DataFrame({'X': [0, 0, 1, 1], 'Y': [1, 2, 1, 2], 'pr': [0.3, 0.25, 0.15, 0.3]})

x_0,x_1 = distr_table.groupby('X').pr.sum()
y_1,y_2 = distr_table.groupby('Y').pr.sum()

x_u = distr_table.X.unique()
y_u = distr_table.Y.unique()
for index, row in distr_table.iterrows():
    print(row['X'], row['Y'], row['pr'])
pandas probability python
1 Answers

The marginal probabilities can be gotten easily though groupby:

>>> distr_table.groupby('Y')['pr'].sum()
Y
0    0.45
1    0.55
Name: pr, dtype: float64
>>> distr_table.groupby('X')['pr'].sum()
X
0    0.55
1    0.45
Name: pr, dtype: float64

So from here we can easilty reconstitute the independent probability with a pd.merge(…, how='cross'):

>>> cmp = pd.merge(distr_table.groupby('X', as_index=False)['pr'].sum(), distr_table.groupby('Y', as_index=False)['pr'].sum(), how='cross')
>>> cmp['indep_pr'] = cmp['pr_x'] * cmp['pr_y']
>>> cmp
   X  pr_x  Y  pr_y  indep_pr
0  0  0.55  0  0.45    0.2475
1  0  0.55  1  0.55    0.3025
2  1  0.45  0  0.45    0.2025
3  1  0.45  1  0.55    0.2475

Finally comparing it to your initial probability distribution:

>>> cmp[['X', 'Y', 'indep_pr']].merge(distr_table, on=['X', 'Y'])
   X  Y  indep_pr    pr
0  0  0    0.2475  0.30
1  0  1    0.3025  0.25
2  1  0    0.2025  0.15
3  1  1    0.2475  0.30

If you want to compare those distributions, since we’re using floating point numbers here, I’d suggest np.allclose(), i.e.

>>> np.allclose(cmp['indep_pr'], df['pr'])
False
user_13873460
Related