Correlation between binary variables in pandas

Admin 1
Viewed 679

I am trying to calculate the correlation between binary variables using Cramer's statistics:

def cramers_corrected_stat(confusion_matrix):

    chi2 = ss.chi2_contingency(confusion_matrix)[0]
    n = confusion_matrix.sum()
    phi2 = chi2/n
    r,k = confusion_matrix.shape
    phi2corr = max(0, phi2 - ((k-1)*(r-1))/(n-1))    
    rcorr = r - ((r-1)**2)/(n-1)
    kcorr = k - ((k-1)**2)/(n-1)
    return np.sqrt(phi2corr / min( (kcorr-1), (rcorr-1)))

However I do not know how to apply the code above within my dataset:

    CL  UP  NS  P   CL_S
480  1  0   1   0   1
1232 1  0   1   0   1
2308 1  1   1   0   1
1590 1  0   1   0   1
497  1  1   0   0   1
... ... ... ... ... ...
1066    1   1   1   0   1
1817    1   0   1   0   1
2411    1   1   1   0   1
2149    1   0   1   0   1
1780    1   0   1   0   1

I would appreciate your help in guiding me

confusion-matrix pandas correlation python
1 Answers

The function you made is not proper for your dataset.

So, use the follow function cramers_V(var1,var2) given as follows.

from scipy.stats import chi2_contingency
def cramers_V(var1,var2):
  crosstab =np.array(pd.crosstab(var1,var2, rownames=None, colnames=None)) # Cross table building
  stat = chi2_contingency(crosstab)[0] # Keeping of the test statistic of the Chi2 test
  obs = np.sum(crosstab) # Number of observations
  mini = min(crosstab.shape)-1 # Take the minimum value between the columns and the rows of the cross table
  return (stat/(obs*mini))

example code using the function is as fllows.

cramers_V(df["CL"], df["NS"])

If you want to calculate all possible pairs of your dataset, use this code.

import itertools
for col1, col2 in itertools.combinations(df.columns, 2):
    print(col1, col2, cramers_V(df[col1], df[col2]))
user_131768960
Related