Is there a way to analyze the impact/correlation of categorical variables to the label in python?

user_13999522 0
Viewed 229

The dataset I have is a lot larger than this (about 3000 rows * 50 columns), I'm just going to put a sample here. It's a dataframe including information for every line. Basically, I intended to analyze the attribute of each label, like Level 3 might have a higher annual income; or what contribute to higher level. What statistical functions might be a good fit to analyze it? I'm trying with sklearn.preprocessing.OrdinalEncoder() to labelize every category variable and trying something like stats.chi2.ppf() or correlation matrix. Not sure if they work out in my case.

example = pd.DataFrame(
    {
        "Degree": ['Graduate', 'Undergraduate', 'Undergraduate', 'Graduate', 'Undergraduate', 'Doctorate'],
        "Age": ['Age 26-35','Age 18-25','Age 18-25','Age 18-25', 'Age 26-35', 'Older than 35'],
        "Location": ['VA','DC','DC','CA','DC','MA'],
        "Gender": ['male','male','female','male','male','female'],
        "Annual Income": ['\$5,001 - \$10,000','<$5,000','\$15,001 - \$25,000','>\$50,000','<\$5,000','\$15,001 - \$25,000'],
        "Level": [0,1,2,0,0,3],
    }
)

Degree  Age Location    Gender  Annual Income   Level
0   Graduate    Age 26-35   VA  male    $5,001 - $10,000    0
1   Undergraduate   Age 18-25   DC  male    <$5,000 1
2   Undergraduate   Age 18-25   DC  female  $15,001 - $25,000   2
3   Graduate    Age 18-25   CA  male    >$50,000    0
4   Undergraduate   Age 26-35   DC  male    <$5,000 0
5   Doctorate   Older than 35   MA  female  $15,001 - $25,000   3

Open to any ideas and comments.

pandas data-analysis stat dataframe python
2 Answers

In the correlation I suggest use seaborn.

Heatmaps are used to show relationships between two variables, one plotted on each axis. By observing how cell colors change across each axis, you can observe if there are any patterns in value for one or both variables.

https://chartio.com/learn/charts/heatmap-complete-guide/

import seaborn as sns
sns.heatmap(example[['Level']])

enter image description here

However for heatmap - there is need for having integer - so Annual Income and Age can be transformed into integer.

There is some approximation (getting first number from income & age - not range - also there can be use. .mean()):

example['Income'] = example['Annual Income'].str.extract('(\d+)')
example['Age'] = example['Age'].str.extract('(\d+)')
example['Income'] = pd.to_numeric(example['Income'])
example['Age'] = pd.to_numeric(example['Age'])


import seaborn as sns
sns.heatmap(example[['Level', 'Income', 'Age']])

enter image description here

When the annual income and age is integer - there is also option for .corr() function:

example.corr()

enter image description here

user_94559020 
import pandas as pd
example = pd.DataFrame(
    {
        "Degree": ['Graduate', 'Undergraduate', 'Undergraduate', 'Graduate', 'Undergraduate', 'Doctorate'],
        "Age": ['Age 26-35','Age 18-25','Age 18-25','Age 18-25', 'Age 26-35', 'Older than 35'],
        "Location": ['VA','DC','DC','CA','DC','MA'],
        "Gender": ['male','male','female','male','male','female'],
        "Annual Income": ['\$5,001 - \$10,000','<$5,000','\$15,001 - \$25,000','>\$50,000','<\$5,000','\$15,001 - \$25,000'],
        "Level": [0,1,2,0,0,3],
    }
)
unique_items = []
for key in example:
    unique_items.append(example[key].unique())
for item in unique_items:
    print(item)
# figure out how to sort each unique item,
# for example, degree= by more education
#              income = ascending
#              level = ascending , etc
# now use the index as the value and you can start to do math and pictures
# Analyze for me means:
# Now what you would do is pick any two and scatterplot it to see if there is a relationship
# then pick all pairs for any one and make a collage of thumbnail scatterplots
# then measure correlation or other math properties that put them in groupings you like
# think of this like categorizing galaxies, 
# straight lines sloping up is one type that would be high on the list
# but randomness is another type and some might look like butterflies
# then sort by correlation and groupings to show all the strongest top 100 list
# good luck ;)
user_145600400
Related