I splitted my dataframe into two, one with having null values of a specifc column and other is null free. Then I selected a couple of metrics and tried to present them in a comparable way. I have already created the output. However, i believe this code is not very attractive and I would like to know if there is an easier way to get the output I wanted. and If I can even make it two metrics per row it would be perfect.
dataframe1 (with null):
| CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | DIS | RAD | TAX | PTRATIO |
|---|---|---|---|---|---|---|---|---|---|---|
| 0.02731 | 0.0 | 7.07 | 0.0 | NaN | 6.421 | 78.9 | 4.9671 | 2.0 | 242.0 | 17.8 |
| 0.08829 | 12.5 | 7.87 | 0.0 | NaN | 6.012 | 66.6 | 5.5605 | 5.0 | 311.0 | 15.2 |
| 0.22489 | 12.5 | 7.87 | 0.0 | NaN | 6.377 | 94.3 | 6.3467 | 5.0 | 311.0 | 15.2 |
dataframe1 (without null):
| CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | DIS | RAD | TAX | PTRATIO |
|---|---|---|---|---|---|---|---|---|---|---|
| 0.32711 | 0.0 | 8.07 | 0.0 | 0.538 | 6.851 | 75.9 | 3.9371 | 1.0 | 242.0 | 14.2 |
| 0.21859 | 16.5 | 8.85 | 0.0 | 0.469 | 6.124 | 61.6 | 6.5601 | 3.0 | 242.0 | 11.6 |
| 0.42282 | 13.5 | 8.85 | 0.0 | 0.458 | 6.732 | 97.3 | 5.3627 | 3.0 | 311.0 | 18.1 |
print('min', 'Null Free', 'With Nulls', sep='\t')
for i in housing.columns:
if i == 'NOX':
pass
else:
print(i, "{:.2f}".format(housing_nullfree.min().T.loc[i]),"{:.2f}".format(housing_nulls.min().T.loc[i]), sep="\t\t")
print()
print('Perc25', 'Null Free', 'With Nulls', sep='\t')
for i in housing.columns:
if i == 'NOX':
pass
else:
print(i, "{:.2f}".format(np.percentile(housing_nullfree[i], 25)),"{:.2f}".format(np.percentile(housing_nulls[i], 25)), sep="\t\t")
print()
print('mean', 'Null Free', 'With Nulls', sep='\t')
for i in housing.columns:
if i == 'NOX':
pass
else:
print(i, '{:.2f}'.format(housing_nullfree.mean().T.loc[i]),'{:.2f}'.format(housing_nulls.mean().T.loc[i]), sep="\t\t")
print()
print('median', 'Null Free', 'With Nulls', sep='\t')
for i in housing.columns:
if i == 'NOX':
pass
else:
print(i, '{:.2f}'.format(np.median(housing_nullfree[i])), '{:.2f}'.format(np.median(housing_nulls[i])), sep="\t\t")
print()
print('Perc75', 'Null Free', 'With Nulls', sep='\t')
for i in housing.columns:
if i == 'NOX':
pass
else:
print(i, "{:.2f}".format(np.percentile(housing_nullfree[i], 75)),"{:.2f}".format(np.percentile(housing_nulls[i], 75)), sep="\t\t")
print()
print('Perc90', 'Null Free', 'With Nulls', sep='\t')
for i in housing.columns:
if i == 'NOX':
pass
else:
print(i, "{:.2f}".format(np.percentile(housing_nullfree[i], 90)),"{:.2f}".format(np.percentile(housing_nulls[i], 90)), sep="\t\t")
print()
print('max', 'Null Free', 'With Nulls', sep='\t')
for i in housing.columns:
if i == 'NOX':
pass
else:
print(i, "{:.2f}".format(housing_nullfree.max().T.loc[i]),"{:.2f}".format(housing_nulls.max().T.loc[i]), sep="\t\t")
my output:
min Null Free With Nulls
CRIM 0.01 0.02
ZN 0.00 0.00
INDUS 0.46 1.25
CHAS 0.00 0.00
RM 3.56 4.65
AGE 2.90 6.50
DIS 1.13 1.20
RAD 1.00 1.00
TAX 188.00 187.00
PTRATIO 12.60 13.00
B 0.32 27.49
LSTAT 1.73 2.96
NOX_cat 0.00 1.00
Perc25 Null Free With Nulls
CRIM 0.08 0.06
ZN 0.00 0.00
INDUS 5.32 4.05
CHAS 0.00 0.00
RM 5.88 6.01
AGE 45.70 42.40
DIS 2.06 2.51
RAD 4.00 4.00
TAX 281.00 276.00
PTRATIO 17.40 16.60
B 374.68 378.95
LSTAT 7.18 6.90
NOX_cat 0.00 1.00
mean Null Free With Nulls
CRIM 3.78 2.38
ZN 10.70 16.20
INDUS 11.42 9.09
CHAS 0.07 0.08
RM 6.27 6.36
AGE 68.96 65.77
DIS 3.70 4.49
RAD 9.59 9.28
TAX 410.89 388.90
PTRATIO 18.47 18.33
B 353.87 377.10
LSTAT 12.89 10.90
NOX_cat 0.00 1.00
median Null Free With Nulls
CRIM 0.29 0.15
ZN 0.00 0.00
INDUS 9.90 7.38
CHAS 0.00 0.00
RM 6.18 6.31
AGE 78.10 72.70
DIS 3.09 3.95
RAD 5.00 5.00
TAX 334.00 307.00
PTRATIO 19.10 18.60
B 391.13 392.52
LSTAT 11.66 9.45
NOX_cat 0.00 1.00
Perc75 Null Free With Nulls
CRIM 3.69 1.61
ZN 12.50 25.00
INDUS 18.10 13.92
CHAS 0.00 0.00
RM 6.59 6.79
AGE 94.30 93.30
DIS 5.10 5.72
RAD 24.00 8.00
TAX 666.00 432.00
PTRATIO 20.20 20.20
B 396.24 395.59
LSTAT 17.12 14.10
NOX_cat 0.00 1.00
Perc90 Null Free With Nulls
CRIM 11.00 7.99
ZN 40.00 70.00
INDUS 19.58 18.10
CHAS 0.00 0.00
RM 7.16 7.04
AGE 98.80 97.40
DIS 6.75 7.83
RAD 24.00 24.00
TAX 666.00 666.00
PTRATIO 20.90 20.20
B 396.90 396.90
LSTAT 23.28 18.80
NOX_cat 0.00 1.00
max Null Free With Nulls
CRIM 88.98 24.39
ZN 100.00 90.00
INDUS 27.74 18.10
CHAS 1.00 1.00
RM 8.78 8.25
AGE 100.00 100.00
DIS 10.71 12.13
RAD 24.00 24.00
TAX 711.00 666.00
PTRATIO 22.00 22.00
B 396.90 396.90
LSTAT 37.97 28.28
NOX_cat 0.00 1.00