Residual plot diagnostic and how to improve the regression model

user_4822772 0
Viewed 1709

When creating regression models for this housing dataset, we can plot the residuals in function of real values.

from sklearn.linear_model import LinearRegression

X = housing[['lotsize']]
y = housing[['price']]

model = LinearRegression()
model.fit(X, y)

plt.scatter(y,model.predict(X)-y)

enter image description here

We can clearly see that the difference (prediction - real value) is mainly positive for lower prices, and the difference is negative for higher prices.

It is true for linear regression, because the model is optimized for RMSE (so the sign of the residual is not taken into account).

But when doing KNN

from sklearn.neighbors import KNeighborsRegressor
model = KNeighborsRegressor(n_neighbors = 3)

We can find a similar plot.

enter image description here

In this case, what interpretation can we give, and how can we improve the model.

EDIT: we can use all the other predictors, the results are similar.

housing = housing.replace(to_replace='yes', value=1, regex=True)
housing = housing.replace(to_replace='no', value=0, regex=True)
X = housing[['lotsize','bedrooms','stories','bathrms','bathrms','driveway','recroom',
        'fullbase','gashw','airco','garagepl','prefarea']]

The following graph is for KNN with 3 neighbors. With 3 neighbors, one would expect overfitting, I can't figure out why there is this trend.

enter image description here

knn linear-regression regression python
1 Answers

If you look at the fit:

plt.scatter(X,y)
plt.plot(X,model.predict(X), '--k')

enter image description here

You get negative values for higher values of y because there is a cluster of data around x=8000 with high y values that deviate a lot from what you expect.

Now if you do a knn, bear in mind your independent variable is only 1 dimensional, meaning, you are defining neighbours based on your lotsize, and you use the mean of the groups as a predictive value. For those high outlier values around x=8000, they will group together with values lower than them, making the difference negative

If you plot this out:

plt.scatter(X,y)
plt.scatter(X,model.predict(X))

enter image description here

How to improve the model? With only one predictor, there's not much you can do, maybe categorize lotsize but I doubt it changes much. Most likely you need other variables to see what is causing that bump around lotsize = 8000, then you can model the dependent variable better.

user_122584590
Related