Coefficients of Linear Model are way too large/low

user_2908475 0
Viewed 8003

During implementing a linear regression model on a bag of words, python returned very large/low values. train_data_features contains all words, which are in the training data. The training data contains about 400 comments of each less than 500 characters with a ranking between 0 and 5. Afterwards, I created a bag of words for each document. While trying to perform a linear regression on the matrix of all bag of words,

from sklearn import linear_model 
clf = linear_model.LinearRegression()
clf.fit(train_data_features, train['dim_hate'])

coef = clf.coef_
words = vectorizer.get_feature_names()

for i in range(len(words)):
    print(str(words[i]) + " " + str(coef[i]))

the result seems to be very strange (just an example of 3 from 4000). It shows the factors of the created regression function for the words.

btw -0.297473967075
land 54662731702.0
landesrekord -483965045.253

I'm very confused because the target variable is between 0 and 5, but the factors are so different. Most of them have very high/low numbers and I was expecting only values like the one of btw.

Do you have an idea, why the results are like they are?

scikit-learn data-analysis python-3.x numpy python
2 Answers

Check for correlated features in your data-set.

You may run into the problem if your features are highly correlated. For example expenses per customer - jan_expenses, feb_expenses, mar_expenses, Q1_expenses the Q1 feature is the sum of the jan-mar and therefore your coefficients, when trying to fit, will go 'crazy' as it will struggle to find a line that best describes the monthly features and the Q feature. Try and remove the highly correlated features and re-run.

(btw Ridge regression also solved the problem for me but I was curious as to why this happens so i dug in a bit)

user_73964300
Related