Handling Python IndexError from regressor_OLS.pvalues

Viewed 58

I want to reduce the number of features of a sparse matrix as below, but after passing a range I get an IndexError. It seems that after 2, access to regressor_OLS.pvalues is not possible because the print command also leads to the same error. Do you have any idea how to solve this?

import statsmodels.formula.api as sm

def backwardElimination(x, Y, sl, columns):
    numVars = len(x[0])
    pvalue_removal_counter = 0

    for i in range(0, numVars):
        print(i, 'of', numVars)
        regressor_OLS = sm.OLS(Y, x).fit()
        maxVar = max(regressor_OLS.pvalues).astype(float)

        if maxVar > sl:  # sl = 0.05
            for j in range(0, numVars - i):
                # print (regressor_OLS.pvalues)
                if (regressor_OLS.pvalues[j].astype(float) == maxVar):
                    x = np.delete(x, j, 1)
                    pvalue_removal_counter += 1
                    columns = np.delete(columns, j)

    regressor_OLS.summary()
    return x, columns

Output:

0 of 1970
1 of 1970
2 of 1970
Traceback (most recent call last):
  File "main.py", line 142, in <module>
    selected_columns)
  File "main.py", line 101, in backwardElimination
    if (regressor_OLS.pvalues[j].astype(float) == maxVar):
IndexError: index 1967 is out of bounds for axis 0 with size 1967
0 Answers
Related