Logistic Regression is predicting only 1 class

user_1635253 0
Viewed 3598

I'm a newbie in data science or machine learning. I try to implement code from here, but the prediction only return 1 class. Here is my code:

classification_data = data.drop([10], axis=1).values
classification_label = data[10].values

class LogisticRegression:
    def __init__(self, lr=0.01, num_iter=100000):
        self.lr = lr
        self.num_iter = num_iter
        self.weights = None
        self.bias = None

    def fit(self, X, y):
        '''Build a logistic regression classifier from the training set (X, y)'''

        n_samples, n_features = X.shape

        # init parameters
        self.weights = np.zeros(n_features)
        self.bias = 0

        # gradient descent
        for _ in range(self.num_iter):
            # approximate y with linear combination of weights and x, plus bias
            linear_model = np.dot(X, self.weights) + self.bias
            # apply sigmoid function
            y_predicted = self._sigmoid(linear_model)

            # compute gradients
            dw = (1 / n_samples) * np.dot(X.T, (y_predicted - y))
            db = (1 / n_samples) * np.sum(y_predicted - y)
            # update parameters
            self.weights -= self.lr * dw
            self.bias -= self.lr * db
        #raise NotImplementedError()

    def predict_proba(self, X):
        return self._sigmoid(X)
        raise NotImplementedError()

    def predict(self, X, threshold=0.5): # default threshold adalah 0.5
        '''Predict class value for X'''
        '''hint: you can use predict_proba function to classify based on given threshold'''
        linear_model = np.dot(X, self.weights) + self.bias
        #print (linear_model)
        y_predicted = self._sigmoid(linear_model)
        #print (self.predict_proba(linear_model))
        y_predicted_cls = [2 if i > threshold else 1 for i in y_predicted]

        return np.array(y_predicted_cls)
        raise NotImplementedError()

    def _sigmoid(self, x):
        return 1 / (1 + np.exp(-x))

when I try to call predict, it only return one class:

model.predict(classification_data, threshold=0.5)

result:

array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, etc])

this is when try to call predict_proba:

model.predict_proba(classification_data)

result:

array([[0.58826319, 0.5       , 0.52721189, ..., 0.60211507, 0.64565631,
        0.62245933],
       [0.58586893, 0.73105858, 0.52944351, ..., 0.57793101, 0.62245933,
        0.61387647],
       [0.63513751, 0.73105858, 0.57590132, ..., 0.6357912 , 0.55971365,
        0.52497919]. etc ]])

Really appreciate any help.

logistic-regression machine-learning python
2 Answers

Your algorithm, in terms of classification, works correctly but you have incorrectly implemented predict_proba.

The way you are using it right now, self._sigmoid is applied on each of the predictors separately. You want to apply it on the result of the linear model instead - the same way you are applying it in the predict function.

As you can see from the output that you have provided for predict_proba, the result is a 2D tensor instead of the expected 1D array. The correct implementation of the function is 

def predict_proba(self, X):
    linear_model = np.dot(X, self.weights) + self.bias
    return self._sigmoid(linear_model)

I have run the algorithm on the iris dataset just to see whether it works and it classifies everything correctly. You can test it yourself.

from sklearn.datasets import load_iris
from sklearn.metrics import confusion_matrix

iris = load_iris()
X = iris.data
y = iris.target
y[y == 2] = 1 # turning the problem into binary classification

log_reg = LogisticRegression()
log_reg.fit(X, y)

yproba = log_reg.predict_proba(X)
ypred = log_reg.predict(X)

cm = confusion_matrix(y, ypred)

The confusion matrix in this case is 

50  |  0
----------
0   |  100

In the above example, the model is trained on the full dataset but the same result (everything is correctly classified) is obtained even for train/test split.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

log_reg = LogisticRegression()
log_reg.fit(X_train, y_train)

cm = confusion_matrix(y_test, ypred)

In this case, the confusion matrix is

8   |  0
----------
0   |  22

Conclusion is that your algorithm works correctly. Strange behavior, if any, should probably be attributed to the data that you are feeding into the algorithm. (are you sure that it should not predict the same class for all tested observations in your case?)

Note that I have changed one more line in your code 

# from the original where you are returning 1s and 2s
y_predicted_cls = [1 if i > threshold else 0 for i in y_predicted]

for the sake of simplicity, and you can call it best practice I guess.

user_95188900

afterall it is because i use sigmoid, and it returns a value between 0 and 1, so i change i y values on the dataset to 0 and 1. now it works perfectly. but the accuracy still not so good.

user_16352530
Related