Simple Linear Regression in plain python

Question

I just started learning machine learning recently by taking Machine Learning by Andrew ng from Coursera. And I tried implementing simple linear regression in plain python without using any ML library. And this code turns out to be failing. The cost function is increasing as the loop iterates and reaches very high value. What am I doing wrong here ?

def cost_function(train_set, theta0, theta1):
  total_error = 0
  for i in range(len(train_set)):
    x = train_set[i][0]
    y = train_set[i][1]
    total_error += ((theta0 + theta1 * x) - y) ** 2
  return total_error / 2 * len(train_set)

def gradient_descent(train_set, learning_rate, theta0, theta1):
  theta0_der, theta1_der = 0, 0
  for i in range(len(train_set)):
    x = train_set[i][0]
    y = train_set[i][1]
    theta0_der += ((theta0 + theta1 * x) - y)
    theta1_der += ((theta0 + theta1 * x)- y) * x
  new_theta0 = theta0 - (1/len(train_set) * learning_rate * theta0_der)
  new_theta1 = theta1 - (1/len(train_set) * learning_rate * theta1_der)
  return new_theta0, new_theta1

def main():
  theta0, theta1 = 0, 0
  learning_rate = 0.001
  iterations = 100
  x_train = data_frame.iloc[:,0]
  y_train = data_frame.iloc[:,1]
  train_set = list(zip(x_train, y_train))[:280] # [(1, 2.444), (2, 3.555), (3, 6.444) ..... ]
  print('Initial cost: ' + str(cost_function(train_set, theta0, theta1)))
  for i in range(iterations):
    x = train_set[i][0]
    y = train_set[i][1]
    new_theta0, new_theta1 = gradient_descent(train_set, learning_rate, theta0, theta1)
    theta0 = new_theta0
    theta1 = new_theta1
    print([theta0, theta1])
  print('Final cost: ' + str(cost_function(train_set, theta0, theta1)))

main()

Admin · Answer

You have set the learning rate too high, try changing it to 0.0001.

But, You can directly implement Simple linear regression by using its Closed-form equation which is:

Implementing this with python is pretty easy you can do it like this:-

class LinearRegression:
    def fit(self, X, y):
        ones = np.ones(len(X)).reshape(-1, 1)
        X = np.concatenate((ones, X), axis=1)

        B = np.matmul(np.linalg.pinv(np.matmul(X.T, X)), np.matmul(X.T, y))

        self.slope = B[1:]
        self.intercept = B[0]

    def predict(self, X):
        self.predicted = np.dot(X, self.slope) + self.intercept
        return self.predicted

The fit function is talking X and y values and computing Beta (by the above formula using NumPy). Beta is a matrix in which the first index value is intercept and the rest all slopes!

The prediction function takes the 2D array and then computes the prediction!

Simple Linear Regression in plain python

1 Answers