Difference in output when using panda.read_csv vs numpy.loadtext

user_11706289 0
Viewed 363

I am currently doing Andrew Ng's course on ML using Python, I googled to see if my outputs matched others but I realised just because of using numpy instead of panda my final output shifted

Expected : 2105448288.6292474 My output : 2064911681.618526

is this much discrepancy often observed when using different modules?

Reference Code:

import numpy as np
import pandas as pd
data = pd.read_csv('ex1data2.txt', sep = ',', header = None)
X = data.iloc[:,0:2] # read first two columns into X
y = data.iloc[:,2] # read the third column into y
m = len(y) # no. of training samples
data.head()

X = (X - np.mean(X))/np.std(X)

ones = np.ones((m,1))
X = np.hstack((ones, X))
alpha = 0.01
num_iters = 400
theta = np.zeros((3,1))
y = y[:,np.newaxis]

def computeCostMulti(X, y, theta):
    temp = np.dot(X, theta) - y
    return np.sum(np.power(temp, 2)) / (2*m)
J = computeCostMulti(X, y, theta)

def gradientDescentMulti(X, y, theta, alpha, iterations):
    m = len(y)
    for _ in range(iterations):
        temp = np.dot(X, theta) - y
        temp = np.dot(X.T, temp)
        theta = theta - (alpha/m) * temp
    return theta
theta = gradientDescentMulti(X, y, theta, alpha, num_iters)

J = computeCostMulti(X, y, theta)
print(J)

My Code:

import numpy as np
from matplotlib import pyplot as plt
data = np.loadtxt("ex1data2.txt", delimiter = ",", dtype = 'int')
x = data[:, 0:2]
y = data[:, 2]
m = len(y)

x = x.reshape(m,2)
y = y.reshape(m,1)
one = np.ones((m,1))

X = np.matrix(x)

X = (X - np.mean(X))/np.std(X)

X = np.concatenate((one,X), axis = 1)

theta = np.zeros((3, 1))

def cc(theta, X, y):
    A = np.dot(X,theta)-y
    return float(((1/(2*m)) * np.dot(A.T, A)))


def gd(theta, X, y, alpha, iterations):
    for i in range(iterations):
        h = np.dot(X, theta)-y
        h = np.dot(X.T, h)
        theta = theta - (alpha/m) * h
    return theta

theta = (gd(theta, X, y, 0.01, 400))

print(cc(theta, X, y))
numpy python
1 Answers

The difference is due to how matrix operations are applied to a pandas dataframe vs numpy matrix.

e.g., pandas dataframe:

data = pd.read_csv('ex1data2.txt', header=None)
X_df = data.iloc[:, 0:2]
np.mean(X_df)

# 0    2000.680851
# 1       3.170213
# dtype: float64

vs numpy matrix:

data = np.loadtxt('ex1data2.txt', delimiter=',', dtype='int')
x = data[:, 0:2]
X_mat = np.matrix(x)
np.mean(X_mat)

# 1001.9255319148937

Here axis=0 is required to replicate the pandas behavior:

np.mean(X_mat, axis=0)

# matrix([[2000.68085106,    3.17021277]])

I haven't walked through every line of your numpy version to debug every discrepancy, but this is the underlying cause. The data loaded from pd.read_csv and np.load_txt are equivalent. It's the subsequent matrix operations that differ due to axis handling.

user_131383640
Related