Implementing OLS in matrix form

Question

I'm having problems implementing this exercise from a quantitative economics course.

Here's my code:

N = 50
M = 20

a = 0.1
b = 0.2 
c = 0.5
d = 1.0
σ = 0.1 

estimates = zeros(M, 5)

for i ∈ 1:M
    x₁ = Vector{BigFloat}(randn(N))
    x₂ = Vector{BigFloat}(randn(N))
    w = Vector{BigFloat}(randn(N))
    
    # Derive y vector (element wise operations)
    y = a*x₁ .+ b.*(x₁.^2) .+ c.*x₂ .+ d .+ σ.*w
    
    # Derive X matrix
    X = [x₁ x₁ x₂ fill(d, (N, 1)) w]
    
    # Implementation of the formula β = inv(XᵀX)Xᵀy
    estimates[i, :] = (X'*X)\X'*y
end

histogram(estimates, layout=5, labels=["a", "b", "c", "d", "σ"])

I get a SingularException(5) error, as the matrix X'X has a determinant of 0 and has no inverse. My question is, where have I gone wrong in this exercise? I heard that a reason the determinant might be zero is floating point inaccuracy, so I made the random variables BigFloats to no avail. I know the mistake I'm making isn't very complicated but I'm lost. Thank you!

user_9957710 · Answer

Your X should be

X = [x₁ x₁*x₁ x₂ fill(d, (N, 1))]

Explanation

It looks that you are trying to test OLS to estimate the parameters of the model:

y = α₀ + α₁x₁ + α₁₁x₁² + α₂x₂ + ϵ

where α₀, is the intercept of the model, α₁, α₁₁, α₂ are parameters for explanatory variables, and ϵ is the random error with the expected value 0 and variance σ². Hence the structure of X must match your case.

Putting the α₁ twice you introduced co-linearity and got the error.

You also do not want to "estimate" the parameter for ϵ because it represents the randomness.

Implementing OLS in matrix form

1 Answers

Explanation