SGDRegressor() constantly not increasing validation performance

EDIT: Regarding your output, my guess is that your results are so close for the validation set because a linear model like SGDregressor tends to underfit on complex data

To see this you can check the weights outputted by the model at every iteration. You'll see that they are the same or really close

To enhance variability in the output you need to introduce non linearity and complexity

You are obtaining what is referred as "Bias" in machine learning (in contraposition to the "variance")

I think I got it now.

SamAmani In the end I think that the problem is underfitting. And the fact that you are using incremental sizes of the dataset. The model underfit quite fast (which means that the model is stuck at the beginning to a more or less fixed model)

Only the first training output a different result for the test set because it hasn't reached the final model, more or less

The underlying variability is in the incremental training sets. Simply speaking the test results are a more accurate estimate of the performance of the underfitted model. And adding training sample will lead in the end to near results between test and training without improving too much.

You can check the fact that are the incremental datasets of the training to be different from the test set. What you did wrong was to check the stats on all the training set

First of all, why are you training on incremental training set size? The strange results are due to the fact that you are training in an incremental fashion your dataset.

When you do this:

for m in my_rng:
    modelSGD = SGDRegressor(alpha=0.00001, penalty='l1')
    modelSGD.fit(X_train[:m], y_train[:m])
    [...]

you are basically training your model in incremental fashion, with this incremental sizes:

for m in range(10, 180001, 30000):
    print(m)


10
30010
60010
90010
120010
150010

If you are trying to make mini-batch gradient descent, you should split your dataset in independent batches instead of making incremental batches. Something like this:

previous = 0
for m in range(30000, 180001, 30000):
    modelSGD.partial_fit(X_train[previous:m], y_train[previous:m])
    previous = m

# training set ranges
0 30000
30000 60000
60000 90000
90000 120000
120000 150000
150000 180000

Also note that I am using partial_fit method, instead of fit (because I am not retraining the model from zero and I am making only a step, iteration of the gradient descent), and I am not going to initialize a new model every time (my sgd initialization is out of the for loop). The full code should be something like this:

my_rng = range(0 ,len(X_train), 30000)
previous = 0
modelSGD = SGDRegressor(alpha=0.00001, penalty='l1')
for m in my_rng:
    modelSGD.partial_fit(X_train[previous:m], y_train[previous:m])
    ypred_train = modelSGD.predict(X_train[previous:m])
    ypred_test = modelSGD.predict(X_test)
    mse_train = mean_squared_error(y_train[previous:m], ypred_train)
    mse_test = mean_squared_error(y_test, ypred_test)
    scores_train.append(mse_train)
    scores_test.append(mse_test)

In this way you are simulating one epoch mini-batch stochastic gradient. To make more epochs an outer loop is needed

From sklearn:

SGD allows minibatch (online/out-of-core) learning via the partial_fit method. For best results using the default learning rate schedule, the data should have zero mean and unit variance.

Details here

I think what you are trying to do is see the effect of increase in dataset size on the performance of the model. If that is the case as you can see, as the size of data increases the error increases and it reaches stage where with enough data it starts learning (slight decrease in train error) and eventually stabilizing as increase in dataset does not matter now. Also it does not over fit as by default the SGDRegressor runs for only 1000 epochs.

I suggest you normalize the data before fitting (0 mean and unit variance).

Test Code:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import SGDRegressor
from sklearn.metrics import mean_squared_error
import pandas
import matplotlib.pyplot as plt

X = np.random.randn(336144+144063, 10)
y = np.sum(X, axis=1) + np.random.randn(336144+144063)
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3, random_state=85, shuffle=True)

print (X_train.shape, X_test.shape)

scores_test = []
scores_train = []

my_rng = range(5000,len(X_train),10000)
for m in my_rng:
    modelSGD = SGDRegressor(alpha=0.00001, penalty='l1')
    modelSGD.fit(X_train[:m], y_train[:m])
    
    ypred_train = modelSGD.predict(X_train[:m])
    ypred_test = modelSGD.predict(X_test)
    mse_train = mean_squared_error(y_train[:m], ypred_train)
    mse_test = mean_squared_error(y_test, ypred_test)
    scores_train.append(mse_train)
    scores_test.append(mse_test)

plt.plot(scores_train)
plt.plot(scores_test)

In case you are trying batch gradient decent follow answer by @Nikaido