PyTorch NN and linear regression: Why bigger Training Data gives worst results than a much smaller dataset?

I am trying to create a bare minimum PyTorch example only for learning purpose. I found out that my PyTorch code works fine with a really small training data but as soon as I increase the input data size it stops working. This seems very counterintuitive, ideally bigger training data size should give better results.
[ I have intentionally not used Object Oriented Paradigm as I am trying to first learn the core functionality hence keeping the code to a bare minimum. ]

import numpy as np
import torch 

x_train = np.float32(np.random.rand(25,1)*10)

#Synthesize training data; we will verify the weights and bias later with the trained model
def synthesize_output(input):
    return (1.29*input[0] + 13)

y_train = np.array([synthesize_output(row) for row in x_train]).reshape(-1,1)

X_train = torch.from_numpy(x_train) 
Y_train = torch.from_numpy(y_train)

learning_rate = 0.001

# Initialize Weights and Bias to random starting values
w = torch.rand(1, 1, requires_grad=True) 
b = torch.rand(1, 1, requires_grad=True) 

for iter in range(1, 4001):

    #forward pass : predict values    
    y_pred = X_train.mm(w).clamp(min=0).add(b)

    #find loss  
    loss = (y_pred - Y_train).pow(2).sum() 

    #Backword pass for computing gradients   
    loss.backward()
    
    #Just printing the loss to see how it is changing over the iterations
    if (iter % 100) == 0:
        print(f"Iter: {iter}, Loss={loss}")

    #Manually updating weights       
    with torch.no_grad():
        w -= learning_rate * w.grad
        b -= learning_rate * b.grad
        w.grad.zero_()
        b.grad.zero_()
               
#finally check the weight and bias 
print(f"Weights: {w} \n\nBias: {b}")

Above code works as it is but as soon as I increase ( just doubling ) the data size it stops working.
x_train = np.float32(np.random.rand(50,1)*10)
Unlike the above code the basic sklearn sample I had created however seems to work fine even with a much larger input dataset.

import numpy as np
from sklearn.linear_model import  LinearRegression

x_train = np.float32(np.random.rand(2000000,1)*10)

def synthesize_output(input):
    return (1.29*input[0] + 13)

y_train = np.array([synthesize_output(row) for row in x_train]).reshape(-1,1)

lm = LinearRegression()

lm.fit(x_train, y_train)

#finally check the weight and constant 

lm.score(x_train, y_train)
print(f"Weight: {lm.coef_}")
print(f"Bias: {lm.intercept_}")

Why is PyTorch not able to handle large input data like sklearn?
When I use input very large size ( > 5000 ) training datasize the loss goes to a NaN. How do we typically work around this problem?