numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares on first run only

user_10762433 0
Viewed 4083

I have been wrestling with a known and documented SVD converge issue. Having read up on similar issues raised by others, I have double checked my data and reduced this to a tiny DataFrame - 10 rows/2 columns only - both float64's. There are definitely no NaN or infinities.

On first run, I pause at the offending line via breakpoint. First time I manually execute the next (offending) line i get a console error (see below) - but on subsequent runs it resolves without errors!! I am using numpy 1.19.1

I would greatly appreciate thoughts or ideas on how to resolve this. It is driving me nuts and its shaken my confidence.

Thanks in advance for any suggestions. I really want to get to the bottom of this.

Luthor

The code:

# Simplifying the df
df = df.head(10)
df = df[['dti','close']]
print(df)
ltt2_poly = np.polyfit(df['dti'] - df['dti'][0], df['close'], 2)

At runtime:

pydev debugger: process 36368 is connecting
Connected to pydev debugger (build 202.6948.78)
Importing local settings
   dti  close
0    0  11.28
1    3  11.35
2    4  11.10
3    5  10.95
4    6  11.07
5    7  11.45
6   10  11.46
7   11  11.46
8   12  11.74
9   13  11.96

**ltt2_poly = np.polyfit(df['dti'] - df['dti'][0], df['close'], 2)**

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2020.1.1\plugins\python\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "<__array_function__ internals>", line 5, in polyfit
  File "C:\Users\luthor\PycharmProjects\MC\venv\lib\site-packages\numpy\lib\polynomial.py", line 629, in polyfit
    c, resids, rank, s = lstsq(lhs, rhs, rcond)
  File "<__array_function__ internals>", line 5, in lstsq
  File "C:\Users\luthor\PycharmProjects\MC\venv\lib\site-packages\numpy\linalg\linalg.py", line 2306, in lstsq
    x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj)
  File "C:\Users\luthor\PycharmProjects\MC\venv\lib\site-packages\numpy\linalg\linalg.py", line 100, in _raise_linalgerror_lstsq
    raise LinAlgError("SVD did not converge in Linear Least Squares")
numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares

*In the SAME debug session:*

**ltt2_poly = np.polyfit(df['dti'] - df['dti'][0], df['close'], 2)**    now works!!!

print(ltt2_poly)
[ 1.00902938e-02 -8.70161869e-02  1.13247743e+01]

print(np.version) 1.19.1

To add insult to injury, when I reduce the df to between 5-9 - it works without the failure. What am I missing??

pandas numpy python
3 Answers

I don't have a solution to that, but I can tell you that you're not alone. I have the same bug.

I "fixed" it by simply wrapping the NumPy function in a while-try statement.

while True: try: NumPy-function break except: continue

user_140604250

Try using a conda environment with numpy 1.18

This version seems to predate these kinds of errors and it worked for me

user_143565660

In my case, the following code may lead to exactly this non-convergence error:

pipe = make_pipeline(StandardScaler(), LinearRegression())
pipe.fit(trainingDf, ySer)

This happens on rare occasions, even when the data does not contain Nan or infinite data points. However, as soon as I removed the normalization, the code runs fine:

pipe = make_pipeline(StandardScaler(with_std=False), LinearRegression())
pipe.fit(trainingDf, ySer)

Similar in the past before pipelines with:

regr = LinearRegression(normalize=True)
regr.fit(XDf.to_numpy(),ySer.to_numpy())

and:

regr = LinearRegression(normalize=False)
regr.fit(XDf.to_numpy(),ySer.to_numpy())
user_74161020
Related