NaN, inf or invalid value detected in weights detected error when training statsmodels GLM model

xxww1201_13930340 0
Viewed 3619

I am using my data to train a GLM model (poisson family) using python statsmodels package. The data I have contains both numeric values and categorical values. I did standardization for numeric values and one-hot-encoding on categorical values (drop the first level). When I fit the data into the model, I got the following exceptions:

~/miniconda3/envs/losscost/lib/python3.7/site-packages/insite/losscost/losscost.py in evaluate(self, x, control, peril_descs)
    271                     family=sm.families.Poisson(link=sm.families.links.log()),
    272                 )
--> 273                 freq_fitted = freq_glm.fit()
    274                 freq_results[name].append(freq_fitted)
    275 

~/miniconda3/envs/losscost/lib/python3.7/site-packages/statsmodels/genmod/generalized_linear_model.py in fit(self, start_params, maxiter, method, tol, scale, cov_type, cov_kwds, use_t, full_output, disp, max_start_irls, **kwargs)
   1025             return self._fit_irls(start_params=start_params, maxiter=maxiter,
   1026                                   tol=tol, scale=scale, cov_type=cov_type,
-> 1027                                   cov_kwds=cov_kwds, use_t=use_t, **kwargs)
   1028         else:
   1029             self._optim_hessian = kwargs.get('optim_hessian')

~/miniconda3/envs/losscost/lib/python3.7/site-packages/statsmodels/genmod/generalized_linear_model.py in _fit_irls(self, start_params, maxiter, tol, scale, cov_type, cov_kwds, use_t, **kwargs)
   1163             wls_mod = reg_tools._MinimalWLS(wlsendog, wlsexog,
   1164                                             self.weights, check_endog=True,
-> 1165                                             check_weights=True)
   1166             wls_results = wls_mod.fit(method=wls_method)
   1167             lin_pred = np.dot(self.exog, wls_results.params)

~/miniconda3/envs/losscost/lib/python3.7/site-packages/statsmodels/regression/_tools.py in __init__(self, endog, exog, weights, check_endog, check_weights)
     46         if check_weights:
     47             if not np.all(np.isfinite(w_half)):
---> 48                 raise ValueError(self.msg.format('weights'))
     49 
     50         if check_endog:

ValueError: NaN, inf or invalid value detected in weights, estimation infeasible.

I tried to train only on numeric values and it works fine. What could be the reason to cause this issue?

statsmodels
1 Answers

Can you add a keyword argument to your fit call like this and see if it helps:

model = sm.GLM(...)
model.fit(method="lbfgs")

I think the Inf/NaN is in the IRLS weights. IRLS is slightly less robust than direct optimization.

Also, make sure your design matrix is not singular:

model = sm.GLM(...)
u, s, vt = numpy.linalg.svd(model.exog, 0)
print(s)

All elements of s (the singular values) should be strictly positive.

If you continue to have troubles, what is the sample size and dimension of your model?

Kerby_Shedden_19417450
Related