Python polynomial regression values ceiling <0,1>

user_12410677 0
Viewed 46

I am trying to find the best curve that will describe my data. my data are stored in numpy arrays of t and dur they are both in values only from 0-1. However the best fit I get according to R**2 score is this yellow line with score of 0.979388 which doesn't fit my data because it is way off from expected values when it is well above 1 in Y axis:

enter image description here

t = [1.0, 1.0, 1.0, 1.0, 1.0, 0.33695652173913043, 0.010869565217391304, 1.0, 0.018518518518518517, 1.0, 1.0, 1.0, 1.0, 1.0, 0.005076142131979695, 1.0, 1.0, 1.0, 1.0, 0.03225806451612903, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 0.25, 1.0]
dur = [1.0, 1.0, 1.0, 1.0, 0.9999999999999998, 0.2688679245283018, 0.2688679245283018, 1.0, 0.46692607003891046, 1.0, 1.0, 1.0, 1.0, 1.0, 0.4444444444444444, 1.0, 1.0, 1.0, 1.0, 0.34210526315789475, 1.0, 1.0, 1.0, 1.0, 1.0, 0.4714285714285715, 0.4714285714285715, 1.0]

#polynomial curve fitting
mymodel1 = np.poly1d(np.polyfit(t, dur, 1))
mymodel2 = np.poly1d(np.polyfit(t, dur, 2))
mymodel3 = np.poly1d(np.polyfit(t, dur, 3))
mymodel4 = np.poly1d(np.polyfit(t, dur, 4))

#polynomial score
p1 = r2_score(dur, mymodel1(t))
p2 = r2_score(dur, mymodel2(t))
p3 = r2_score(dur, mymodel3(t))               
p4 = r2_score(dur, mymodel4(t))

#append results of R**2 to list of tuples from which I extract best score
fit = []
fit.append(p1)
fit.append(p2)
fit.append(p3)
fit.append(p4)
fitname = []
fitname.append('p1')
fitname.append('p2')
fitname.append('p3')
fitname.append('p4')

#append best result value
resultValue.append(max(fitTuple,key=lambda item:item[0])[0])

#append best result name
resultName.append(max(fitTuple,key=lambda item:item[0])[1])

#plot values from regression models
myline = np.linspace(0, 1, 100) 
plt.plot(myline, mymodel1(myline),color = "black")
plt.plot(myline, mymodel2(myline),color = "black")
plt.plot(myline, mymodel3(myline),color = "black")
plt.plot(myline, mymodel4(myline),color = "yellow")
pandas curve-fitting statistics numpy python
1 Answers

This is what is called "overfitting". If you fit overly complex models to your data, the models will usually have very high R^2 and indeed meet the data points of the data you use for training quite well, but are clearly not the appropriate choice, as can be seen when trying to fit new data, e.g. they don't interpolate well. And fitting polynomials of high degree is usually taken as a standard example for overfitting.

If you want to stick with polynomial models, you should think about what the least complex model, i.e. in this case the lowest degree polynomial, is, that you would still think appropriate for your data. In your case, quadratic seems OK.

One usually employs more sophisticated methods for regression, like those provided in e.g. scikit learn, which can help you find the right model (e.g. via cross-validation) and also provide regularization techniques. For model selection, see here.

user_6591560
Related