Relationship between datapoints and order of polynomial in least square method?

Question

So I implemented the Least Square method for curve fitting, but I am not sure what the relationship between the number of the (x,y) values I want to fit to and the order of the polynomial is?

What happens if I try to fit a higher order polynomial to a small number of points rather than lower order one?

Will at some point the fitted function go through all the data points when the order is big enough, such that the regression becomes an interpolation?

At what order of polynomial will the regression stop to "get better"?

Answer 1

A polynomial of degree $n$ can interpolate $n+1$ data points perfectly. In this situation the least squares method will yield the interpolating polynomial. The reason beeing that the method returns the polynomial with the smallest least squares error, which is zero in this case for the interpolating polynomial. If you have less than $n+1$ data points, you will probably end up with a rank-defficient matrix in your algorithm and your method will break down.

Answer 2

There is no general answer to your question. You need to, in a case by case basis, visualize the data and make and informed choice of fitting functions. In a limit situation, if you don't have repeated x's, you can use the interpolating polynomial, that will fit the data exactly.