Let's say I have a set of data points $(x_i,y_i), i=1,2,...,N$, and I want to approximate it using a polynomial $p(x)=\sum_{i=0}^n a_i x^i$ with a least squares fit (so $n<N$).
I know that the coefficient $R^2$ is a measure for goodness of fit. But as I increase the order $n$ of the polynomial $p$, $R^2$ approaches $1$. (In the extreme case when $n=N$ the fit is an interpolation with $R^2=1$.)
But a high order polynomial fit is likely an unphysical result and doesn't describe the population well. So is there a mathematical characteristic or coefficient or model that describes and explains this?
I think what you might be looking for is the Akaike information criterion. It takes the number of parameters and the maximised value of the liklihood function to produce a number; the AIC says the model which produces the minimum value is preferred. Essentially it rewards well-fitting but penalises use of extra parameters.
http://en.m.wikipedia.org/wiki/Akaike_information_criterion