I was trying to approximate a sine curve with a finite number of polynomials terms using linear regression (or the pseudo-inverse). I construct a Vandermonde matrix (or a Kernel polynomial feature matrix) and then solve the linear system (usually using the pseudo-inverse):
$$ y = \Phi(x)w$$
then I try to visualize the solution. For low degree polynomials the approximation seems fine but eventually when the degree of the polynomial is pretty high, there is a weird funky bit at the edge:
This reminded me of Gibbs phenomenon where at discontinuities there seems to be high oscillations near the jump. I know that Gibbs phenomena happens with a finite sum of Fourier series. However, this empirical observation really made me wonder. Is this the reason the edge of the approximation looks strange?
Also from a statistics/machine learning point of view it seems clear that if the solution is not regularized, then a high complexity model should overfit to noise. However, in this model there is no noise. Therefore, I was not quite sure what was going on and was wondering what it was.
Thus, my question is, is there a Gibbs phenomena for approximating sinusoidal with a finite number of polynomial terms? If yes, then what is it and what are its details?
Another observation that I found odd is that the high oscillation/jump only happened at the right discontinuity. I am not sure why that is but I thought it was quite puzzling.
In general, it's a bad idea to use interpolation polynomials for equally spaced points as approximations. See Runge's phenomenon.
On the other hand, there are much better ways to get good polynomial approximations to continuous functions on an interval. See Chebyshev series and Remez algorithm.