I'm not quite sure this is the appropriate place to post the question, but hopefully someone who has a deeper understanding about the subject than me sees this.
We are given $n + 1$ data points $(x_0, y_0),(x_1, y_1), . . . ,(x_n, y_n)$ and we want to find a smooth function $y = f(x)$ to represent the trend in these points. We define a smooth function to be one that is continuous and has continuous first and second derivatives. One approach is to use penalized regression, i.e. minimise
$$\sum_{j=0}^{n}(y_j-f(x_j))^2+\lambda\int (f''(x))^2dx$$
where $\lambda >0$.
There is a theorem that says that the solution to this is a natural cubic spline that interpolates the data.
My question is: What is the role of $\lambda$? My lecture notes say that it is a a parameter that makes connection between the fit and the penalty. However, since the natural spline is uniquely determined by the points, won't $\lambda$ be useless in the expression?
Thanks!
If the theorem says that the natural cubic spline is the solution, then it means that there are other cubic splines which can be obtained by other criteria (e.g. $\lambda$). In fact, cubic splines are underdetermined by a set of points because the first and last endpoints do not place enough constraints on the system of equations. This is resolved in the case of a natural cubic spline by constraining the second derivatives at the first and last point to be 0.