I Have a Least Squares problem. I have a very large data set $Z$, define as follows:
$Z = \begin{bmatrix} z_0 \\ z_1 \\ \vdots \\ z_{N} \end{bmatrix}$
First, I want to fit it into the following polynomial: $z_k = a + cx_{k}^2$. after that, the residual error can be calculated: $J_{1} = \tilde{Z}^{T}\tilde{Z}$
then, i'm trying to fit it into $z_k = a + bx_{k} + cx_{k}^2$, this will result in diffrent residual - $J_{2}$.
MY question: is $J_{2} \leq J_{1}$. It's seemed like that since b can always be taken as $0$. is there any proof for that? proof for general case when increasing the model will result in a smaller residual?
Thank you all.
If $A \subset B$, then $\min_{x \in B} f(x) \le \min_{x \in A} f(x) $. This is because if $\min_{x \in A} f(x) = a$, then $\exists x \in A : f(x) = a$. Since $A \subset B, x \in A \rightarrow x \in B$. Thus, $\exists x \in B :f(x) =a$. The minimum is less than or equal to all other values, by the definition of "minimum", so $\min_{x \in B} f(x) \le a$.
One could still ask whether the LS gives the minimum (proof by nomenclature is a bit of a fallacy). There is also the issue of whether "least" here is a minimum. In some contexts, it's actually the infimum. But there is a similar, albeit more complicated, proof that this holds for infima in addition to minima, and LS does have indeed have a minimum anyway.