In my notes there is the following theorem:
Let $X_k : [a,b] \rightarrow \mathbb{R}$, $k=1, \dots , n$ an orthogonal system of functions and $X: [a,b] \rightarrow \mathbb{R}$, then $\forall c_1, \dots , c_n \in \mathbb{R}$ we have the inequality: $$\int_a^b |X(x)-\sum_{i=1}^n c_i X_i(x)|^2 dx \geq \int_a^b | X(x)-\sum_{i=1}^n A_i X_i(x)|^2 dx \\ \text{ where } A=\frac{\int_a^b X(x) X_i(x)dx}{\int_a^b X_i^2(x)dx}, i=1, 2, \dots , n$$
The proof is the following:
$$\int_a^b \left |X(x)-\sum c_i X_i(x)\right |^2dx \\ =\int_a^b \left [X^2(x)+\sum_{i=1}^n c_i^2 X_i^2-2\sum_{i=1}^n c_iX(x)X_i(x)+2\sum_{i<j}c_ic_jX_iX_j \right ] dx \\ =\int_a^b X^2(x)dx+\sum_{i=1}^n c_i^2 \int_a^b X_i^2-2 \sum_{i=1}^n c_i \int_a^b XX_i+2\sum_{i<j}c_ic_j \int_a^b X_iX_jdx \overset{ \sum_{i<j}c_ic_j \int_a^b X_iX_jdx=0 }{ \Longrightarrow } \int_a^b X^2(x)dx+\sum_{i=1}^m \left [c_i^2 \int_a^b x^2_idx-2c_i\int_a^b XX_idx\right ]=(*)$$
We want to minimize the expression $$c_i^2 \int_a^b x^2_idx-2c_i\int_a^b XX_idx$$ in resprect to $c_i$.
$$At^2-2Bt=A\left (t-\frac{B}{A}\right )^2-\frac{B^2}{A} \geq -\frac{B^2}{A}$$
$$(*)=\int_a^b X^2(x)dx+\sum_{i=1}^n \left [\int_a^b X^2_i(x)dx \left (c_1-\frac{\int_a^b XX_idx}{\int_a^bX^2_idx}\right )^2-\frac{(\int_a^bXX_idx)^2}{\int_a^b X^2_i(x)dx}\right ]=\int_a^b X^2(x)dx-\sum_{i=1}^n \frac{(\int_a^b XX_idx)^2}{\int_a^b X^2_i(x)dx}+\sum_{i=1}^n \int_a^b X^2_i(x)dx \left (c_i-\frac{\int_a^b XX_idx}{\int_a^b X_i^2dx}\right )^2\geq \int_a^b X^2(x)dx-\sum_{i=1}^n \frac{\int_a^b XX_idx)^2}{\int_a^b X^2_i(x)dx}=\int_a^b X^2(x)dx-\sum_{i=1}^n X^2_i(x)dxA_i^2=\int_a^b \left |X(x)-\sum_{i=1}^n A_i X_i\right |^2 dx, \text{ where } A_i=\frac{\int_a^b XX_idx}{\int_a^b X^2_idx} \\ \Rightarrow \int_a^b |X(x)-\sum A_i X_i|^2=\int_a^b X^2(x)dx-\sum_{i=1}^n \int_a^b X^2_i(x)dxA^2_i \geq 0 \\ \Rightarrow \sum_{i=1}^n \int_a^b X^2_i(x)dxA^2_i \leq \int_a^b X^2(x)dx \\ \Rightarrow \sum_{i=1}^{\infty} \int_a^b X^2_i(x)dxA^2_i \leq \int_a^b X^2(x)dx$$
$$$$
I haven't understood the following points:
- $$\int_a^b \left |X(x)-\sum c_i X_i(x)\right |^2dx =\int_a^b \left [X^2(x)+\sum_{i=1}^n c_i^2 X_i^2-2\sum_{i=1}^n c_iX(x)X_i(x)+2\sum_{i<j}c_ic_jX_iX_j \right ] dx$$
- $$\int_a^b X^2(x)dx-\sum_{i=1}^n X^2_i(x)dxA_i^2=\int_a^b \left |X(x)-\sum_{i=1}^n A_i X_i\right |^2 dx$$
$\int_a^b \left |X(x)-\sum c_i X_i(x)\right |^2dx =\int_a^b \left [X^2(x)+\sum_{i=1}^n c_i^2 X_i^2-2\sum_{i=1}^n c_iX(x)X_i(x)+2\sum_{i<j}c_ic_jX_iX_j \right ] dx $
because
$\begin{array}\\ |X(x)-\sum c_i X_i(x) |^2 &=(X(x)-\sum c_i X_i(x))^2\\ &=X^2(x)-2X(x)\sum c_i X_i(x)+(\sum c_i X_i(x))^2\\ &=X^2(x)-2\sum c_i X(x)X_i(x)+\sum_i c_i X_i(x)\sum_j c_j X_j(x)\\ &=X^2(x)-2\sum c_i X(x)X_i(x)+\sum_i c_i X_i(x)\sum_{j=i} c_j X_j(x)+\sum_i c_i X_i(x)\sum_{j\ne i} c_j X_j(x)\\ &=X^2(x)-2\sum c_i X(x)X_i(x)+\sum_i c_i^2 X_i^2(x)+\sum_i c_i X_i(x)\sum_{j\ne i} c_j X_j(x)\\ \end{array} $
and
$\begin{array}\\ \sum_i c_i X_i(x)\sum_{j\ne i} c_j X_j(x) &=\sum_i c_i X_i(x)\left(\sum_{j< i} c_j X_j(x)+\sum_{j> i} c_j X_j(x)\right)\\ &=\sum_i c_i X_i(x)\sum_{j< i} c_j X_j(x)+\sum_i c_i X_i(x)\sum_{j> i} c_j X_j(x)\\ &=\sum_i c_i X_i(x)\sum_{j< i} c_j X_j(x)+\sum_{j} c_j X_j(x)\sum_{i<j} c_i X_i(x) \quad\text{(reverse order of summation)}\\ &=\sum_i c_i X_i(x)\sum_{j< i} c_j X_j(x)+\sum_{i} c_i X_i(x)\sum_{j<i} c_j X_j(x) \quad\text{(exchange i and j)}\\ &=2\sum_i c_i X_i(x)\sum_{j< i} c_j X_j(x)\\ \end{array} $