Simple proof that $\min \|\mathbf{f} - \sum_i \lambda_i\hat{\mathbf{g}_i}\|$ is given by $\lambda_i = \langle \mathbf{f}, \hat{\mathbf{g}}_i \rangle$

Question

Let $\mathbf{f}$ be a real valued vector and $\{\hat{\mathbf{g}}_i\}$ be a set of orthonormal vectors of the same dimension. What is a simple proof that

$$\min \|\mathbf{f} - \sum_i \lambda_i\hat{\mathbf{g}_i}\|$$

is given by $\lambda_i = \langle \mathbf{f}, \hat{\mathbf{g}}_i \rangle$

This is my attempt:

$$\min \|\mathbf{f} - \sum_i \lambda_i\hat{\mathbf{g}}_i\|$$ $$ = \min \|\mathbf{f}\|^2 + \|\sum_i \lambda_i\hat{\mathbf{g}}_i\|^2 - 2\langle\mathbf{f},\sum_i \lambda_i\hat{\mathbf{g}}_i\rangle$$

Since

$$\|\sum_i \lambda_i\hat{\mathbf{g}}_i\|^2 = \langle\sum_i\lambda_i\hat{\mathbf{g}}_i,\sum_i \lambda_i\hat{\mathbf{g}}_i\rangle$$ $$ = \sum_i\sum_j\langle\lambda_i\hat{\mathbf{g}}_i, \lambda_j\hat{\mathbf{g}}_j\rangle$$ $$ = \sum_i\langle\lambda_i\hat{\mathbf{g}}_i, \lambda_i\hat{\mathbf{g}}_i\rangle$$ $$ = \sum_i\lambda_i^2$$

Thus $$\min \|\mathbf{f} - \sum_i \lambda_i\hat{\mathbf{g}_i}\| = \min \sum_i \lambda_i^2 - 2\langle\mathbf{f},\sum_i \lambda_i\hat{\mathbf{g}}_i\rangle$$ $$ = \min \sum_i \lambda_i^2 - 2 \sum_i \lambda_i\langle\mathbf{f},\hat{\mathbf{g}}_i\rangle$$

Since at the minimum we have

$$\frac{d}{d\boldsymbol{\lambda}} \sum_i \lambda_i^2 - 2 \sum_i \lambda_i\langle\mathbf{f},\hat{\mathbf{g}}_i\rangle = 0$$

Taking partial derivative w.r.t. $\lambda_i$ we get

$$2 \lambda_i - 2 \langle\mathbf{f},\hat{\mathbf{g}}_i\rangle = 0$$ $$\lambda_i = \langle\mathbf{f},\hat{\mathbf{g}}_i\rangle$$

but is there something simpler?

Answer 1

Hint: An arbitrary $\mathbf{f}$ can be expressed uniquely as $\mathbf{f} = \sum_i \mu_i \mathbf{\hat{g}}_i + \mathbf{f}'$ where $\mathbf{f}'$ is orthogonal to all the $\mathbf{\hat{g}}_i$.

---

Second hint: $$\lVert \mathbf{f} - \sum_i \lambda_i \hat{\mathbf{g}}_i \rVert^2 = \lVert \mathbf{f}'\rVert^2 + \sum_i \lVert (\mu_i - \lambda_i)\hat{\mathbf{g}}_i\rVert^2$$

---

Summary of a solution: Using the above two hints, we find that

$$\lVert \mathbf{f} - \sum_i \lambda_i \hat{\mathbf{g}}_i \rVert^2 = \lVert \mathbf{f}'\rVert^2 + \sum_i |\mu_i - \lambda_i|^2$$

This is a sum of a constant and a series of positive numbers, so it is minimized if we can make those positive numbers zero. We can: $\lambda_i = \mu_i$. But looking back at the first hint, $\mu_i = \langle \hat{\mathbf{g}}_i, \mathbf{f} \rangle$ so we're done.

Answer 2

As a function of $\lambda_i$, the norm is convex in each $\lambda_i$. Picking $\lambda_i=\langle f,g_i\rangle$ gives a value of 0 for the norm. Since norms are non-negative, you're done.