For $m<n$ let $\vec{u_1}...\vec{u_m} \in R^n$ be an orth basis for span{$\vec{u_i}$}$^m_{i=1}$. $\vec{b} \in R^n$ closest vector to $\vec{b}$ in $U$?

Question

For $m<n$ let $\vec{u_1}...\vec{u_m} \in R^n$ be an orthonormal basis for span{$\vec{u_i}$}$^m_{i=1}$. Let $\vec{b} \in R^n$ what is closest vector to $\vec{b}$ in $U$?

I have seen this question and I'm not entirely sure what they mean't by "what is the closest vector to b in U" and they wanted the solution in a linear combination of $\vec{u_i}$'s

Answer 1

The closest vector to $b$ in $U$ is given by

$b_U = \displaystyle \sum_{i=1}^m c_i u_i $

where $c_i = \dfrac{b \cdot u_i}{u_i \cdot u_i} $

To derive this formula, consider the error function:

$E = \displaystyle (b - \sum_{k=1}^m c_k u_k) \cdot (b - \sum_{k=1}^m c_k u_k)$

We want to minimize $E$ over all possible $\{ c_i \}$

Differentiating E with respect to $c_i$, we get

$\dfrac{\partial E}{\partial c_i} =\displaystyle 2 (b - \sum_{k=1}^m c_k u_k) \cdot u_i = 2 (b \cdot u_i - c_i u_i \cdot u_i )$

Setting this to zero, results in the formula for $c_i$ stated above.

Answer 2

They want you to find the vector $u \in U := \text{span}(u_1, \dots, u_m)$ that minimizes $\lVert u - b \rVert$. The answer is $P_Ub$, the orthogonal projection of $b$ onto $U$. The proof is simple. Let $w = b - P_Ub \in U^{\perp}$. We have \begin{align} \lVert u - b \rVert^2 &= \lVert u - P_Ub - w \rVert^2 \\ &= \lVert u - P_Ub \rVert^2 + \lVert w \rVert^2 \\ &\geq \lVert u - P_Ub \rVert^2, \end{align} where the second equality is by the Pythagorean theorem.

$P_Ub$ can be written explicitly as $P_Ub = \sum_{i = 1}^{m}(b, u_i)u_i$, where $(\cdot, \cdot)$ denotes your inner product on $\mathbb{R}^n$. This formula is often used to prove the existence of the direct sum decomposition $\mathbb{R}^n = U \oplus U^{\perp}$, which then gives the projection $P_U$ associated to this direct sum decomposition.