Smallest possible value of the norm?

Question

The vectors $ \vec{u_1} = \begin{bmatrix} 1 \\ 1 \\ 1\\ 1 \end{bmatrix} $ and $ \vec{u_2} = \begin{bmatrix} 1 \\ -1 \\ 1\\ -1 \end{bmatrix} $ are orthonormal in $ \mathbb{R}^4$.

In this exercise, I was supposed to find the parallel and normal components of the vector $ \vec{w} = \begin{bmatrix} 1 \\ 2 \\ -1\\ 2 \end{bmatrix} $ relative to the their span ($\vec{u_1}, \vec{u_2}$).

I found that the parallel component was (0, 2, 0, 2) and the normal component was (1, 0, -1, 0).

The next part of the exercise asks:

What is the smallest possible value of the norm $||\vec{w}- \vec{v}|| $ if the vector $\vec{v}$ belongs to their span: $ \vec{v} \in span(\vec{u_1}, \vec{u_2}$).

Could someone help me with this part?

Answer 1

Lets try the layman's approach:

$$\vec v=\begin{bmatrix}a+b\\a-b\\a+b\\a-b\end{bmatrix}\ \ (a,b\in \Bbb{R})$$ We need to find $$\min_{a,b}||\vec v-\vec w||$$Check that the values of $a,b$ for which $||\vec v-\vec w||$ will be minimum $=$ values of $a,b$ for which $||\vec v-\vec w||^2$ will be minimum. Now $$\min_{a,b}||\vec v-\vec w||^2\\=\min_{a,b}\{(a+b-1)^2+(a-b-2)^2+(a+b+1)^2+(a-b-2)^2\}\\=\min_{a,b}\{4a^2+4b^2+10-8a+8b\}\\=\min_{a,b}\{(2a-2)^2+(2b+2)^2+2\}=2 \ \ \ \text{for $a=1,b=-1$}\\\therefore\min{||\vec v-\vec w||=\sqrt 2}$$

BINGO!

Answer 2

Recall the following fact:

The projection of a vector $\vec{w}\in\mathbb{R}^n$ onto a subspace $M\subset\mathbb{R}^n$ is the vector in $M$ closest to $\vec{w}$.

In your case, the projection of $\vec{w}$ onto $\textrm{span}\{\vec{u}_1,\vec{u}_2\}$ is the parallel component (denoted $\vec{w}_{||}$). So, the minimum of $\|\vec{w}-\vec{v}\|$ where $\vec{v}$ belongs to $\textrm{span}\{\vec{u}_1,\vec{u}_2\}$ is $$\|\vec{w}-\vec{v}\|=\|\vec{w}-\vec{w}_{||}\|=\|\vec{w}_\perp\|, $$ where $\vec{w}_\perp$ is the orthogonal component you found. Thus, the answer is $\sqrt{2}$

Answer 3

If $\mathrm v \in \mbox{span} (\mathrm u_1, \mathrm u_2)$, then there is a $\mathrm x \in \mathbb R^2$ such that $\mathrm U \mathrm x = \mathrm v$, where the columns of $\mathrm U$ are $\mathrm u_1$ and $\mathrm u_2$. However, $\mathrm w \notin \mbox{span} (\mathrm u_1, \mathrm u_2)$. Thus, we have an instance of the famous least-squares problem

$$\begin{array}{ll} \text{minimize} & \| \mathrm U \mathrm x - \mathrm w \|_2^2 \end{array}$$

whose solution is

$$\hat{\mathrm x} := (\mathrm U^T \mathrm U)^{-1} \mathrm U^T \mathrm w$$

The error vector is

$$\mathrm w - \mathrm U \hat{\mathrm x} = \mathrm w - \mathrm U (\mathrm U^T \mathrm U)^{-1} \mathrm U^T \mathrm w = (\mathrm I_4 - \mathrm U (\mathrm U^T \mathrm U)^{-1} \mathrm U^T) \, \mathrm w$$

where $\mathrm P := \mathrm U (\mathrm U^T \mathrm U)^{-1} \mathrm U^T$ is the projection matrix that projects onto the column space of $\mathrm U$.

Using SymPy,

>>> U = Matrix([[1, 1], [1, -1], [1, 1], [1, -1]])
>>> U
⎡1  1 ⎤
⎢     ⎥
⎢1  -1⎥
⎢     ⎥
⎢1  1 ⎥
⎢     ⎥
⎣1  -1⎦
>>> w = Matrix([1, 2, -1, 2])
>>> w
⎡1 ⎤
⎢  ⎥
⎢2 ⎥
⎢  ⎥
⎢-1⎥
⎢  ⎥
⎣2 ⎦
>>> P = U * (U.T * U)**-1 * U.T
>>> P
⎡1/2   0   1/2   0 ⎤
⎢                  ⎥
⎢ 0   1/2   0   1/2⎥
⎢                  ⎥
⎢1/2   0   1/2   0 ⎥
⎢                  ⎥
⎣ 0   1/2   0   1/2⎦
>>> (eye(4) - P) * w
⎡1 ⎤
⎢  ⎥
⎢0 ⎥
⎢  ⎥
⎢-1⎥
⎢  ⎥
⎣0 ⎦
>>> P * w
⎡0⎤
⎢ ⎥
⎢2⎥
⎢ ⎥
⎢0⎥
⎢ ⎥
⎣2⎦

The $2$-norm of the error vector is

$$\| \mathrm w - \mathrm U \hat{\mathrm x} \|_2 = \| (\mathrm I_4 - \mathrm U (\mathrm U^T \mathrm U)^{-1} \mathrm U^T) \, \mathrm w \|_2 = \| (\mathrm I_4 - \mathrm P) \, \mathrm w \|_2 = \sqrt 2$$