L2 Norm Decomposition

Question

I've been struggling to understand how the L2 norm decomposition of the error in PCA: $\|\vec x_i - \langle\vec x_i, \vec w\rangle \vec w\|$ results in $\|\vec x_i\|^2 - \langle\vec w, \vec x_i\rangle^2$, note that $\vec w^{\mathsf{T}} \vec w = 1$

Is my approach of foiling out the norm as follows valid:

$(\vec x_i - \langle\vec x_i, \vec w\rangle \vec w) ^{\mathsf{T}} (\vec x_i - \langle\vec x_i, \vec w\rangle \vec w) $

$ \|\vec x_i\|^2 - 2\vec x_i^{\mathsf{T}}(\langle\vec x_i, \vec w\rangle \vec w) + \langle\vec x_i, \vec w\rangle^2 \|\vec w\|^2$

$ \|\vec x_i\|^2 - 2\vec x_i^{\mathsf{T}}(\langle\vec x_i, \vec w\rangle \vec w) + \langle\vec x_i, \vec w\rangle^2 $

This is where I get stuck and don't know how to proceed. Any pointers would be greatly appreciated. Thanks!

Answer 1

Note $<x_i, w>$ is a scalar, which you can move to the front in the middle term. Then $$<x_i, w>x_i^Tw=<x_i,w><x_i,w>=<x_i,w>^2$$ So you multiply this by $-2$ and add it to the last term. If you notice that $<x_i,w>=<w,x_i>$, you get the desired expression.

Answer 2

Note that the inner product is a scalar and $\langle \vec{x},\vec{y} \rangle = \vec{x}^{\mathsf{T}} \vec{y}$, so $$ - 2\vec x_i^{\mathsf{T}}(\langle\vec x_i, \vec w\rangle \vec w) =- 2\langle\vec x_i, \vec w\rangle \vec x_i^{\mathsf{T}} \vec w =- 2\langle\vec x_i, \vec w\rangle^2 $$ Bear in mind, though, $$ \|\vec{x}\|^2 = \langle x,x \rangle = \vec{x}^{\mathsf{T}} \vec{x} $$ so, unless you're trying to decompose the square of the norm in your context, to get $$ \|\vec x_i - \langle\vec x_i, \vec w\rangle \vec w\|^2 = \|\vec x_i\|^2 - \langle\vec w, \vec x_i\rangle^2 $$ this might not be enough.