I'm reading a paper on a particular form of PCA and I'm struggling to understand one part.
We have $||\mathbf{X}-\mathbf{XGH}^T||^2$ with $\mathbf{X}$ a $n\times p$ (data) matrix, $\mathbf{G}$ and $\mathbf{H}$ some $p\times k$ matrices, and $\mathbf{H}$ is orthonormal. They then proceed to say that since $\mathbf{H}$ is orthonormal, let $\mathbf{H}_\bot$ be any orthogonal matrix such that $[\mathbf{H};\mathbf{H}_\bot]$ is $p\times p$ orthogonal.
Then $||\mathbf{X}-\mathbf{XGH}^T||^2=||\mathbf{XH}_\bot||^2+||\mathbf{XH}-\mathbf{XG}||^2$
So not having a really strong linear algebra background, I really don't understand what are the step allowing us to do that, my initial intuition was that we use the fact that $\mathbf{H}^T\mathbf{H}=\mathbf{I}_{k\times k}$ but I can't find the same result as them and don't understand what is the goal of constructing this $p\times p$ orthonormal matrix. My second thought was that maybe it's related to some properties of the norm that I'm not aware of.
Any help is welcomed.
Thanks !
Because $[H \ \ H_\perp]$ is orthogonal (i.e. square with orthonormal columns), we have $\|M[H \ \ H_\perp]\| = \|M\|$ for any matrix $M$. Moreover, we must have $H^\top H_\perp = 0$. On the other hand, we have $$ (X - XGH^\top)\pmatrix{H & H_\perp} = \pmatrix{(X - XGH^\top)H & (X - XGH^\top)H_\perp} \\ = \pmatrix{XH- XGH^\top H & XH_{\perp} - GXH^\top H_\perp} \\ = \pmatrix{XH - XG & XH_\perp}. $$ Thus, we have $$ \|X - XGH^\top\|^2 = \|\pmatrix{XH - XG & XH_\perp}\|^2 = \|XH - XG\|^2 + \|XH_\perp\|^2. $$