Converting between SVD and Eigenvector-based expressions

Question

For the well known ordinary least squares there are the well known solution

$$\beta=(X'X)^{-1}X'Y$$

This can be expressed in "canonical form" using eigenvector decomposition.

$$[W,\Delta^2]=\operatorname{eig}(X'X)$$

Transforming:
$$Z=XW$$ $$\alpha=W'\beta$$

so

$$\alpha=(W'X'XW)Z'Y$$ $$=\Delta^{-2}Z'Y$$ $$=\Delta^{-2}W'X'Y$$

Similarly using SVD

$$[Q,\Delta,P']=\operatorname{SVD}(X)$$

where $P'=W$ from the eigenvector decomposition
and $\Delta$ is the square root of $\Delta^{2}$

$$X=Q\Delta P'$$

$$X'X=P\Delta Q'Q\Delta P'$$ $$=P\Delta^2P'$$

$$X'Y=P\Delta Q'Y$$

$$\beta=P\Delta^{-2}P'P\Delta Q'Y$$ $$\beta=P\Delta^{-1}Q'Y$$

pre-multiply by $P'$

$$\alpha=\Delta^{-1}Q'Y$$

so here $\alpha=P'\beta=W\alpha$ rather than $W' \alpha$ as above

Q1.) Are the derivations above correct?

To covert back in the first case simply multiply both sides by W

$$W W'\beta=W(W'X'XW)W'X'Y$$ $$\beta=(X'X)^{-1}X'Y$$

which is clearly the OLS expression.

for the second case multiply by $P$

$$P P' \beta=P\Delta^{-1}Q'Y$$ $$\beta=P\Delta^{-1}Q'Y$$

which is not the OLS expression but $P\Delta^{-1}Q'=X^{-1}$

Given that the each expression is based on the same original OLS expression and use related transformations I would expect that these expressions to be equivalent. However the expressions derived seem quite different?

Q2.) Are the SVD and Eigen based transformations equivalent?

Q3.) If so how can I transform between the two?

Answer 1

Hint : The eigenvalue decomposition of $X'X$ and the singular value decomposition of $X'X$ are the same.

Answer 2

Start with the linear system $$ \mathbf{A} x = b. $$ The target matrix $\mathbf{A}\in\mathbb{C}^{m\times n}_{\rho}$, the data vector $b \in\mathbb{C}^{n}$ and not in the null space. Use the method of least squares to find the solution vector $x\in\mathbb{C}^{n}$.

The least squares solution is defined as the set $$ x_{LS} = \left\{ x \in \mathbb{C}^{n} \colon \lVert \mathbf{A}x - b \rVert_{2}^{2} \text{ is minimized} \right\} $$

Because you used the normal equations, we will constrain $\matrix{A}$ to having full column rank. The normal equations then $$ \mathbf{A}^{*} \mathbf{A} x = \mathbf{A}^{*} b $$ can be solved by $$ x_{LS} = \left( \mathbf{A}^{*} \mathbf{A} \right)^{-1} \mathbf{A}^{*} b $$

To compute the singular value decomposition $$ \mathbf{A} = \mathbf{U} \, \Sigma \, \mathbf{V}^{*} $$ start by forming the product matrix $\mathbf{A}^{*}\mathbf{A}$. Compute the eigenvalue spectrum $$ \lambda \left( \mathbf{A}^{*} \mathbf{A}\right). $$ Order the eigenvalues, and remove the terms which are $0$ or close enough to $0$, admittedly a subjective task. Call this spectrum $\tilde{\lambda}$. The singular values are then $$ \sigma = \sqrt{\tilde{\lambda}}. $$ The normalized eigenvectors of the product matrix are the column vectors of the domain matrix $\mathbf{V}$.

You question brings up an interesting issue: matrix diagonalization. If a matrix can be diagonalized, then we would compute the eigenvectors of $\mathbf{A}$.

But if $\mathbf{A}$ can't be diagonalized, the SVD will get as close to diagonalization as possible. In this case we must compute the eigensystem of the product matrix $\mathbf{A}^{*}\mathbf{A}$.