I was trying to solve the standard inoffensive standard linear system:
$$ Xw = y $$
but I wanted to only use the pseudo-inverse. Usually when I solve equations I just apply whatever operation I did to both sides of the equation and on the left or right depending where I applied the matrix operation, e.g.
$$ X^{\dagger} X w = X^{\dagger}y$$
hopefully to get:
$$ w = X^{\dagger}y$$
people that know about left and right pseudo-inverses should expect problems. The above obviously works if $X$ has a left inverse (e.g. if $X$ is full column rank). However the above fails to "cancel out" or solve the equation if $X$ has full row rank. When it has full row rank it only has a right inverse and if we do the above we get:
$$ X^{\dagger} X w = X^{\dagger}y$$
$$ X^T (X X^T)^{-1} X w = X^T (X X^T)^{-1} X y$$
which doesn't cancel out further (as far as I know at least). Thats because we can only "cancel" $X$ with a right multiplication i.e. as in $X X^{\dagger}$. Ok so the only way to do this in my mind is to image the system as follows:
$$ Xw = y I$$
where $I$ is an identity and proceed to modify both sides of the equation with the same right multiplication:
$$ X X^{\dagger}w = y X^{\dagger} I$$
$$ w = y X^{\dagger}$$
$$ w = y X^T (X X^T)^{-1}$$
however if one plugs that back in one notices that it obviously does not actually solve the equation because the right inverse is obstructed by $y$ so it can't cancel things out properly:
$$ Xw $$ $$ X(y X^T (X X^T)^{-1}) \neq y$$
however, this makes me suspect that maybe there is a deeper issue about solving and manipulating equations with matrices that I don't understand. Does this mean we can't actually do operations on both sides and have the system remain intact? What is the actual rules for applying operations to both sides to equations when matrices are involved? Or is there a set of rules that always work for all equations?
Essentially I can't tell anymore when that method will yield wrong or correct solutions which has me a bit worried...
I wanted to address this since it seems there is something important that I need to learn (or actually, unlearn).
Actually to make things worse I just realize that plugging back in the solution when there is a left inverse does yields problems:
$$ X^{\dagger}y = c$$
plug to $Xw$ to get if it yields y:
$$ X(X^{\dagger}y) $$
which does not yield $y$ because $X^{\dagger}$ is a left inverse so only $X^{\dagger} X$ yields cancellation. How can I resolve this issue?!