My question is, why is not $XY^{-1} = A$? I think at some other point of time I also see people using transpose instead of inverse in 'dividing' a matrix from another while solving matrix equations. But I searched a lot and don't seem to find why such thing can be done so? Or am I missing something?
Thx.
In order for a matrix to be invertible it has to at the very least be a square matrix. If you write the equation $x_i = Ay_i$ for $i = 1,\ldots, N$, this doesn't guarantee that the matrices $X$ and $Y$ formed from this are square since there's no assumption made on the length of each of the vectors $x_i$ or $y_i$. In general we may find that $X = AY$ is not exactly solvable since we may have $X$ is a $d\times N$ matrix, $Y$ is a $l\times N$ matrix thus forcing $A$ to be $d\times l$ where $d$ is not necessarily the same size as $l$.
The standard approach to solving this instead, is to multiply on the right by $Y^T$ obtaining the equation
$$ XY^T \;\; =\;\; AYY^T. $$
What we see here is that $YY^T$ is now an $l\times l$ matrix. There is still no guarantee that $YY^T$ will be invertible, but many times in applications it is. This is, in fact, the solution to the least squares fitting problem. In the case that $YY^T$ is now invertible we can multiply on the right by its inverse, thus obtaining the equation you were inquiring about:
$$ A \;\; =\;\; XY^T\left (YY^T\right )^{-1}. $$