A colleague of mine told me about a “trick” for guarding against numerical instability in vector computations, and I wanted to check it here.
When I have something like this:
$\frac{xy^T}{x^Ty}$
or perhaps this:
$\frac{xy^T+yx^T}{x^Ty}$
and $x^Ty$ might be extremely small, then I probably want to check the numerical stability of this computation. So, the “trick” that I was taught was that you can always pre-compute the square root of the thing that you’re worried about being too small. E.g., in this case $x^Ty$. So, I would compute $\sqrt(x^Ty)$, in this case. Then, if I do the following substitutions, I can make this division just a multiplication:
$x’ = \frac{x}{\sqrt{x^Ty}}$
$y’ = \frac{y}{\sqrt{x^Ty}}$
Then the examples above would become
$x’y’^T$
and
$x’y’^T+x’y’^T$
respectively.
Does this protect against the numerical instability here? And if so, how? It is supposed to double the decimal place accuracy so you multiply by a larger, more accurate number instead of dividing by a small, possibly inaccurate number.