I have a point $(x, y)$ inside a rectangle of size $(W, H)$. How to select a direction for that point, what would be equations whose solutions would give something like this:
A direction to the nearest clockwise point on inscribed circle (along one of the axes) if the point is inside and on a line that touches circle if point is outside?
Let the circle have radius $R$ and be centered at the origin. Given $(x,y)$ let $\phi={\rm arg}(x,y)$ be its polar angle, determined up to $2\pi$. Then $${\rm arg}\,{\bf v}(x,y)=\left\{\eqalign{\left\lfloor{2\phi\over\pi}+{1\over2}\right\rfloor{\pi\over2}\qquad&\qquad\bigl(\sqrt{x^2+y^2}<R)\cr \phi+\pi\pm\arcsin{R\over\sqrt{x^2+y^2}}&\qquad\bigl(\sqrt{x^2+y^2}>R)\ .\cr}\right.$$ A decision about the $\pm$ as well as some exception handling will be necessary.