Given 2 points A,B,O on sphere of radius $R$. Point O is in middle of AB. E and F are deviation from O by geodesic distance $d$ (angle between EF and AB is $90^o$). Consider 2 circles $C_1,C_2$ on sphere which go through points A,B,E and $A,B,F$, respectively. Given a point X (x,y,z) on sphere, check $X$ belongs to intersection of $C_1$ or $C_2$.
Input: R: radius of sphere, coordinate of A,B and deviation $d$ on sphere of radius $R$, a point X on sphere.
Output: X lies in area defined by $AEBF$ or not? Every points lie in sphere and the distance between 2 points is given by the great circle distance.
My idea is to first find coordinate of 2 points E and F, then find center and radius of 2 circles which go through ABE and ABF. With this information, checking X in AEBF is very easy. I am able to find the great circle distance between 2 points on sphere. But I have no idea on locating the coordinate of E and F and the circle on sphere with given 3 points.
Does anyone have a better algorithm (a trick)? An approximate solution is also appreciated. Thank you in advance !
Let $\phi=(d/R)(180°/\pi)$ be the angular size of arc $OE$ and $\gamma=(OA/R)(180°/\pi)$ the angular size of arc $OA$ (see picture below). If $S$ is the center of the sphere and $C$ is the center of circle $AEB$, then $SC$ is perpendicular plane $AEB$.
Let now $H$ be the point where chord $AB$ intersects radius $SO$. By applying the sine rule to triangle $EHS$ one has $HS/\sin\alpha=SE/\sin(\angle SHE)$. But $SE=R$, $HS=R\cos\gamma$ and $\angle SHE=\pi-\alpha-\phi$, so the preceding relation becomes $\cos\gamma\sin(\alpha+\phi)=\sin\alpha$, which can be solved to yield $$\alpha=\arctan{\cos\gamma\sin\phi\over1-\cos\gamma\cos\phi}.$$
The angular size of arc $OC$ can thus be found as: $\theta=90°-\phi-\alpha$. Point $C$ lies on arc $EOF$, opposite to $E$ if $\theta>0$ and towards $E$ if $\theta<0$.