My question is rather simple. Can I use the dot product to determine if a coordinate lies between two others?
With coordinates I mean a Point P(latitude, longitude) on the surface of the sphere. I know that the dot product can be used in the plane for this problem. Do I have to convert the coordinates to cartesian ones to use the same technique on a sphere? Or can I use it straight away?
Edit:
The three points don't have to be collinear. Let's say the points A and B are the start and endpoints of a geodesic. The third point C could lie on that geodesic or it is an unknown distance away from it. I need to check if the latitude and longitude of point C is between the ones of A and B. So, I need to check if C is in between the two black lines.
May be it is more clear if you have a look at the picture.
For this answer I will not rely on Cartesian coordinates, but I will not work directly in terms of latitude and longitude either. Instead, it will be convenient to represent a point with spherical coordinates $(\phi,\theta)$ as $z=e^{i\phi}\tan\dfrac\theta2$, corresponding to a stereographic projection of the sphere into the complex plane. We can now apply Mobius transformations to move the complex numbers $\alpha$, $\beta$ (corresponding to $A$, $B$) to more helpful locations.
Specifically, we can use rotations to move $\alpha$ to zero, then map $\beta$ to the real line, and finally rotate $\{\alpha,\beta\}$ to $\{r,r^{-1}\}$ on the positive real line. This amounts to rotating our coordinates so that $A$ and $B$ have zero longitude and opposite latitudes. To make this simpler, we shall assume that $\alpha$ is real without loss of generality (i.e. we take the longitude of $B$ relative to $A$).
We now state these transforms explicitly. First, note that the subgroup of the Mobius group corresponding to rotations consists of mappings of the form $z\mapsto \dfrac{u z-\overline{v}}{v z+\overline{u}}$ for complex $u,v$. A mapping of this form which takes $\alpha$ to the origin is $z\mapsto \dfrac{z-\alpha}{\alpha z+1}$; this also maps $\beta\mapsto \dfrac{\beta-\alpha}{1+\beta\alpha}= R e^{i\psi}$ for some real $R,\psi$. A rotation $z\mapsto e^{-i\phi} z$ then brings $\beta$ to $R$.
For the last transformation, consider the mapping $z\mapsto \dfrac{1+z r}{r-z}$ for real $r>0$ which takes the real line to itself. Then $\{0,R\}\mapsto \left\{\dfrac{1}{r},\dfrac{1+r R}{r-R}\right\}$; since we want the images of $0$ and $R$ to be on equal and opposite sides of the 'equator', we require their product to be unity. From this we deduce $r=R+\sqrt{1+R^2}$. Composing the three transformation gives the net rotation
$$\mathcal{T}:\;z \mapsto \dfrac{z-\alpha}{1+\beta\alpha} \mapsto e^{i\phi}\left(\frac{z-\alpha}{1+\beta\alpha}\right) \mapsto \dfrac{1+e^{i\phi}\left(\frac{z-\alpha}{1+\beta\alpha}\right)r}{r-e^{i\phi}\left(\frac{z-\alpha}{1+\beta\alpha}\right)} =\dfrac{e^{-i\phi}(1+\beta\alpha)+(z-\alpha)r}{r e^{-i\phi}(1+\beta\alpha)-(z-\alpha)}$$
We now can give a criterion for the original question: a point $z$ lies between $\alpha$ and $\beta$ if $r^{-1}\leq | \mathcal{T}(z)|r$. In principle one can do inverse transformations to map this annulus back to the original coordinates, but I won't do so explicitly. Instead I'll just summarize the result: