I am interested in solving a celestial navigation for myself. This reference https://aa.usno.navy.mil/downloads/reports/Kaplan1996b.pdf by George Kaplan describes how the US Navy’s STELLA software uses a best-fit approach to not only determine latitude and longitude, but also ship’s course and speed, from several celestial observations made over an interval of time. I do not understand where equation (1) in that reference came from, so I’m trying to determine the solution to a simpler, related problem: how to find the great-circle that best fits three points on a spherical Earth that are not necessarily ‘collinear’ in the spherical trigonometry sense.
Specifically, given points with lat and lon ($\phi_1$, $\lambda_1$), ($\phi_2$, $\lambda_2$), and ($\phi_3$, $\lambda_3$), I want to find an equation relating $\phi$ and $\lambda$ that describes a great circle. If $d_i$ is the (perpendicular) shortest distance from point ($\phi_i$, $\lambda_i$) and the great circle, the great circle should be the one that minimizes the overall root-sum-squared-distances $d_{rss} = \sqrt{d_1^2 + d_2^2 + d_3^2}$.
The squaring of the distances is not crucial to the spirit of the problem, but like a least-squares best fit line on a plane, I imagine that it guarantees all the terms are positive and is easier to work with than absolute values, as in $d = |d_1| + |d_2| + |d_3|$. I recognize that the least-square process on a plan actually minimizes the sum of the vertical squared distances for convenience, but I don’t see that there’s an obvious analogous quantity on the surface of a sphere, so I choose to minimize the shortest squared distances between points and the great circle, which are great circle arcs perpendicular to the best fit great circle.
The only similar question I found was this, but the only answer seems to be more of a suggestion, and I am wholly unfamiliar with tensor-related math: Line of best fit on the surface of a sphere
Thank you for any insight you might be willing to share.