Suppose there are two right triangles formed by points {U, V, W1} and {U, V, W2} on the surface of a sphere. The distances between these points form the sides of the triangles, a, b1, c1 and a, b2, c2, where U-V is a, V-W# is b#, and U-W# is c#. The angles C1 & C2 are opposite from sides c1 & c2, and both are 90 degrees.
We seek to determine a precise value for R, the radius of the sphere upon whose surface these triangles exist.
Based on the spherical law of cosines, we can make the following assumptions in the form of equations:
(1) cos(c1/R) = cos(a/R) * cos(b1/R)
(2) cos(c2/R) = cos(a/R) * cos(b2/R)
(Since cosine is a transcendental function, we cannot extract out R by itself from these equations.)
Now, if we know the values for c1, a, and b1, then due to the periodic nature of cosine, there are many possible valid values for R that will satisfy the equation. I.e. if c1 = 4140, a = 3069, and b = 2765, then R can be 2.05 and the equation will be true.
However, since we know that c1, a, and b1 are sections of the great circle whose radius is R, then we know that R can never be less than c1. Still though, if we restrict R to being greater than c1, we still get many possible correct values for R: 228,974,671.337 works; 1,553,282,877.94 also works... and many others.
However, since we must find an R that will work for both (1) and (2), is there any relationship (ratio) between c2 and c1, and between b2 and b1, that would allow us to eliminate all but one valid value for R? If so, what is it?
If not, if we add yet a third right triangle with a new point W3, would this new triangle allow us now to divine a single possible value for R?
(I am trying to devise a way to map things inside a video game that has no coordinate system or means of measuring angles other than right angles available to the player, but does give waypoint distances. Sadly my math isn't strong enough to figure this out.)
Note: I have reworded the question from its original form, which had been related to the determination of coordinates. I realized after further research that determination of the radius is a prerequisite to determining coordinates. Once you know the radius, then the coordinates become relatively simple to determine, so I rewrote the question in terms of radius.
I find this a very interesting and amusing problem. I believe that it should be possible to find the radius from the linear measure of the three sides of a single right spherical triangle; let me justify that belief.
Since I’m going to talk about only one right spherical triangle, I’ll call the legs $a$ and $b$, the hypotenuse $c$. My insight, such as it is, comes from considering the case of an isosceles right triangle, $a=b$. Then for very small triangles, $c$ is very nearly $\sqrt2a$, but definitely less. (If we had equality, the surface would be flat, i.e. Euclidean, with “infinite radius”.) On the other if $c=a$, we’d be dealing with an octant right triangle, all angles $\pi/2$, and $a=\frac\pi2R$, $R=2a/\pi$. Finally, if $c$ is very small in comparison to $a=b$, the legs would be nearly $\pi R$, $R\approx a/\pi$. Thus it seems “obvious” that as $c/a$ decreases from $\sqrt2$ to $0$, the radius will also decrease from infinite down to $a/\pi$. And this seems sure to be a monotone relationship, thus with a unique solution.
Now let’s look at your problem. Given your three measurements $a$, $b$, and $c$, you want to adjust $R$ so that the angles $\alpha=a/R$, $\beta=b/R$, and $\gamma=c/R$ satisfy the spherical Pythagorean Theorem $\cos\gamma=\cos\alpha\cos\beta$. To do a numerical problem it seemed to me most sensible to introduce a variable $t=1/R$, and find a root of the basic function $$ f(t)=\cos(at)\cos(bt)-\cos(ct)\,. $$ I decided to use $a=0.6$, $b=0.8$, and for $c$ I chose something smaller than $1.0$, namely $c=0.95$, and I thought I’d do a Newton-Raphson approximation. I’m sure you can get this sort of thing automated, but I did it almost by hand, with my HP15-c. Of course I used $f'(t)=-a\sin(at)\cos(bt)-b\sin(bt)\cos(at)+c\sin(ct)$. I started with the guess $t=1$, and the fifth iteration gave me nine decimal digits of accuracy, with $t=1.085162010$, and thus $R=.921521387\,$.
I would be greatly surprised if one could get a solution in closed form by messing with some combination of inverse trigonometric functions, so the method of successive approximation is probably as good as you can get. Once it was programmed, it wouldn’t be at all demanding on the computation abilities of any computer.