Background: I have three known points: A, B, and C—each with known spherical coordinates, $\rho$ and $\lambda$ on the surface of a sphere of unit radius that form a spherical triangle. I want to find the (spherical) coordinates of the points that are a given distance, d, away from one of the three original points, along the great circles of two arcs of the spherical triangle. See (roughly drawn) image:
Extrapolating arcs of a spherical triangle
Ultimately, I want to find the coordinates of points a and b, which are the same problem.
I know the values for the coordinates of the other points (A, B, and C) and the arc distance, d. I was able to calculate the angle $\theta$ as well, for what it is worth.
Following your depiction of the situation in the attached image, I will list the steps for points $C$ and $A$, and the same would follow for point $C$ and $B$.
Step 1: Write the Cartesian coordinates of $A$ and $C$:
$C = (\sin \theta_C \cos \phi_C, \sin \theta_C \sin \phi_C, \cos \theta_C )$
$A = (\sin \theta_A \cos \phi_A, \sin \theta_A \sin \phi_A, \cos \theta_A )$
Step 2: Find the cross product between $C$ and $A$ and normalize it
$N = \dfrac{ C \times A }{| C \times A |}$
Step 3: Find the perpendicular vector $D$ that lies $+90^\circ$ along the great circle containing $C$ and $A$, as follows,
$D = N \times C $
Step 4: Now we have two vectors $C$ and $D$ that span the great circle containing $C$ and $A$, therefore, point $a$ can be computed as follows
$a = (\cos d) C - (\sin d) D$
where a minus sign is used because $a$ is clockwise from $C$ whereas $A$ is anticlockwise.
Step 5: Now that we have the vector $a$ in Cartesian coordinates, we want to find $\theta_a$ and $\phi_a$, and they can be computed as follows:
$\theta_a = \cos^{-1} a_z $
$ \phi_a = ATAN2( a_x, a_y ) $