I am looking for a closed-form solution to find the mutual slant of two cones with a common vertex shown in the image. Any insight on the problem-solving approach would be useful.
If $\hat{k}_1$ and $\hat{k}_3$, are known unit vectors and the angle between an unknown unit vector $\hat{k}_2$ and , $\hat{k}_1$ and $\hat{k}_3$, are given as $\alpha_{12}$ and $\alpha_{23}$ such that $$\hat{k}_1.\hat{k}_2=\cos\alpha_{12}$$ and $$\hat{k}_2.\hat{k}_3=\cos\alpha_{23}$$, the angle between $\hat{k}_1$ and $\hat{k}_3$ is $\beta$. I am looking for the vectors where the cones intersect.
Start by parameterizing $\mathbf{k_2}$ in terms of $\mathbf{k_1}$. Let $\mathbf{u_1}, \mathbf{u_2}$ be two mutually orthogonal unit vectors that are also orthogonal to $\mathbf{k_1}$, then any vector $\mathbf{k_2}$ on the cone with axis $\mathbf{k_1}$ and semi-vertical angle $\alpha_{12} $ can be written as
$\mathbf{k_2} = \cos \alpha_{12} \mathbf{k_1} + \sin \alpha_{12} (cos \phi \mathbf{u_1} + \sin \phi \mathbf{u_2}) $
Now, you want $\mathbf{k_2}$ to lie on the cone that has axis $\mathbf{k_3}$, and semi-vertical angle $\alpha_{23}$, so the angle between $\mathbf{k_2}$ and $\mathbf{k_3}$ will be $\alpha_{23} $
So you need to solve the equation
$ \mathbf{k_2} \cdot \mathbf{k_3} = \cos \alpha_{23} $
for $\phi$. Upon plugging in the expression of $\mathbf{k_2}$ the above equation becomes
$ \cos \alpha_{12} (\mathbf{k_1} \cdot \mathbf{k_3}) + \sin \alpha_{12} \left(cos \phi (\mathbf{u_1} \cdot \mathbf{k_3} ) + \sin \phi (\mathbf{u_2} \cdot \mathbf{k_3})\right) = \cos \alpha_{23} $
which is of the form
$ a \cos \phi + b \sin \phi = c $
and can be solved easily (see below). It will result in two solutions if $\alpha_{12} + \alpha_{23} \gt \beta $, one solution if $\alpha_{12} + \alpha_{23} = \beta $, and no solutions if $\alpha_{12} + \alpha_{23} \lt \beta $.
Now for the step-by-step solution of $ a \cos \phi + b \sin \phi = c $, divide through by $\sqrt{a^2 + b^2} $, resulting in
$\dfrac{a}{\sqrt{a^2 + b^2}} \cos \phi + \dfrac{b}{\sqrt{a^2 + b^2}} \sin \phi = \dfrac{c}{\sqrt{a^2 + b^2}} $
Define angle $\psi$ such that $\cos \psi = \dfrac{a}{\sqrt{a^2 + b^2}} $ and $\sin \psi = \dfrac{b}{\sqrt{ a^2 + b^2}} $, then
$ \cos \phi \cos \psi + \sin \phi \sin \psi = \dfrac{c}{\sqrt{a^2 + b^2}} $
The left hand side is simply $\cos(\phi - \psi) $, hence
$ \phi = \psi \pm \cos^{-1} \dfrac{c}{\sqrt{a^2 + b^2}} $
There will two solutions if $| c | \lt \sqrt{a^2 + b^2} $ , one solution if $ |c| = \sqrt{a^2 + b^2} $ and no solutions if $|c| \gt \sqrt{a^2 + b^2} $