I'm trying to refactor/solve the following equation for theta if $ \vec{v} , \vec{r} $ and $ \vec{n} $ are known vectors:
$ \left(\vec{v} \cdot \cos\left(\theta\right) + \left(\vec{r} \times \vec{v}\right) \cdot \sin\left(\theta\right) + \vec{r} \cdot \left(\vec{r} \cdot \vec{v}\right) \cdot \left(1 - \cos\left(\theta\right)\right)\right) \cdot \vec{n} = 0 $
If this looks familiar, it's probably because it is - it's a combination of the Rodrigues equation for rotating a vector (in this case, $ \vec{v} \,$) around an axis $ \left(\vec{r} \right)$ and a simple dot operation to determine the distance of the resulting vector to a given plane defined by a normal $ \left(\vec{n}\right) $.
In essence, I'm trying to solve for $ \theta $ such that the vector $ \vec{v} $ will be rotated onto plane $ \vec{n} $ if possible. Near as I can tell, there should be either $ 0, 1 $ or $ 2 $ solutions to this problem - making me think that it might be possible to refactor this into something that can be solved by a quadratic equation.
Alternatively (and I have no idea how this would work) - I'm guessing it might also be possible to somehow "project" $ \vec{v} $ around the axis $ \vec{r} $ so that it falls on the plane with a normal defined by $ \vec{n} $ and then calculate the angle between the original vector and the projected vector $ \left(\vec{s}\right) $, but I'm not really sure how you would do that.
How can I solve this problem?
In the case where the cone intersects the plane orthogonal to $\vec{n}$ (that we will call the horizontal plane) along two of its generatrices, let $2\varphi$ be the angle between these 2 generatrices.
Let
$$\alpha=angle(r,v) \ \text{and} \ \ \beta=angle(r,n)$$
and $$a:=\frac{1}{\tan(\alpha)\tan(\beta)}$$
$$\sin \varphi = a =\frac{1}{\tan(\alpha)\tan(\beta)}\tag{1}$$
(using a certain convention for angles that will be clear from the explanation below).
Here is why.
I use for the proof unitarized vectors $v,r,n$, without loss of generality.
Let us consider the orthonormal basis of the plane orthogonal to $r$ :
$$V_1=\tfrac{1}{k}(r \times n), \ \ V_2=\tfrac{1}{k}(r \times n) \times r \tag{2}$$
where $k=\|r \times n \|= \sin \beta$ is a normalization factor.
Considering now the 3D orthonormal basis $V_1,V_2,r$, we have the following decomposition of the rotated vector $v_{\varphi}$ of $v$ around $n$:
$$v_{\varphi}=\sqrt{1-(v \cdot r)^2}\left(\cos \varphi \ V_1 + \sin \varphi \ V_2\right) \ + \ (v \cdot r) \ r$$
(sanity check : verify that $v_{\varphi}$ has a unit norm)
Now, let us express that we must have the following zero dot product (in the same way as you have expressed the problem):
$v_{\varphi} \cdot n = 0 \tag{3}$
It suffices now to replace in (3) $V_1$ and $V_2$ by their expressions from (2), and take into account the fact that $r \cdot n = \cos \beta$ and $r \cdot v = \sin \beta$ gives relationship (1).
Edit: it remains to get angle $\theta$ from angle $\varphi$ or more precisely from the knowledge of $V_1$ and $V_2$. The idea is to operate in the plane defined by points $V_1,V_2,v$ ("tips of associated vectors"), whose center is the projection of the origin. In this way the rotation angle bringing $v$ onto $V_1$ or to $V_2$ for example will be easily obtained.
Fig. 1: $\textit{All vectors are with unit length. If one projects $V_1,V_2,v$ on the plane}$ $\textit{ defined by the "tips" of $V_1,V_2,v$, one has a circle where one can visualize}$ $\textit{ in true size angles $\theta=\angle(v,V_1)$ and $\theta=\angle(v,V_2)$ and act on them}$.