Let's consider the following equation:
$$R = \frac{p D\cos\theta}{p - D \sin\theta}$$
Where $R,p,D>0$ and $-\pi/2\le\theta\le0$.
The goal is to transform this formula in order to write something like $\theta = f(R,p,D)$
My first try was to write the equation like below:
$$Rp = D(p\cos\theta + R\sin\theta)$$
Then considering the case where $p=R$, I was able to do this:
\begin{align} R & = D(\cos\theta + \sin\theta) \\ & = D\sqrt{2}[\sin(\pi/4)\cos\theta + \cos(\pi/4)\sin\theta] \\ & = D\sqrt{2}\sin(\pi/4 + \theta) \\ \theta& = \sin^{-1}\left(\frac{R}{D\sqrt{2}}\right) - \frac{\pi}{4} \\ \end{align}
I can do the same considering the angles $\pi/3$ and $\pi/6$ but in all the cases I had to fix the relation between $p$ and $R$ which is not what I want.
I also tried to use the following equation:
$\cos(x)=\frac{1-t^2}{1+t^2}$ , $\sin(x)=\frac{2t}{1+t^2}$ where $t = \tan(\frac{x}{2})$
I ended with a second degree equation
$$(\frac{R}{D} - 1)t^2 - 2\frac{R}{p}t + (\frac{R}{D} - 1) = 0$$
I calculated the solution but I obtained wrong values when testing with the initial formula. probably I missing something on the road.
Is there a magic step or an equation that can help me find $\theta = f(R,p,D)$?
For the reference, this question is a extension of my answer in StakOverflow and below is the figure from where the initial equation was extracted. Probably I didn't extract the good equation and there is a better one that suits my needs.
From here: $$ Rp = D(p\cos\theta + R\sin\theta) $$ divide through by $D\sqrt{p^2+R^2}$ to get $$ \frac{Rp}{D\sqrt{p^2+R^2}} = \frac{p}{\sqrt{p^2+R^2}}\cos\theta + \frac{R}{\sqrt{p^2+R^2}}\sin\theta = \sin\left(\theta+\tan^{-1}\frac{p}{R}\right). $$ Solving for $\theta$ gives $$ \theta = \sin^{-1}\left(\frac{Rp}{D\sqrt{p^2+R^2}}\right)-\tan^{-1}\left(\frac{p}{R}\right), $$ which agrees with what you found in the $p = R$ case and will always be negative if $R < D$, as it is in your diagram.
There's actually a way to get this same answer from simple trigonometry. The law of sines says that the angle $\phi$ between the red and brown lines satisfies $$ \frac{\sin\phi}{p} = \frac{\sin O}{D}\;\; \Longrightarrow\;\;\phi = \sin^{-1}\left(\frac{pR}{\sqrt{p^2+R^2}}\right) $$ Meanwhile, the angle $\psi$ between the brown and black lines is just $\tan^{-1}(p/R)$. Since the exterior angle of a triangle is the sum of the other two interior angles (and we're treating $\theta$ as negative), we have $-\theta + \phi = \psi$, or $$ \theta = \phi -\psi = \sin^{-1}\left(\frac{pR}{\sqrt{p^2+R^2}}\right) - \tan^{-1}\left(\frac{p}{R}\right) $$