I've tried multiple searches but have been unable to find answer or guidance on this question (that I can follow). I also apologize in advance if my terminology is not quite correct.
Consider a right had coordinate system where y is up (i.e. x forward, y up, z out / towards you).
Now for example, consider a plane that is rotated about the x axis by 30 degrees and the z axis by -15 degrees and defined by the point P0(x0,y0,z0) as (5000, 3500, -500). Given these rotations, a normal N(a,b,c) to this plane can be defined as (-48.3, 83.65, -25.88). So in point / normal notation we can define this plane as:
$ a * (x-x0) + b * (y-y0) + c * (z-z0) = 0 $
$ -48.3(x-5000) + 83.65(y-3500) - 25.88(z+500) = 0 $
Now consider a 2d circle (r/theta notation) that lies on the xz plane at a known height y(3000) with a know radius r(5000). You can find the x / z points that lie on the circle as follows:
$ x = r * cos(\theta) = 5000 * cos(\theta)$
$ z = -r * sin(\theta) = -5000 * sin(\theta)$
Substituting the height/y of the xz plane circle and the formulas for x and z into the formula for the 3d plane you now have:
$ -48.3((5000 * cos(\theta))-5000) + 83.65(3000-3500) - 25.88((-5000 * sin(\theta))+500) = 0 $
Now, if the circle intersects the plane, there should be two values of $\theta$ that cause the above equation to equal $0$. Using an iterative solver to two decimal places, the values for this example are $\theta(18.85, 284.78)$.
In other situations there could be no solution if the circle does not intersect, one if it just touches and I suppose infinite if it lies exactly on the 3d plane.
My hope is to arrive at a series of equations that allow me to solve for $\theta$ rather than using the iterative approach.
Substituting circle into the plane we can get equation for $\theta$:
$ a (r\cos \theta-x_0) + b (h_y-y_0) + c (r\sin \theta -z_0) = 0 $ or $a\cos \theta + c \sin \theta =\frac{ax_0+cz_0-b (h_y-y_0)}{r}$
This can be rewritten as $$\sin(\theta+\alpha)=\frac{ax_0+cz_0-b (h_y-y_0)}{r\cdot\sqrt{a^2+c^2}}$$ where $\alpha=\tan^{-1}\frac{a}{c}$. The last equation has one or two solutions when right hand side is between $-1$ and $1$.