I have a hemisphere centered on the origin of radius R. I also have a plane which runs through the origin, is parallel to the y-axis, and is tilted at an angle $\alpha$ with respect to the z-axis.
An image is below (the dotted line represents the plane):
From the setup we can write: $$\hat{n} = -\cos(\alpha)\hat{x} + \sin(\alpha)\hat{z}$$ which means the equation of the plane is: $$0=-\cos(\alpha)x + \sin(\alpha)z$$ The equation of the sphere is: $$R^2 = x^2 + y^2 +z^2$$ Setting these equations equal, converting to spherical coordinates, and solving for $\theta(\phi,\alpha)$, gives:
$$\theta(\phi,\alpha) = \arcsin[(\cos^2(\phi)\csc^2(\alpha) +\sin^2(\phi))^{-1/2}]$$
I made a polar plot of this in python by parameterizing this equation for $\phi = [0,180]$. I set values on one side of the intersection to 1 and the other side to zero.
I expected to see a crescent shape across the polar plot, however when values approach the minimum polar angle, I notice a cusp that I don't believe should be there:
The above image shows the plane tilted at around 20 degrees.
I'm not sure where the error lies, but would appreciate any feedback.
As you’ve already worked out, an equation of the plane is $x\cos\alpha=z\sin\alpha$. Substituting spherical coordinates into this equation results in $\rho\cos\phi\sin\theta\cos\alpha = \rho\cos\theta\sin\alpha$, or $$\cos\phi\tan\theta = \tan\alpha. \tag1$$ Now, the polar coordinates of the orthogonal projection onto the $x$-$y$ plane of the point with Cartesian coordinates $(\rho\cos\phi\sin\theta,\rho\sin\phi\sin\theta,\rho\cos\theta)$ are $$(\rho\sin\theta,\phi).\tag2$$ From equation (1) we have $$\sin\theta = {\tan\theta \over \sqrt{1+\tan^2\theta}} = {\tan\alpha\sec\phi \over \sqrt{1+\tan^2\alpha\sec^2\phi}} = {1\over\sqrt{1+\cot^2\alpha\cos^2\phi}}\tag3$$ and $\rho=R$, therefore the polar equation of the projected curve is $$r(\phi) = {R \over \sqrt{1+\cot^2\alpha\cos^2\phi}}. \tag 4$$ The expression for $\sin\theta$ in (3) is not equal to the one that you derived, but if you replace the expression inside the $\arcsin$ in yours with its reciprocal they are equal, so it looks like you got a fraction flipped somewhere along the way.
It’s also possible to find a Cartesian equation of the projected curve fairly directly: the plane-sphere intersection is a circle, so its projection onto the $x$-$y$ plane is an axis-aligned ellipse centered at the origin. The major axis lies along the $y$-axis and the semiaxis length is simply $R$. The semiminor axis length is foreshortened to $R\sin\alpha$, so a Cartesian equation of the ellipse is $${x^2\over R^2\sin^2\alpha}+{y^2\over R^2}=1. \tag 5$$ If you plug these semiaxis lengths into the generic polar equation of an ellipse, you’ll end up with something equivalent to equation (4). Another relatively painless approach to finding an equation of the curve is to parameterize the circle as $(R\cos t)(\sin\alpha,0,\cos\alpha)+(R\sin t)(0,1,0).$ Projecting this onto the $x$-$y$ plane is a simple matter of dropping the $z$-coordinate to get $(R\sin\alpha\cos t,R\sin t)$. Eliminating $t$ produces equation (5) again. Note, though, that $t$ in this parameterization doesn’t represent the polar angle.