$$I=\cot^{-1}\frac{y}{\sqrt{1-x^2-y^2}} = 2\tan^{-1}\sqrt{\frac{3-4x^2}{4x^2}} - \tan^{-1}\sqrt{\frac{3-4x^2}{x^2}} $$
Expression of $I$ as a rational integer equation in $x$ and $y$ is $$ (a) 27x^2=y^2(9-8y^2) \\(b) 27y^2=x^2(9-8x^2) \\(c) 27x^4 - 9x^2 + 8y^2=0 \\(d) 27y^4 - 9y^2 + 8x^2=0 $$
My attempt : I tried to assume $$x=\frac{\sqrt{3}}{2}sin\theta$$ That reduces the first arctan function to a simple expression $2\theta$, but the second one assumes the form $\tan^{-1}(2\tan\theta)$. How can I find the correct answer? Any help will be appreciated.
Hint:
Let $\sqrt{\dfrac{3-4x^2}{x^2}}=2a$
$$2\tan^{-1}a-\tan^{-1}(2a)=\tan^{-1}a+\tan^{-1}a+\tan^{-1}(-2a)$$
$$\tan\left(\tan^{-1}a+\tan^{-1}a+\tan^{-1}(-2a)\right)=\dfrac{a+a-2a+4a^3}{1-a^2-a(-2a)-a(-2a)}$$
which is $$=\dfrac{\sqrt{1-x^2-y^2}}y$$
Now square both sides and replace the value of $a^2$