The question is pretty clear, we must find the inverse function of $$ {f:x \rightarrow \tan^2(x) - 2 \sqrt{3}(\tan(x))}$$
This function is a bijection from $(-\frac {\pi}{2}+kπ, \frac {\pi}{3}+kπ)$ to $(-3,+\infty)$ since it is strictly decreasing and continuous.
I've tried factorizing with tangent and playing with the equation so I can add Arctangent to eliminate tangent like with usual functions but I do not seem to be capable succeeding using this method.
P.S: This function is also a bijection from $(-\frac {\pi}{3}+kπ, \frac {\pi}{2}+kπ)$ to $(-3,+\infty)$.
Consider$$\begin{array}{rccc}p\colon&\left[-1,\sqrt3\right]&\longrightarrow&\left[1+2\sqrt3,-3\right]\\&x&\mapsto&x^2-2\sqrt3x.\end{array}$$Then $f=p\circ\tan$ and therefore $f^{-1}=\tan^{-1}\circ p^{-1}=\arctan\circ p^{-1}$. Finally, $p^{-1}(x)=\sqrt3-\sqrt{x+3}$ and therefore$$f^{-1}(x)=\arctan\left(\sqrt3-\sqrt{x+3}\right).$$