Here is the integral I want to evaluate:
$$\int_{0}^{2\pi} \frac{dx}{a+b \cos x }, \quad a>b >0$$
Apparently there are limitations as to what values $a, b$ are supposed to take but let us not concern about this. Since using the sub $u =\tan \frac{x}{2}$ (the Weiersstrass sub) results that the integral is $0$ (as it should, since it is not $1-1$ function in this interval) I got down down the way of contour integration. Hence:
$$\begin{aligned} \int_{0}^{2\pi}\frac{dx}{a+ b\cos x} &\overset{x=i \ln u}{=\! =\! =\!} \oint \limits_{|z|=1} \frac{dz}{iz \left [ a + \frac{b}{2}\left ( z+z^{-1} \right ) \right ]} \\ &= \frac{1}{i} \oint \limits_{|z|=1} \frac{dz}{za + \frac{bz^2}{2}+\frac{b}{2}}\\ &=\frac{2}{i} \oint \limits_{|z|=1} \frac{dz}{bz^2 +b +2za} \\ &= \frac{2}{i} 2\pi i \sum_\text{residues} f(z)\\ &= \frac{4\pi}{1+ \sqrt{1-8ab}+2a} \end{aligned}$$
However, judging by intuition this must not be the result. Because this one restricts the integral too much. What i mean is, that for $a=6, b=3$ we have that:
$$\int_0^{2\pi} \frac{dx}{6+3\cos x}= \frac{2\pi}{3\sqrt{3}}$$
My formula cannot derive the result because then radical would be negative. What am I doing wrong here?
Your transformation to a contour integral is correct; we note that $$ \int_0^{2\pi}\frac{dx}{a+b\cos x}=\int_0^{2\pi}\frac{dx}{a+\frac{b}{2}(e^{ix}+e^{-ix})} $$ and set $z=e^{ix}$ (a parametrization of the unit circle) so that $dz=ie^{ix}dx=izdx\to dx=\frac{dz}{iz}$. This yields $$ \int_0^{2\pi}\frac{dx}{a+b\cos x}=\oint_{|z|=1} \frac{dz}{iz(a+\frac{b}{2}(z+\frac{1}{z}))}. $$
The problem is in your residue calculation. By the quadratic formula, the zeros of $bz^2+2az+b$ are $\frac{-a\pm\sqrt{a^2-b^2}}{b}$. By Vieta's formula, the product of the roots is $1$ so only one is within the unit disc; clearly it must be $w=\frac{-a+\sqrt{a^2-b^2}}{b}$. You can then calculate the residue $$ \text{res}_{w}\left(\frac{1}{bz^2+2az+b}\right) = \lim_{z\to w}\frac{1}{b\left(z-\frac{-a-\sqrt{a^2-b^2}}{b}\right)}= \frac{1}{2\sqrt{a^2-b^2}} $$ This will give you a final result of $\frac{2\pi}{\sqrt{a^2-b^2}}$.