So for this problem I need to use the fact that $\frac {1-r^2}{1-2r\cos\theta+r^2}$=$1+2\sum_{n=1}^{\infty} r^n\cos n\theta$.
I replaced the term in the integral but i ended up getting $\sum_{n=1}^{\infty} r^n+2$ as my final answer. Im not sure if I got it right and i need to reframe it to get $\frac {2(1+r^2)}{1-r^2}$ or my answer is completely wrong.
Thanks for the help.
A much simpler way is to note that $$\int_{0}^{\pi}\frac{dx}{a+b\cos x} =\frac{\pi} {\sqrt{a^2-b^2}}, a>|b|$$ and differentiate it with respect to $a$ to get $$\int_{0}^{\pi}\frac{dx}{(a+b\cos x) ^2}=\frac{\pi a} {(a^2-b^2)^{3/2}}$$ Putting $a=1+r^2,b=-2r$ and noting that $$a^2-b^2=(1-r^2)^2$$ we can see that $$\int_{0}^{\pi}\frac{dx}{(1-2r\cos x+r^2)^2}=\frac{\pi(1+r^2)}{(1-r^2)^3}$$ which is equivalent to your result in question.
The integral at the beginning of this answer is evaluated by using non-obvious substitution $$(a+b\cos x) (a-b\cos t) =a^2-b^2$$
Your own approach should lead to $$\int_{0}^{\pi}\left(\frac{1-r^2}{1-2r\cos x+r^2}\right)^2\,dx=\pi+4\sum_{n=1}^{\infty}r^{2n}\int_{0}^{\pi}\cos^2 nx\, dx$$ which equals $$\pi+2\pi\sum_{n=1}^{\infty}r^{2n}=\frac{\pi(1+r^2)}{1-r^2}$$ Note that while squaring you also get terms like $\cos mx\cos nx$ but their integral vanishes so I have ommitted these in the above calculation.
Also $x$ has been used in place of $\theta$ to reduce typing effort.