I have the function
$$f(t):=\frac{1-r^2}{1+r^2-2r\cos t}=\frac{1-r^2}{|1-re^{it}|^2},\quad t\in\Bbb R\tag1$$
Hence
$$\begin{align}\hat f_k&=\frac1{2\pi}\int_0^{2\pi}f(t)e^{-ikt}\, dt\\&=\frac1{2\pi}\int_0^{2\pi}\frac{1-r^2}{(1-re^{it})(1-re^{-it})e^{ikt}}\cdot\frac{r^k\cdot ire^{it}}{r^k\cdot ire^{it}}\, dt\\ &=\frac{r^k}{2\pi i}\int_0^{2\pi}\frac{(1-r^2) ire^{it}}{(1-re^{it})(re^{it}-r^2)(re^{it})^k}\, dt\\ &=\frac{r^k}{2\pi i}\int_{\partial\Bbb D(1,r)}\frac{1-r^2}{(1-z)(z-r^2)z^k}\, dz,\quad\text{for } z=re^{it}\\&=\frac{r^k}{2\pi i}\int_{\partial\Bbb D(1,r)}\frac{g(z)}{z-1}\, dz\end{align}\tag2$$
for $g(z):=-\frac{1-r^2}{z-r^2}z^{-k}$ what is well defined when $|r|\neq 1$ and $z\notin\{0, r^2\}$. Then for $|r|<1/2$ clearly $g$ is holomorphic on $\overline{\Bbb D}(1,r)$, thus we can apply the Cauchy integral formula to find that
$$\hat f_k=r^k g(1)=-r^k\tag3$$
However this result is wrong. The correct answer is $\hat f_k=r^{|k|}$ but I cant see where are my mistakes. Some help will be appreciated.
Your integration path is wrong. It must be $\partial D(0,r)$, because $z=r e^{it}$ is a parametrisation of $\partial D(0,r)$.
Assuming $0<r<1$ we get $$f_k = \frac{r^k}{2\pi \rm{i}} \int_{\partial D(0,r)} h(z)\mathop{dz}, $$ where $$h(z) = \frac{1-r^2}{(1-z)(z-r^2)} \frac{1}{z^k}.$$ This function has also two poles in $\overline{D(0,r)}$, namely in $z=0$ (depending on $k$) and $z=r^2$, because $0<r<1$ and thus $r^2 <r$.
If $k\leq 0$, then we have only a pole in $z=r^2$ with residue $r^{-2k}$ and we get by the residue-theorem $$f_k = r^{-k}.$$ If $k>0$, then we have again the residue $r^{-2k}$ in $z=r^2$ and also in $z=0$ the residue $$-r^{-2}(1-r^2) \sum_{j=0}^{k-1}r^{-2j}.$$ The last residue can be determined by rewritting both $(1-z)^{-1}$ and $(1-z/r^2)^{-1}$ with the help of the the geometric series. A short calculation shows that the sum of both residues is $1$. Thus, we get also in the second case that $f_k = r^k$.