Find the Fourier transform of the function $\frac{1}{x^2-2x+10}$
This was my attempted solution
Solution: $$F(k) = \frac{1}{2\pi} \int_{-\infty}^{\infty}\frac{e^{-ikz}}{z^2-2z+10}dz$$
Then note that $$g(z) = \frac{e^{-ikz}}{z^2-2z+10}$$ has singularities at $z_0 = 1+3i$ and $z_1 = 1-3i$. Now we'll apply the residue theorem.
$$\int_{-\infty}^{\infty}\frac{e^{-ikz}}{z^2-2z+10}dz = 2\pi i \sum_{k=0}^1 \operatorname{Res}_{z=z_k}f(z)$$
Only $z_0$ is in the upper half plane, and calculating the residue at this point we get $$\operatorname{Res}_{z=z_0} f(z) = \frac{e^{3k}\left(\cos k - i\sin k\right)}{6i}$$ thus we get
$$\int_{-\infty}^{\infty}\frac{e^{-ikz}}{z^2-2z+10}dz = 2\pi i \sum_{k=0}^1 \operatorname{Res}_{z=z_k}f(z) = 2\pi i \operatorname{Res}_{z=z_0} f(z) = \frac{\pi e^{3k}\left(\cos k - i\sin k\right)}{3}$$
However the correct answer is $$\frac{e^{-3|k|-ik}}{6}$$
so where have I gone wrong in my solution?
The good news is that $\cos(k)-i\sin(k)=e^{-ik}$; this means that one part of your answer that maybe you think is wrong is actually right.
You dropped the $\frac1{2\pi}$; putting that back will change your $\frac\pi3$ to $\frac16$.
The biggest error is this: You have a magical version of the Residue Theorem in your head that says $\frac1{2\pi i}\int_{-\infty}^\infty$ is always the sum of the residues in the upper half plane. The Residue theorem does not say that! It says something about the integral over a closed curve; $(-\infty,\infty)$ is not a closed curve.
You're ignoring a large part of the argument in the examples you've seen of applying the Residue Theorem. The RT says something about the integral over a semicircle; now you have to show the integral over the circular part tends to zero.
And here's the deal on that: If $k<0$ then $$\lim_{y\to+\infty}e^{-ik(iy)}=0.$$ If $k>0$ that same limit is $\infty$.
So, in order to show the integral over the circular part of the contour tends to zero, you need to use a semicircle in the upper half-plane for $k<0$ and a semicircle in the lower half-plane for $k>0$. This will give you two different answers, depending on the sign of $k$; then finally it turns out that the two answers can be expressed as one formula in terms of $|k|$.
(Sure enough, if $k<0$ then $e^{-3|k|}=e^{3k}$; except for the dropped $1/2\pi$ your answer is actually right, for $k<0$.)