I need to compute the Fourier transform of the following function: $f(x)=|x| \exp(-t|x|)$ where $t>0, \ t \in \mathbb{R}$.
Since this is an even function only the real part of the Fourier transform will contribute so the integral becomes:
$\frac{2}{\sqrt{2\pi}}\int_{0}^{\infty}x\exp(-t x)\cos(\lambda x) dx$
which according to my textbook can be rewritten by using the following rule:
$\mathcal{F}[x^nf(x)](\lambda)=i^n \frac{d^n}{d\lambda^n} \mathcal{F}[f](\lambda)$
so that we have:
$\frac{2}{\sqrt{2\pi}}i \frac{d}{d\lambda} \left(\int_{0}^{\infty}\exp(-t x)\cos(\lambda x)dx\right)=\frac{2}{\sqrt{2\pi}}\frac{-2\lambda it}{(\lambda^2+t^2)^2}$
However this is not equivalent to the first integral:
$\frac{2}{\sqrt{2\pi}}\int_{0}^{\infty}x\exp(-t x)\cos(\lambda x) dx=\frac{2}{\sqrt{2\pi}}\frac{t^2-\lambda^2}{(\lambda^2+t^2)^2}$
Which is the right answer. You can arrive at the same answer by using the rule from the textbook but replacing the cosine term with an exponential and taking two times the real part like this:
$\frac{2}{\sqrt{2\pi}}\text{Re}\left[i \frac{d}{d\lambda} \left(\int_{0}^{\infty}\exp(-t x)\exp(-i\lambda x)dx\right)\right]=\frac{2}{\sqrt{2\pi}}\frac{t^2-\lambda^2}{(\lambda^2+t^2)^2}$
Why does the rule from the textbook not work in the case where I'm using $\cos(\lambda x)$? Is that not equivalent to taking the real part since I've just thrown away the $i\sin(\lambda x)$ term of the integral because it's going to be an odd function integrated over a symmetric interval so it'll be 0 anyways?
Your original formula does work. The problem is that the added $x$ term turns an even integrand $|x|e^{-t|x|}$ into the odd one $x|x|e^{-t|x|}$. And that also makes sense directly because $\frac{d}{d\lambda}\cos(\lambda x)=-x\sin(\lambda x)$. The error comes in thinking that the derivative is going remain a Fourier cosine transform instead of switching to a Fourier sine transform.
For example, $$ \frac{d}{ds}\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}e^{-|x|t}e^{-isx}dx = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}(-ix)e^{-|x|t}e^{-isx}dx $$ The integral on the left may be written as a cosine transform, while the integral on the right is a sign transform. So, without changing the derivative, but rewriting the integrals only, $$ \frac{d}{ds}\sqrt{\frac{2}{\pi}}\int_{0}^{\infty}e^{-|x|t}\cos(sx)ds =-\sqrt{\frac{2}{\pi}}\int_{0}^{\infty}xe^{-|x|t}\sin(sx)dx $$ And, you can see that the second identity also follows by interchanging the derivative with respect to $s$ and the derivative with respect to $s$. You can compute these \begin{align} \int_{0}^{\infty}e^{-|x|t}\cos(sx)ds &=\Re \int_{0}^{\infty}e^{-xt}e^{isx}dx\\ &= \Re \left(\left.\frac{1}{is-t}e^{x(is-t)}\right|_{x=0}^{\infty}\right) \\ &= \Re \frac{1}{t-is} = \Re \frac{(t+is)}{t^2+s^2}=\frac{t}{t^2+s^2}. \end{align} That gives $$ \int_{0}^{\infty}xe^{-xt}\sin(sx)dx = -\frac{d}{ds}\frac{t}{t^2+s^2} = \frac{2ts}{(t^2+s^2)^2} $$ Everything seems to work out: \begin{align} \int_{-\infty}^{\infty}(-ix)e^{-|x|t}e^{-ixs}dx & = \int_{-\infty}^{0}(-ix)e^{xt}e^{-isx}dx+\int_{0}^{\infty}(-ix)e^{-xt}e^{-ixs}dx \\ & = \left.(-ix)\frac{e^{x(t-is)}}{t-is}\right|_{-\infty}^{0}-\int_{-\infty}^{0}(-i)\frac{e^{x(t-is)}}{t-is}dx \\ & + \left.(-ix)\left(-\frac{e^{-x(t+is)}}{t+is}\right)\right|_{0}^{\infty} -\int_{0}^{\infty}i\frac{e^{-x(t+is)}}{t+is}dx \\ & =\int_{-\infty}^{0}i\frac{e^{x(t-is)}}{t-is}dx-\int_{0}^{\infty}i\frac{e^{-x(t+is)}}{t+is}dx \\ & = \left.i\frac{e^{x(t-is)}}{(t-is)^2}\right|_{x=-\infty}^{0} + \left. i\frac{e^{-x(t+is)}}{(t+is)^2}\right|_{0}^{\infty} \\ & = \frac{i}{(t-is)^2}-\frac{i}{(t+is)^2} \\ & = \frac{i(t+is)^2}{(t^2+s^2)^2}-\frac{i(t-is)^2}{(t^2+s^2)^2} \\ & = -\frac{4st}{(t^2+s^2)^2} \end{align} The factor of $2$ comes from the difference in Fourier coefficients. I'll let you sort out the $-1$.
You can do the same thing when you differentiate with respect to $t$ as well.