I have a function $f$ defined by $$f(x)=\begin{cases} xe^{-x} \textrm{ if } x>0,\\ 0,\textrm{otherwise}. \end{cases}$$
I wish to know the Fourier transform of $f$, i.e, $${\cal F}(f(x))=\frac1{\sqrt{2\pi}}\int_{-\infty}^{\infty}f(x)e^{-ixy}dx.$$
I could do this with two different methods using the fact that$${\cal F}(f'(x))=iy{\cal F}(f(x)):$$
Note that $f'(x)=e^{-x}-xe^{-x}=e^{-x}-f(x).$ So,$$iy{\cal F}(f(x))={\cal F}(f'(x))={\cal F}(e^{-x})-{\cal F}(f(x))$$ that leads to $$ {\cal F}(f(x))=\frac{{\cal F}(e^{-x})}{1+iy}=\frac1{\sqrt{2\pi}(1+iy)^2}. $$
Put $g(x)=-e^{-x}-xe^{-x}$, then $g'(x)=f(x).$ So,$${\cal F}(f(x))={\cal F}(g'(x))=iy{\cal F}(g(x))=-iy{\cal F}(e^{-x})-iy{\cal F}(f(x))$$ that leads to $$ {\cal F}(f(x))=-iy\frac{{\cal F}(e^{-x})}{1+iy}=-iy\frac1{\sqrt{2\pi}(1+iy)^2}. $$
My question is: why are they different? Isn't the image of $f$ unique?
First note your function is summable, so the Fourier transform is really defined by the integral, as you write. However your function $f(x)$ is really not differentiable. Indeed you are looking at the Fourier transform of $f(x) = \theta(x) x e^{-x}$ where $\theta$ is Heaviside function.
The integral can be done by integration by part (which is almost what you are doing there). Setting $k= 1+iy$ one has
$$ \hat{f}(y) = \frac{1}{\sqrt{2\pi}} \int_0^\infty e^{-k x} x dx = \frac{1}{\sqrt{2\pi}} \int_0^\infty \frac{e^{-kx}}{k} dx =\frac{1}{\sqrt{2\pi}} \frac{1}{k^2}=\frac{1}{\sqrt{2\pi}} \frac{1}{(1+i y)^2}\,. $$
Note that the border term in the integration by part formula is zero.