I have the system: \begin{align} \tag{1} y_t&=ay_{xx}+by_x, \ \ \ x\in\mathbb{R},t>0,\\ \tag{2}y(x,0)&=f(x) , \qquad \ \ \ \ \ \ x\in\mathbb{R} \end{align} where $y$ is bounded and $$\lim_{|x|\rightarrow +\infty}y_x=0.$$ Also $f$ is sufficiently smooth, and $a,b\in\mathbb{R}$.
I want to transform (1) and (2) using the Fourier transformation. $$\hat y(λ,t)=\int^{\infty}_{-\infty}y(x,t) e^{iλx}dx$$
\begin{align} \tag{3} \hat{y}_{xx}(λ,t)&=\int^{\infty}_{-\infty}y(x,t)_{xx} e^{iλx}dx=\dots \\ &=-λi\cdot \lim_{a\rightarrow\infty}y(a,t)e^{iλa}+λi \lim_{a\rightarrow -\infty }y(a,t)e^{iλa}+λ^2 i^2 \hat y(λ,t)\end{align}
and \begin{align} \tag{4} \hat y_x(λ,t) &=\int^{\infty}_{-\infty}y_x(x,t) e^{iλx}dx=\dots \\ &= \lim_{a\rightarrow\infty}y(a,t)e^{iλa}- \lim_{a\rightarrow -\infty }y(a,t)e^{iλa}-λ i \hat y(λ,t) \end{align}
and $$\tag{5} \hat y_t(λ,t)=\frac{d}{dt} \left (\int^{\infty}_{-\infty}y e^{iλx}dx \right )$$
$$(1),(3),(4),(5) \Rightarrow \hat {y}_t +(aλ^2 +iλ)\hat u(λ,t)=(b-aλi)\left [\lim_{a\rightarrow \infty}y(a,t)e^{iλa}-\lim_{a\rightarrow -\infty}y(a,t)e^{iλa} \right]$$
That is the transformation I find, is it correct?
How can I eliminate the limits? Can I substitute them with a number? If $$\lim_{x\rightarrow \infty}y(x,t)=0$$ then I eliminate them. But I don't know that.
You need $\lim_{x\rightarrow \pm\infty}y(x,t)=0$ for the Fourier transform to exist, so you have assumed this implicitly in defining $\hat{y}(\lambda,t)$. It then follows that the boundary terms coming from integration by parts when calculating $\hat{y}_x(\lambda,t)$ and $\hat{y}_{xx}(\lambda,t)$ will be zero.