If $f'$ is continuous in [a,b] use integration by parts to show Riemann–Lebesgue lemma:
$$\lim_{\lambda \to \infty}\int_a^b f(x)\sin(\lambda x) \, dx =0$$
I did the following $$\int_a^b f(x) \sin(\lambda x) \, dx = -f(x)\frac{\cos(\lambda x)}{\lambda} + \int_a^b \frac{\cos(\lambda x)}{\lambda} f'(x) \, dx$$
If then I insert limit on both sides I don't know what to do next
Since $f'$ is continuous on a closed and bounded domain $[a,b]$ it is bounded. Since $f$ is differentiable and therefore also continuous, the same conclusion can be drawn. Similarly for $\cos(\lambda x)$. Therefore as $\lambda \rightarrow \infty$ both terms in your sum tend to zero.