Using lebesgue dominated convergence, calculate $\displaystyle \lim_{n\to \infty}\int_a^{\infty} n e^{-nx} \cos{x}\, dx$ when $a > 0$ and $a = 0$.
I pretty much understand when $a>0$ since $ne^{-nx} \cos{x}$ is approaching $0$. But, there is no dominating function when $x=0$ since $n e^{-nx} \cos{x}$ is becoming infinity.
As you determined,
$$\lim_{n \to \infty}ne^{-nx}\cos x = \begin{cases} 0 & x>0, \\ \infty & x = 0.\end{cases}$$
For $a = 0$, there is no dominating integrable function on $[0, 1]$ and the LDCT does not apply.
Note that on $[0,1],$
$$ne^{-nx}\left(1 - x^2/2\right) \leqslant ne^{-nx}\cos x \leqslant ne^{-nx},$$
and
$$\int_0^1 ne^{-nx}\left(1 - x^2/2\right)\,dx \leqslant \int_0^1 ne^{-nx} \cos x\,dx \leqslant \int_0^1 ne^{-nx}\,dx .$$
Using integration by parts twice for the integral on the LHS, we get
$$1 - \frac1{n^2} - \left(\frac1{2}- \frac1{n} - \frac1{n^2}\right)e^{-n} \leqslant \int_0^1 ne^{-nx} \cos x\,dx \leqslant 1 - e^{-n}.$$
Using the squeeze principle,
$$\lim_{n \to \infty}\int_0^1 ne^{-nx} \cos x\,dx = 1 \\ \neq \int_0^1 \lim_{n \to \infty}[ne^{-nx} \cos x]\,dx = 0.$$
On $[1, \infty),$
$$\left| \int_1^{\infty} ne^{-nx} \cos x\,dx\right| \leqslant \int_1^{\infty} |ne^{-nx} \cos x|\,dx \\ \leqslant \int_1^{\infty} ne^{-nx}\,dx = e^{-n}.$$
Hence,
$$\lim_{n \to \infty}\int_0^{\infty} ne^{-nx} \cos x\,dx = 1 .$$