How can I solve the following differential equation?
$$y''+y=xe^x\cos(x)$$
I've studied the (1) Undeterminate Coefficients, (2) Variation of Parameters and (3) Laplace Transforms methods, but I don't know which of them to use since $xe^x\cos(x)$ is too complicated for (1), I don't know how to solve the integral I obtain with (2) and I don't have the initial values for (3). What should I do?
For Laplace transforms, define $Y(s)$ as the LT of $y(x)$.
Use integration by parts to establish that the LT of $y''(x)$ is $s^2 Y(s) - y_0 s - y'_0$, where $y_0=y(0)$ and $y'_0 = y'(0)$.
You will also need to take the LT transform of the RHS:
$$\begin{align}\int_0^{\infty} dx \, x \, e^x \, \cos{x} \, e^{-s x} &= \int_0^{\infty} dx \, x \cos{x} \, e^{-(s-1) x}\\ &= - \frac{d}{d(s-1)} \Re{\left [\int_0^{\infty} dx \, e^{-[(s-1)-i] x} \right ]}\\&= -\frac{d}{d(s-1)} \left [\frac{s-1}{(s-1)^2+1}\right ] \\ &= \frac{(s-1)^2-1}{[(s-1)^2+1]^2} \end{align} $$
Then
$$Y(s) = \frac{(s-1)^2-1}{[(s-1)^2+1]^2 (s^2+1)} + \frac{y_0 s+y'_0}{s^2+1}$$
To find $y(x)$, apply an inverse LT either using tables or the residue theorem.