I am stuck trying to understand an example in a book, they skipped a few steps that I don't understand. I have the ODE $$x''+\omega^2 x+\lambda x^3=0,\quad \lambda<<1 $$ as $\lambda$ is small I can use perturbation theory $$x(t)=x_0+\lambda x_1\dots$$ then I can equate coefficients in powers of $\lambda$.
Then the first order is $$x_0''+x_0=0$$ with initial condition $x_0'(0)=0$ and $x_0(0)=a$ I get $$x_0=a\cos(\omega t).$$ Then I can equate up to terms in $\lambda$, I get $$x_1''+\omega^2 x_1=-x_0^3,\quad x_1(0)=0, x_1'(0)=0.$$ The homogenus solution is just $x_{h1}=c_1\cos(\omega t+\phi),$ the RHS can be rewritten to $$-\frac{a^3}{4}(\cos(3\omega t)+3\cos(\omega t)).$$ Then with the method of undetermined coefficients $$x_{1p}=C\cos 3t+Dt\cos t+Et\sin t$$ and after I plug it in the ODE I get $$-9\cos 3t-2D\sin t -Dt\cos t+2E\cos t-Et\sin t+\omega^2(C\cos 3t+Dt\cos t+Et\sin t)=-\frac{a^3}{4}(\cos(3\omega t)+3\cos(\omega t))$$. and I equate terms $$-9C\cos 3t+C\omega^2\cos 3t=-\frac{a^3}{4}\cos 3t\Rightarrow C=\frac{-a^3}{4(-9+\omega^2)}$$ $$2E\cos t=\frac{-a^3}{4}\cos t\Rightarrow E=-\frac{3a^3}{8}$$ but then I get that $$-Et\sin t+\omega^2Et\sin t =0$$ which is strange, I get that $D=0$ as well. I can see by the conclusion of the example that I get the wrong answer, but I don't see where I screw up.
Set $ω=1$ to avoid all omissions and wrong powers of it, then, as you found mostly correctly, $$ x_{1p}''+x_{1p}=-8C\cos(3t)−2D\sin t+2E\cos t = -\frac{a^3}4(\cos(3t)+3\cos(t)) $$ Comparing the coefficients of like terms one finds $D=0$, $C=\dfrac{a^3}{32}$ and $E=-\dfrac{3a^3}{8}$ giving you $$ x_1(t)=\dfrac{a^3}{32}(\cos(3t)-\cos(t))-\dfrac{3a^3}{8}t\sin t $$
One can integrate the growing term as part of a Taylor expansion as \begin{align} x_0+λx_1&=a(\cos(t)-\sin(t)λ\dfrac{3a^2}{8}t) + λ\dfrac{a^3}{32}(\cos(3t)-\cos(t)) \\ &= a\bigl(1-λ\dfrac{a^2}{32}\bigr)\cos\left(\bigl(1+λ\dfrac{3a^2}{8}\bigr)t\right)+ λ\dfrac{a^3}{32}\cos(3t)+O(λ^2) \end{align} capturing the perturbation of the frequency.
That this is sensible shows the following diagram. The blue line is the numerical solution for $a=2$, $λ=0.1$, which has to be considered as the most exact one, the green graph represents the first order perturbation solution without and the red graph with the frequency modification. As one sees, leaving the $t\sin t$ term free gives a rapidly increasing error.
Note that you can get rid of $ω$ in a "legal" way by considering $x(t)=y(ωt)$, since then $x''(t)=ω^2y''(ωt)$ and the transformed equation is $$ y''(s)+y(s)+\frac{λ}{ω^2}y(s)^3=0 $$