I broke up the non homogeneous part $\cos^2(x)$ into $\frac12 -\frac12[\cos(2x)]$
and I set up my trial solution as $$ \begin{split} Y_p &= A +B\sin(2x)+C\cos(2x)\\ Y'_p &= 2B\cos(2x)-2C\sin(2x)\\ Y''_p &= -4B\sin(2x)-4C\cos(2x) \end{split} $$ and after substituting these values in the ODE I don't get anywhere. I compare $8B\cos(2x)-4C\cos(2x)$ with $-0.5\cos(2x)$ and I get $C=(16B+1)/8$ and I think I've made a mistake here as I don't know what to do next and the other solutions didn't have anything like this. And can the annihilator method be more suited for this type of question or is the undetermined coeffs. method fine for this?
The equation has only derivatives so your constant A will disapear. $$y"+ 4y'= cos^2( x).$$ $$r^2+4r=0 \implies r=0,-4 \implies y_h=k_1+k_2e^{-4x}$$ $$y"+ 4y'= \frac {\cos(2x)}2+\frac 12$$ For the constant $( \frac 12)$ use $y_{1p}=Ax \implies A=\frac 18 \implies y_{1p}=\frac x8$
( for the trig function you are correct $y_{2p}=c_1\cos(2x)+c_2\sin(2x)$ $$(8c_2-4c_1)\cos(2x)+(-8c_1-4c_2)\sin(2x)=\frac {\cos(2x)}2$$ $$ \begin{cases} 8c_2-4c_1=\frac 12 \\ 8c_1+4c_2=0 \\ \end{cases} \implies (c_1,c_2)=(-1/40,1/20) $$ Finally, $$\boxed {y(x)=k_1+k_2e^{-4x}+\frac x8-\frac 1 {40} \cos(2x)+\frac 1 {20}\sin(2x)}$$