The problem is as follows.
Consider the following linear second-order non-homogeneous ordinary differential equation with constant coefficients $$\frac{d^2y}{dx^2}+y=x\cdot\sin(x),\quad x>0.$$ Find the homogeneous solution and the general solution by any method.
The method of unknown coefficients fails me (assuming $y_p(x)=(Ax+B)\sin(x)+(Cx+D)\cos(x)$). I can solve it using LaPlace transforms, but it's very time consuming. The homogeneous solution is easy ($y_h(x)=c_1\sin(x)+c_2\cos(x)$), but the particular solution is trickier.
What is the preferred method to solve this?
There's a lot of unnecessary guessing in the other answers, so I'll tell you how to exactly pinpoint the correct particular form.
The first thing you should do is try to solve the homogeneous equation, which has the characteristic polynomial $$ r^2 + 1 = 0 $$
whose roots are $r = \pm i$. This gives the natural solution $y_h(x) = c_1 \cos x + c_2 \sin x$
The next step is to classify the inhomogeneous function on the RHS. Note that it contains a factor of $\sin x$, which already solves the homogeneous solution. This is why your original guess fails, because $$ (Ax+B)\sin x + (Cx+D)\cos x = Ax\sin x + Cx\cos x + B\sin x + D\cos x $$
See the last two terms? Those will cancel out when you plug them into the original equation, leaving only the two constants $A$ and $C$. This is troublesome because you don't have enough constants to solve the problem. As you've already seen, the LHS (after plugging in) will not contain any terms proportional to $x\sin x$. This makes the problem unsolvable.
To amend the particular solution, the quick and easy way is to tack on another factor of $x$ throughout the entire expression $$ y_p = (Ax+B)x\sin x + (Cx+D)x\cos x $$
Now this expression will lead to a solution. Note that none of these constants can be left out, as the factor of $x$ in $x\sin x$ requires the particular form to contain the factors $\alpha x + \beta$, in the same way that any factor of $x^n$ in the inhomogeneous function requires the full polynomial of degree $n$
$$ c_0 + c_1 x + c_2 x + \dots + c_nx^n $$
The reason being that the lower powers will appear in the derivative expression, even if the original equation doesn't contain them (same reason why you also need a $\cos x$ in there).
The other answers give particular forms $$ y_h = Ax\sin x + Bx\cos x $$ $$ y_h = (Ax^2 + Bx + C)\sin x + (Dx^2 + Ex + F)\sin x $$ The first of which is identical to your intial guess, which doesn't have enough constants. The second has too many constants (two of which belong to the homogeneous already). These examples give you a feel for how many constants is "just right"
Additional note: The multiplicity of the homogeneous part is also important. If your homogeneous solution is instead something like $$ y_h = c_1\sin x + c_2x\sin x $$
which contains two (linearly independent) factors of $\sin x$. You would need another factor of $x$ to round it out. Then the particular has the form $$ y_p = (Ax+B)x^2\sin x + (Cx+D)x^2\cos x $$
Some people might tell you to gradually tack on one factor of $x$ until you get something that works, but this is extraneous. The power of $x$ you need to pultiply with the particular form is exactly equal to the multiplicity of the root. Knowing this will save you time from having to take derivatives and plugging in over and over again.