I don't know how I could solve this problem.
- $[s^2 X(s)-sX(0)-X'(0)]+3X(s)+2[sX(s)-X(0)]=0$
- $[ s^2 X(s)-as-b ]+3X(s)+2[sX(s)-a]=0$
- $X(s)(s^2+3+2s)-as-b-2a=0$
And I finally found this: $X(s)=(2a+as+b)/(s^2+2s+3)$, but I don't know what to do next. Could someone help me with this problem?
What you have looks like a Laplace transform, which is one way to solve the problem. You can use partial fractions to split that up and take the inverse Laplace transform of each piece to get your answer.
Alternatively, since it is a linear homogeneous ODE with constant coefficients, you can put in the trial solution $y = e^{rx}$ to get the characteristic equation, solve for the values of $r$, write the general form of the solution, and then plug in your initial conditions to find the values of the constants.