I'm trying to solve the general case of second order ODE: $y''(t) + p(t)y'(t) + q(t)y(t) = 0$.
where $p(t)$ and $q(t)$ are polynomials divides contains some symbolic constant and $t$,$t^2$,$t^3$ and $t^4$ elements.
I tried to use MATLAB dsolve but it cannot solve it.
is there a recommended book where I can find a way to solve this problem?
or a different program like Maple or Mathematica to solve it with a symbolic toolbox?
Thanks.
$$y''(t)+p(t)y'(t)+q(t)y(t)=0$$ Analytical solving of this general linear second order ODE is a much too wide question, even if $p(t)$ and $q(t)$ are not any kind of functions, but polynomial fractions.
For a general approach see : http://mathworld.wolfram.com/Second-OrderOrdinaryDifferentialEquation.html
If you don't want a closed form solution, you can try to find a solution on the form of infinite series.
Often the closed form solution requires some special functions which where defined and standardised especially to solve a particular kind of ODE.
For example in case of $p(t)=\frac{1}{t}$ and $q(t)=\frac{t^2-n^2}{t^2}$ the analytic solution is $$y(t)=c_1J_n(t)+c_2Y_n(t)$$ $J_n(t)$ and $Y_n(t)$ are the Bessel functions of first and second kind respectively.
More complicated example : Case of $p(t)=\frac{c-(a+b+1)t}{t^2-t}$ and $q(t)=\frac{ab}{t(t-1)}$ the analytic solution is $$y(t)=c_1\:\:_2F_1(a,b;c;t)+c_2\:(-t)^{1-c}\:_2F_1(a-c+1,b-c+1;2-c;t)$$ $\:_2F_1(a,b;c;t)$ is the Gauss hypergeometric function.
They are a lot of examples of such ODEs which solutions are expressed with convenient special functions. But in the general case of any $p(t)$ and $q(t)$ the convenient special functions where not always standardized.
In case of your ODE with $p(t)=\frac{t^2+t^+1}{t^3+2}$ and $q(t)=\frac{t^4}{t+2}$ as far as I know no convenient special function is available. May be a generalized hypergeometric function ? Sorry I have not enough available time to check it and I doubt the extra effort is worth it. As usual in such a situation, one commonly use numerical method for solving.