I want to evaluate this integral:
$$\int{\frac{ax+b}{(x^2+2px+q)^n}}dx$$
The book only says to integrate by parts $\int{\dfrac{1}{(x^2+2px+q)^{n-1}}dx}$, for simplicity if $n = 2$ I get: $\int{\dfrac{1}{(x^2+2px+q)}dx}=\dfrac{x}{x^2+2px+q}+\int{\dfrac{2x^2+2px}{(x^2+2px+q)^2}}dx$.
Now I don't know what to do.
On
1.for the integrand $\dfrac{1}{x^2+2px+q}$ complete the square to get
$\frac{1}{\dfrac{1}{4}(4q-4p)^2+(p+x)^2}$
2.then substitute $u=p+x$
3.factor out $\dfrac{1}{4}(4q-4p)^2$
The integral reduced to simple trigonometric function
On
General description. A partial fraction of the form $$\dfrac{Bx+C}{\left[ \left( x-r\right) ^{2}+s^{2}\right] ^{n}}$$ is integrable by substitution, using the change of variables $x=r+st$. With this method an integral of the form
$$\displaystyle\int \frac{Bx+C}{\left[\left( x-r\right) ^{2}+s^{2}\right] ^{n}}dx\tag{1}$$
is transformed into an integral of the form
$$\displaystyle\int \frac{Dt+E}{(t^{2}+1)^{n}}dt=D\int \frac{t}{(t^{2}+1)^{n}} dt+E\int \frac{1}{(t^{2}+1)^{n}}dt\tag{2}.$$
The first integral is a table integral. The second one satisfies a recurrence relation.
Application to the given integral. Rewrite the integrand as $\dfrac{ax+b}{\left[(x+p)^{2}+q-p^{2}\right]^{n}}$ and make the substitution $$ \begin{equation*} x=-p+\sqrt{q-p^{2}}\, t.\tag{3} \end{equation*} $$ Then $$ \begin{eqnarray*} I &=&\int \frac{ax+b}{(x^{2}+2px+q)^{n}}dx=\int \frac{ax+b}{ \left[(x+p)^{2}+q-p^{2}\right]^{n}}dx, \tag{4}\\ && \\ \\ M &=&a\sqrt{q-p^{2}},\quad N=b-ap,\quad K=\frac{\sqrt{q-p^{2}}}{\left( q-p^{2}\right) ^{n}} \\ \\ && \\ I &=&K\int \frac{M}{(t^{2}+1)^{n}}t+\frac{N}{(t^{2}+1)^{n}}dt, \\ &=&KM\int \frac{1}{(t^{2}+1)^{n}}tdt+KN\int \frac{1}{(t^{2}+1)^{n}}dt \\ &=&-\frac{KM}{2(n-1)}\frac{1}{(t^{2}+1)^{n-1}}+KN\int \frac{1}{(t^{2}+1)^{n}} dt.\tag{5} \end{eqnarray*} $$ The last integral can be split into two, the latter being integrable by parts which generates the recursive relation $(6)$ bellow. For $n\ne 1$, we have: $$ \begin{eqnarray*}I_n&=& \int \frac{1}{(t^{2}+1)^{n}}dt =\int \frac{1+t^{2}-t^{2}}{(t^{2}+1)^{n}} dt\\ &=&\int \frac{1}{(t^{2}+1)^{n-1}}dt-\int \frac{t^{2}}{(t^{2}+1)^{n}}dt \\ &=&\int \frac{1}{(t^{2}+1)^{n-1}}dt+\frac{t}{2(n-1)\left( t^{2}+1\right) ^{n-1}} \\ &\qquad - &\frac{1}{2(n-1)}\int \frac{1}{\left( t^{2}+1\right) ^{n-1}}\,dt \\ &=&\frac{t}{2(n-1)\left( t^{2}+1\right) ^{n-1}}+\frac{2n-3}{2(n-1)}\int \frac{1}{\left( t^{2}+1\right) ^{n-1}}\,dt\\ &=&\frac{t}{2(n-1)\left( t^{2}+1\right) ^{n-1}}+\frac{2n-3}{2(n-1)}I_{n-1}.\tag{6} \end{eqnarray*} $$
For $n=1$ $$ I_1=\int \frac{1}{t^{2}+1}dt=\arctan t+\text{Constant}.\tag{7} $$
ADDED. For the particular case $n=2$, we get from $(6),(7)$ $$ \begin{eqnarray*} I_2=\int \frac{1}{(t^{2}+1)^{2}}dt &=&\frac{1}{2}\frac{t}{t^{2}+1}+\frac{1}{2}I_1 \\ &=&\frac{1}{2}\frac{t}{t^{2}+1}+\frac{1}{2}\arctan t+\text{Constant}.\tag{6$\mathrm{a}$} \end{eqnarray*} $$
and from $(5)$
$$ \begin{eqnarray*} I &=&\int \frac{ax+b}{(x^{2}+2px+q)^{2}}dx \\ &=&-\frac{a}{2}\frac{1}{x^{2}+2px+q}+\frac{b-ap}{2\left( q-p^{2}\right) } \frac{x+p}{q+x^{2}+2xp}\\ &+& \frac{b-ap}{2\left( q-p^{2}\right) ^{3/2}}\arctan \frac{x+p}{\sqrt{q-p^{2}}}+\text{Constant}. \end{eqnarray*}\tag{5$\mathrm{a}$} $$
Setting $a=0,b=1$ in $(5\mathrm{a})$ we evaluate your last integral for $q>p^2$:
$$ \boxed{\begin{eqnarray*} \int \frac{1}{(x^{2}+2px+q)^{2}}dx&=&\frac{1}{2\left( q-p^{2}\right) }\frac{x+p}{x^{2}+2px+q}\\&+&\frac{1}{2\left( q-p^{2}\right) ^{3/2}}\arctan \frac{x+p}{\sqrt{q-p^{2}}}+\text{Constant}. \end{eqnarray*}} $$ $$\tag{5$\mathrm{b}$}$$
To answer your specific question, here is a trick that will work. Note that $$2x^2+2px=(2x^2+4px+2q)-2px-2q=(2x^2+4px +2q)-p(2x+2p)-2q+2p^2.$$ It follows that $$\int \frac{dx}{x^2+2px+q}=\frac{x}{x^2+2px+q} +2\int \frac{x^2+2px+q}{(x^2+2px+q)^2}\,dx -p\int \frac{2x+2p}{(x^2+2px+q)^2}\,dx+ (2p^2-2q)\int \frac{dx}{(x^2+2px+q)^2}.$$ On the right, the first integral is just the integral on the left. The second integral yields to the substitution $u=x^2+2px+q$. And the third integral is the one we wanted to evaluate. So we can express $\int \frac{dx}{(x^2+2px+q)^2}$ in terms of $\int \frac{dx}{x^2+px+q}$ except in the case $p^2=q$. But in that case $(x^2+2px+q=(x+p)^2$, and the integration is easy without going through the integration by parts.
It would have been easier to make the immediate substitution $x+p=y$, but I wanted to carry on from the point you had reached.
What follows is the beginning of much longer answer I had typed. The reduction formula got unpleasant to work with, so below is only the beginning of that answer.
We have $ax+b=\frac{a}{2}(2x+2p)+ b-ap$. So our integral is $$\frac{a}{2}\int \frac{2x+2p}{(x^2+2px+q)^n}\,dx+ (b-ap)\int \frac{dx}{(x^2+2px+q)^n}.\tag{$1$}$$ The first integral in $(1)$ is found by making the substitution $u=x^2+2px+q$. So we are integrating a power of $u$, easy.
The second integral in $(1)$ is not so easy! We use, repeatedly if necessary, a Reduction Formula (they are discussed in great detail in Wikipedia).
For any positive integer $n$, let $$I_n=\int \frac{dx}{(x^2+2x+q)^n}.$$ We show how to find a general expression for $I_n$ in terms of $I_{n-1}$. Thus if we start with for example $n=3$, we express $I_3$ in terms of $I_2$, then $I_2$ in terms of $I_1$.
Finally, $I_1$ breaks up into cases. Complete the square to get $x^2+2px+q=(x+p)^2+q-p^2$. If $q-p^2=0$, we had an easy integral to begin with, and the Reduction Formula was a waste of time. If $q-p^2\gt 0$, a substitution brings us to $\int \frac{1}{u^2+1}$. If $q-p^2\lt 0$, use partial fractions.
Now we begin the reduction. As in what you did, we use integration by parts. It is easier for me to go backwards. So we calculate $I_{n-1}$ using integration by parts. Let $n\ge 2$. We have $$I_{n-1}=\int \frac{dx}{(x^2+2px+q)^{n-1}}.$$ Let $u=\frac{1}{(x^2+2px+q)^{n-1}}$ and let $dv=dx$. Then $du=-\frac{(n-1)(2x+2p)}{(x^2+2px+q)^n}$ and we can take $v=x$. So $$I_{n-1}= \frac{x}{(x^2+2px+q)^{n-1}}+(n-1)\int \frac{2x^2+2px}{(x^2+2px+q)^{n}}\,dx.$$ To finish, use the same trick as the one at the beginning of the post. However, you will find the calculation far more pleasant with the preliminary change of variable $y=x+p$.