I'm having trouble proving the following reduction formula:
If
$$I_{m,n}=\int\frac{x^m}{(ax^2+bx+c)^n}dx$$
then
$$\int\frac{x^m}{(ax^2+bx+c)^n}dx=-\frac{x^{m-1}}{a(2n-m-1)(ax^2+bx+c)^{n-1}}-\frac{b(n-m)}{a(2n-m-1)}I_{m-1,n}+\frac{c(m-1)}{a(2n-m-1)}I_{m-2,n}$$
My attempt went as follows:
$$\int\frac{x^m}{(ax^2+bx+c)^n}dx\space\begin{vmatrix}u=\frac{1}{(ax^2+bx+c)^n}\\du=\frac{-n(2ax+b)}{(ax^2+bx+c)^{n+1}}dx\end{vmatrix}\space dv=x^mdx\quad v=\frac{1}{m+1}x^{m+1}\\\int\frac{x^m}{(ax^2+bx+c)^n}dx=\frac{x^{m+1}}{(m+1)(ax^2+bx+c)^n}+\bigg(\frac{n}{m+1}\bigg)\int\frac{(2ax+b)x^{m+1}}{(ax^2+bx+c)^{n+1}}dx\\=\frac{x^{m+1}}{(m+1)(ax^2+bx+c)^n}+\bigg(\frac{n}{m+1}\bigg)\bigg(2a\int\frac{x^{m+2}}{(ax^2+bx+c)^{n+1}}dx\\+b\int\frac{x^{m+1}}{(ax^2+bx+c)^{n+1}}dx\bigg)\\=\frac{x^{m+1}}{(m+1)(ax^2+bx+c)^n}+\bigg(\frac{2an}{m+1}\bigg)\int\frac{x^{m+2}}{(ax^2+bx+c)^{n+1}}dx\\+\bigg(\frac{bn}{m+1}\bigg)\int\frac{x^{m+1}}{(ax^2+bx+c)^{n+1}}dx\\=\frac{x^{m+1}}{(m+1)(ax^2+bx+c)^n}+\bigg(\frac{2an}{m+1}\bigg)I_{m+2,n+1}+\bigg(\frac{bn}{m+1}\bigg)I_{m+1,n+1}$$
This is where I'm stuck, any help?
So this means you are using the integration by parts in the wrong direction or with a wrong choice. In the reduction formula you are trying to achieve, you can notice that $m$ is decreasing while $n$ stays constant. Thus, you need a choice of $u,v$ such that: $$udv = \dfrac{x^mdx}{f^n(x)},$$ where $f(x) = ax^2+bx+c$ and: $$vdu = C\dfrac{x^{m-1}dx}{f^n(x)},$$ with $C$ some constant. If you tried to naively take $u = x^m,$ then you will have $v = \int\dfrac{dx}{f^n(x)},$ which you cannot evaluate. Therefore, one needs to improvise somehow. For instance, $$d\left(\dfrac{1}{g^k(x)}\right) = \dfrac{kg'(x)g^{k-1}(x)dx}{g^{2k}(x)} = k\dfrac{g'(x)dx}{g^{k+1}(x)}$$ for appropriately smooth function $g(x).$ This is where I will leave you with a hint: $$\dfrac{x^mdx}{f^n(x)} = \dfrac{x^{m-1}}{2a}\cdot\dfrac{d(ax^2+bx+c)}{(ax^2+bx+c)^n} -\dfrac{b}{2a}\cdot\dfrac{x^{m-1}}{(ax^2+bx+c)^n}=\dots $$