I solve integral considerable number, but the following integral I could not solve. It is about this integral:
$$ \int\frac{\ln^2(a+bx)dx}{x^n} $$
I want to make the solution through partial method: $\int udv=uv-\int vdu$
I've solved the integral $$\int\frac{\ln(a+bx)dx}{x^n}$$.
Please help, thank you for your help preliminarily
This is just an (failed) attempt to find a reduction formula, and as @mtiano noted, the close form requires some hypergeometric functions. Let $$\begin{align} u =& \frac{\ln(a+bx)}{x^n} \\ du =& \frac{\frac{bx^n}{a+bx}-n x^{n-1}\ln(a+bx)}{x^{2n}}dx = \left[ \frac{b}{x^n(a+bx)}-\frac{n\ln(a+bx)}{x^{n+1}}\right]dx\\ dv =& \ln(a+bx) dx \\ v =& \frac{(a+bx)\ln(a+bx)}{b}-x \end{align}$$
Also let (since OP has solved it) $$I_n(x)=\int\frac{\ln(a+bx)dx}{x^n}$$
$$\begin{align} J_n(x)=&\int\frac{\ln^2(a+bx)dx}{x^n}\\ =& \int\frac{\ln(a+bx)}{x^n}\ln(a+bx)dx\\ =& \frac{\ln(a+bx)}{x^n} \cdot \left[ \frac{(a+bx)\ln(a+bx)}{b}-x \right]- \\ & \int{\left[ \frac{(a+bx)\ln(a+bx)}{b}-x\right] \left[\frac{b}{x^n(a+bx)}-\frac{n\ln(a+bx)}{x^{n+1}}\right]dx}\\ =& \frac{(a+bx)\ln^2(a+bx)}{bx^n}-\frac{\ln(a+bx)}{x^{n-1}} -\\ & \int \left[ \frac{\ln(a+bx)}{x^n} - \frac{b}{x^{n-1}(a+bx)}- \frac{n(a+bx)\ln^2(a+bx)}{bx^{n+1}} +\frac{n\ln(a+bx)}{x^{n}}\right]dx \\ =& \frac{(a+bx)\ln^2(a+bx)}{bx^n}-\frac{\ln(a+bx)}{x^{n-1}} - (n+1) I_n(x) +\frac{na}{b} J_{n+1}(x) + nJ_n(x) +\\ & \int\frac{b\,dx}{x^{n-1}(a+bx)}\\ =& \frac{(a+bx)\ln^2(a+bx)}{bx^n}-\frac{\ln(a+bx)}{x^{n-1}} - (n+1) I_n(x) +\frac{na}{b} J_{n+1}(x) + nJ_n(x) +\\ & \left(-\frac{b}{a}\right)^{n-1} \int \frac{b\,dx}{a+bx} - \sum_{k=1}^{n-1} \left(-\frac{b}{a}\right)^{n-k}\int\frac{dx}{x^k} &\text{(1)}\\ =& \frac{(a+bx)\ln^2(a+bx)}{bx^n}-\frac{\ln(a+bx)}{x^{n-1}} - (n+1) I_n(x) +\frac{na}{b} J_{n+1}(x) + nJ_n(x) +\\ & \left(-\frac{b}{a}\right)^{n-1} \ln(a+bx) -\left(-\frac{b}{a}\right)^{n-1} \ln x - \sum_{k=2}^{n-1} \left(-\frac{b}{a}\right)^{n-k}\frac{1}{(-k+1)x^{k-1}} &\text{(2)}\\ \end{align}$$ Notice, for line $\text{(1)}$, the final summation term exists only for $n \ge 2$. For line $\text{(2)}$, the second last term (involving $\ln x$) exists only for $n \ge 2$, and the final summation term exists only for $n \ge 3$.
Making $J_{n+1}(x)$ the main term, $$\begin{align} -\frac{na}{b} J_{n+1}(x)=& \frac{(a+bx)\ln^2(a+bx)}{bx^n}-\frac{\ln(a+bx)}{x^{n-1}} - (n+1) I_n(x) + (n-1)J_n(x) +\\ & \left(-\frac{b}{a}\right)^{n-1} \ln(a+bx) -\left(-\frac{b}{a}\right)^{n-1} \ln x - \sum_{k=2}^{n-1} \left(-\frac{b}{a}\right)^{n-k}\frac{1}{(-k+1)x^{k-1}}\\ J_{n+1}(x)=& -\frac{(a+bx)\ln^2(a+bx)}{nax^n}+\frac{b\ln(a+bx)}{nax^{n-1}} + \frac{b(n+1)}{na} I_n(x) - \frac{b(n-1)}{na}J_n(x) +\\ & \left(-\frac{b}{a}\right)^{n} \frac{\ln(a+bx)}{n} -\left(-\frac{b}{a}\right)^{n} \frac{\ln x}{n} - \frac{1}{n}\sum_{k=2}^{n-1} \left(-\frac{b}{a}\right)^{n-k+1}\frac{1}{(-k+1)x^{k-1}}\\ \end{align}$$
This gives a reduction formula for $J_{n+1}(x)$ for $n = 1, 2, 3, \ldots$. Same as above, the second last term (involving $\ln x$) exists only for $n \ge 2$, and the final summation term exists only for $n \ge 3$. Lastly, I could not solve the base case $$J_1(x) = \int \frac{\ln^2 (a + bx)dx}{x}$$ yet, and reading from WolframAlpha the solution requires polylogarithm function.