Consider this improper integral of first kind: $$\int_0^{+\infty}{\frac{t\ln t}{{(t^2+1)}^{\alpha}}}\,{dt}, \quad \alpha\in\mathbb R$$
Its required to find the nature of this improper integral. We know that the problem of this integral is only at $+\infty$ so we must study it in that neighborhood.
Ok so $t^2+1 \equiv t^2$ thus $(f(t)=\frac{t\ln t}{{(t^2+1)}^{\alpha}}) \equiv t^{1-2\alpha}\ln t$ and from here i tried to make the quotient test but didn't have a result, i had the problem even more complicated.(note:when using the quotient test we take the absolute value of $f$ because the functions must be positive)
How should I proceed from here?
If $a \le 1$, then your estimates give that the integrand is greater than
$$t^{1 - 2\alpha} \ge t^{-1}$$
and the integral is divergent.
If $a > 1$, then estimate the logarithm away with a very small power of $t$ - for example, adjusting the constants carefully yields for any $\beta > 0$ the estimate
$$\ln t < C t^{\beta}$$
where $C$ depends on $\beta$. Then
$$t^{1 - 2\alpha} \ln t < C t^{1 - 2\alpha + \beta}$$
Now if $\beta$ is sufficiently small, the overall exponent is less than $-1$, and we have convergence.