For which values of $p$ does integral $I$ converge? $$I=\int_0^{\infty}\frac{dx}{(x^2+\sqrt{x})^p}$$
My shot in the dark was $p\in(\frac{1}{2},2)$, since $\int_0^{1}\frac{dx}{x^{p/2}}<\infty \iff p/2<1$ and $\int_1^{\infty}\frac{dx}{x^{2p}}<\infty \iff 2p>1$. Sadly, I'm unable to prove that $I=\infty$ for $p<1/2$ or $p>2$ and no other way comes to my mind. But it doesn't look complicated at all, so there must be a simple soultion.
On
$$\int_0^{\infty}\frac{dx}{(x^2+\sqrt{x})^p} = \frac{2 \Gamma(\frac{2-p}{3} )\Gamma(\frac{4p-2}{3})}{3\Gamma(p)}$$
This is the 'answer' supplied by Mathematica. It is clear that this only makes sense when $1/2 < p < 2$ as you've said. In fact Mathematica retuns this as conditional upon $p$ satisfying that inequality.
To actually prove this yourself you will need to use contour integration or some other advanced technique. For example you can see that the poles of the function lie at $x = 0$ and $x = -(1/2) + i \sqrt{3}/2$. So you can use the contour that (has no poles in it) and has the arcs: $x \in [-R, -\rho]$, $x \in [\rho, R]$, $x = \rho e^{i \phi}$ with $\phi \in [-\pi,0]$ and $x= R e^{i \theta}$ with $\theta \in [0,-\pi]$.
As $R\rightarrow \infty$ and $\rho \rightarrow 0$, It is not hard to show that the outer semicircle has no contribution, The "straight" one is the original integral. The small semicircle is possible to integrate, but it is no picnic. Some may consider this a long comment because I won't do it for you but I really don't see the point. The value comes out as above. The Gamma function diverges when its argument is a negative integer or zero. This bounds the value of $p$ as stated.
But it seems to me that you already did. You employed the comparison test, by forcefully factoring either $x^2$ or $\sqrt x=x^{^\frac12}$ outside the parentheses, then used the fact that the remaining term tends to $1$ as x tends to either $0$ or infinity, respectively. Either way, it's bound between $1+0=1$ and $1+1$ $=2$, assuming the split point is $1$.
As far as the actual evaluation is concerned, all integrals of the form $\displaystyle\int_0^\infty\frac{x^{^{k-1}}}{(a^{^n}+x^{^n})^{^m}}dx$ can be solved by letting $x=a~t$, and $u=\dfrac1{1+t^{^n}}$, then recognizing the expression of the beta function in the resulting integral, after which all that's left to do is employing Euler's reflection formula for the $\Gamma$ function in order to express it in terms of ordinary trigonometric functions. In this case, $k-1=$ $=-\dfrac p2,~n=\dfrac32,~m=p,~$ and $~a=1$.