Let $a$ and $b$ be positive reals. Prove $$\frac{1}{3} > \frac{2ab^2}{(a + \sqrt{a^2 + b^2})^3}\;.$$
I am a high school student preparing for math contests, and this question was sent to me by one of my classmates on a contest math training course, but I wasn't able to solve it. I do not know where he got this problem from. I tried using AM-GM, since $2ab \leq 2a^2 + b^2$, but I couldn't progress further. Also, I also tried to manipulate the expression to use the HM-AM-GM-QM inequality, and I got to this:
$\sqrt{\dfrac{a^2 + b^2}{2}} > \left(\dfrac{b\sqrt{3}}{a + \sqrt{a^2 + b^2}} - 1\right)\left(\dfrac{b\sqrt{3}}{a + \sqrt{a^2 + b^2}} + \dfrac{a\sqrt{2}}{2}\right)$
Which didn't help much (and I am not even sure if this is correct). After these attempts, I got stuck. If anyone could give a hint, it would be very appreciated. Full solutions are also fine, but I would like to try to work out with the hints first.
Dividing numerator and denominator by $b^3,$ and letting $u=\frac ab,$ then this becomes:
$$\frac13>\frac{2u}{(u+\sqrt{u^2+1})^3}$$ for $u>0.$
This can be written as: $$(u+\sqrt{1+u^2})^3\geq 6u.$$
But $$(u+\sqrt{1+u^2})^3=u^3+3u^2\sqrt{1+u^2}+3u(1+u^2)+(1+u^2)^{3/2}$$
The third term contributes $3u,$ and via the second and fourth term, we get, via AM/GM, $3u^2(1+u^2)^{1/2}+(1+u^2)^{3/2}\geq 2\sqrt 3 u(1+u^2)>3u.$
So, instead of $6,$ we actually have the stronger $$(u+\sqrt{u^2+1})^3>(3+2\sqrt3)u.$$
Another approach uses $\frac1{u+\sqrt{u^2+1}}=\sqrt{u^2+1}-u.$ So if $w=\sqrt{u^2+1}-u,$ then $0<w<1$ and $2u=\frac1w-w,$ so we want:
$$\frac13>(w^{-1}-w)w^3=w^2-w^4$$
The maximum of the right side is when $2w=4w^3,$ or $w=\frac1{\sqrt 2}.$ (we can obviously exclude $w=0.$) So the maximum of $w^2-w^4$ is $\frac14.$
Since $2u=w^{-1}-w,$ the maximum occurs when $u=\frac{\sqrt2}4.$