I would like to show that $f(x)=\sin^2 x-x^2\cos x, \forall x\in [0,\frac{\pi}{2}]$ is monotonic increasing.
If we can show that $f'(x)>0$, $\forall x\in [0,\frac{\pi}{2}]$, then $f(x)$ is increasing there.
We have
$f'(x)=\sin 2x-2x\cos x+x^2\sin x$.
How can I show that $f'(x)>0, \forall x\in [0,\frac{\pi}{2}]$
One has \begin{align}2\sin x\cos x-2x\cos x+x^2\sin x &=2\cos x(\sin x-x)+x^2\sin x \\ &\ge x^2\sin x-\frac{x^3\cos x}{3} \\&=x^2\cos x\left(\tan x-\frac{x}3\right)\\&\ge \frac23x^3\cos x\ge0,\end{align}where we have used the Taylor expansion of $\sin x$ and the fact that $\tan x\ge x$ on $\left[0,\dfrac\pi2\right]$.