Let $a$ and $b$ be positive parameters.
By integrating the complex function $$ f(z) = \exp \left( iaz^{2} - \frac{ib}{z^{2}} \right)$$ around a wedge in the first quadrant that makes an angle of $\frac{\pi}{4}$ with the positive real axis and is indented at the origin, I get
$$ \begin{align} \int_{0}^{\infty} \sin \left(ax^{2}-\frac{b}{x^{2}} \right) \, dx &= \text{Im} \int_{0}^{\infty} f(x) \, dx \\ &= \text{Im} \int_{0}^{\infty} f(te^{i \pi /4}) \ e^{i \pi /4} \, dt \\ &= \frac{1}{\sqrt{2}} \int_{0}^{\infty} \exp \left( -at^{2} - \frac{b}{t^{2}} \right) \, dt \\ &= \frac{1}{2} \sqrt{\frac{\pi}{2a}} e^{-2 \sqrt{ab}}.\end{align}$$
But when I ask Maple to evaluate the integral numerically for specific values of the parameters, the result disagrees with my result.
Is my answer incorrect, or is it that the highly oscillatory nature of the integrand makes the integral hard to evaluate numerically?
Maple agrees with your symbolic value. Can you provide us with a numerical case of disagreement?