The question starts by asking to show that $e^{iaz^2} \xrightarrow{} 0$ and is analytic as $|z| \xrightarrow{} \infty$, for $0 < \arg{z} \le \frac{\pi}{4}$, which I have done. NB $a$ is a real, positive constant.
It then continues as follows:
By applying Cauchy's theorem to a suitable contour prove that: $$\int^\infty_0\cos(ax^2) dx = \sqrt{\frac{\pi}{8a}}$$
Since $z^2=x^2-y^2+i2xy$, it is clear that the question wants the above equation proven by integrating $e^{iaz^2}$ along some contour $C$ and using Cauchy's theorem, which will hold within the argument range given above (because the function is analytic in this range). However, I don't know how to select an appropriate contour. If $0 \le \arg(z)$ then it would be a lot easier, but because $0<\arg(z)$ the contour has to be a lot more awakward (because we cannot use the $x$ axis).
I have attempted to use the following contour:
Reason being that the contour segment $C_2$ is simple and $C_\infty$ won't contribute anything to the integral because the function is 0 at infinity.
If we now apply Cauchy's theorem:
$$\oint_C e^{iaz^2}dz=\int_{C_1} e^{iaz^2}dz+\int_{C_2} e^{iaz^2}dz=0$$
$$\int_{C_1} e^{ia(x^2-y^2)-2axy}(dx+idy)+\int_{C_2} e^{ia(x^2-y^2)-2axy}(dx+idy)=0$$
$$\int_{C_1} e^{ia\frac{3x^2}{4}-ax^2}(dx+i\frac{dx}{2})+\int_{C_2} e^{i2ax^2-2ax^2}(dx+idx)=0$$
$$\int^\infty_0(1+\frac{i}{2})e^{ax^2(\frac{3i}{4}-1)}dx+\int^0_\infty(1+i)e^{2ax^2(i-1)}dx=0$$
I have no idea where to go from here. It is clear that I must use De Moivre's theorem, but the resulting integral would be hideous, which makes me doubt my contour. Either my understanding is wrong, my contour is poorly chosen or the question just wants me to wade through pages of tedious algebra. I would appreciate any guidance you could offer.
I would still "use the $x$ axis".
Replacing your $C_{\infty}$ with $C_r$ the arc of $|z|=r$, we have \begin{align} \left|\int_{C_r}e^{iaz^2}dz\right|&=\left|\int_{0}^{\pi/4}\exp(iar^2e^{i\phi})ire^{i\phi}\,d\phi\right|&&\text{[substitution $z=re^{i\phi}$]}\\ &\leqslant r\int_{0}^{\pi/4}\exp(-ar^2\sin\phi)\,d\phi&&\text{[taking $|\cdot|$ of the integrand]}\\ &\leqslant r\int_0^{\pi/4}\exp(-2ar^2\phi/\pi)\,d\phi&&\text{[using $\sin\phi\geqslant 2\phi/\pi$]}\\ &=\frac{\pi}{2ar}(1-e^{-ar^2/2})\underset{r\to\infty}{\longrightarrow}0. \end{align} The rest is clear.