I've read over my working many times now and have been unable to resolve where my error lies. Can anyone please have a look and advise. Thank you!
Note, in the following, the Cosine and Sine integrals are defined as: \begin{equation} C(x) = \int_0^x \cos\left(t^2\right)\:dt, \qquad S(x) = \int_0^x \sin\left(t^2\right)\:dt \end{equation}
Here we will address the integral: \begin{equation} I\left(a,b\right) = \int_0^\infty \frac{1}{x}\sin\left(ax\right)\cos\left(bx^2\right)\:dx \end{equation} We first observe: \begin{equation} I\left(a,b\right) = \int_0^\infty \frac{1}{x}\sin\left(ax\right)\cos\left( \left(\sqrt{b}x\right)^2\right)\:dx \end{equation} Here let $u = \sqrt{b}x$ and so $dx = \frac{1}{\sqrt{b}}\:du$: \begin{equation} I\left(a,b\right) = \int_0^\infty \frac{1}{x}\sin\left(a\cdot \frac{u}{\sqrt{b}}\right)\cos\left(u^2\right) \frac{1}{\sqrt{b}}\:du = \frac{1}{\sqrt{b}} I\left(\frac{a}{\sqrt{b}},1 \right) \end{equation} Here let: \begin{equation} J\left(k\right) = \int_0^\infty \frac{1}{u}\sin\left(ku\right)\cos\left( u^2\right)\:du \end{equation} Then $I\left(a,b\right) =\frac{1}{\sqrt{b}} J\left(\frac{a}{\sqrt{b}}\right) $. We now proceed by solving $J\left(k\right)$. We first note that: \begin{equation} J\left(0\right) = \int_0^\infty \frac{1}{u}\sin\left(0\cdot u\right)\cos\left( u^2\right)\:du = 0 \end{equation} We now employ Leibniz's Integral Rule and differentiate under the curve with respect to $k$: \begin{equation} J'\left(k\right) = \frac{d}{d k} \int_0^\infty \frac{1}{u}\sin\left(ku\right)\cos\left(u^2\right)\:du = \int_0^\infty \frac{1}{u} \frac{\partial }{\partial k}\left[\sin\left(ku\right) \right]\cos\left(u^2\right)\:du = \int_0^\infty \cos\left(ku\right)\cos\left(u^2\right)\:du \end{equation} We now employ the Trigonometric Identity: \begin{equation} \cos\left(X\right)\cos\left(Y\right) = \frac{1}{2}\left[\cos\left(X+Y\right) + \cos\left(X - Y\right) \right] \end{equation} And so: \begin{align} J'\left(k\right) &= \int_0^\infty \frac{1}{2}\left[\cos\left(u^2 + ku\right) + \cos\left(u^2 - ku\right) \right]\:du = \frac{1}{2}\left[ \int_0^\infty \cos\left(u^2 + ku\right)\:du + \int_0^\infty \cos\left(u^2 - ku\right)\:du \right] \\ &= \frac{1}{2}\left[A\left(k\right) + B\left(k\right)\right] \end{align} For $A(k)$, we first note: \begin{equation} u^2 + ku = \left(u + \frac{k}{2}\right)^2 - \frac{k^2}{4} \end{equation} And so, \begin{equation} \cos\left( u^2 + ku\right) = \cos\left( \left(u + \frac{k}{2}\right)^2 - \frac{k^2}{4}\right) \end{equation} Here we employ the Trigonometric Identity: \begin{equation} \cos\left(X - Y\right) = \cos\left(X\right)\cos\left(Y\right) + \sin\left(X\right)\sin\left(Y\right) \end{equation} And so, \begin{equation} \cos\left( \left(u + \frac{k}{2}\right)^2 - \frac{k^2}{4}\right) = \cos\left( \left(u + \frac{k}{2}\right)^2 \right)\cos\left(\frac{k^2}{4}\right) +\sin\left( \left(u + \frac{k}{2}\right)^2 \right)\sin\left(\frac{k^2}{4}\right) \end{equation} Thus: \begin{align} A\left(k\right) &= \int_0^\infty \left[ \cos\left( \left(u + \frac{k}{2}\right)^2 \right)\cos\left(\frac{k^2}{4}\right) +\sin\left( \left(u + \frac{k}{2}\right)^2 \right)\sin\left(\frac{k^2}{4}\right)\right]\:du \\ &= \cos\left(\frac{k^2}{4}\right)\int_0^\infty\cos\left( \left(u + \frac{k}{2}\right)^2 \right)\:du +\sin\left(\frac{k^2}{4}\right)\int_0^\infty\sin\left( \left(u + \frac{k}{2}\right)^2 \right)\:du \end{align} For both Integrals let $t = u + \frac{k}{2}$ and so $dt = du$: \begin{align} A\left(k\right) &=\cos\left(\frac{k^2}{4}\right)\int_0^\infty\cos\left( \left(u + \frac{k}{2}\right)^2 \right)\:du +\sin\left(\frac{k^2}{4}\right)\int_0^\infty\sin\left( \left(u + \frac{k}{2}\right)^2 \right)\:du \\ &= \cos\left(\frac{k^2}{4}\right)\int_{\frac{k}{2}}^\infty\cos\left( t^2 \right)\:dt +\sin\left(\frac{k^2}{4}\right)\int_{\frac{k}{2}}^\infty\sin\left( t^2 \right)\:dt \\ &= \cos\left(\frac{k^2}{4}\right)\left[\int_{0}^\infty\cos\left( t^2 \right)\:dt - \int_0^{\frac{k}{2}}\cos\left( t^2 \right)\:dt\right] + \sin\left(\frac{k^2}{4}\right)\left[ \int_{0}^\infty\sin\left( t^2 \right)\:dt - \int_0^{\frac{k}{2}}\sin\left( t^2 \right)\:dt\right] \\ &= \cos\left(\frac{k^2}{4}\right)\left[\int_{0}^\infty\cos\left( t^2 \right)\:dt - C\left(\frac{k}{2}\right)\right] + \sin\left(\frac{k^2}{4}\right)\left[ \int_{0}^\infty\sin\left( t^2 \right)\:dt - S\left(\frac{k}{2}\right)\right] \end{align} Where $C(x), S(x)$ are the Fresnal Cosine and Sine Integrals respectively. For the remaining integral with cosine integrand we rely on the result from Section \ref{int_cos_x^n_0_to_inf}: \begin{equation} P(n) = \int_0^\infty \cos\left(x^n\right)\:dx = \frac{1}{2n}\left[\frac{\Gamma\left(1 - \frac{1}{2n}\right)\Gamma\left(\frac{1}{2n}\right)}{\Gamma\left(1 - \frac{1}{n}\right)} \right] \end{equation} For convergence we require that $n > 1$. Thus, \begin{equation} \int_{0}^\infty\cos\left( t^2 \right)\:dt = P(2) = \frac{1}{2\cdot 2}\left[\frac{\Gamma\left(1 - \frac{1}{2\cdot 2}\right)\Gamma\left(\frac{1}{2\cdot 2}\right)}{\Gamma\left(1 - \frac{1}{2}\right)}\right] = \frac{1}{4}\left[\frac{\Gamma\left(1 - \frac{1}{4}\right)\Gamma\left(\frac{1}{4}\right)}{\Gamma\left(\frac{1}{2}\right)} \right] \end{equation} Here we note that $\Gamma\left(\frac{1}{2}\right) = \sqrt{\pi}$ and via Eulers Reflection Formula which for $z \not \in \mathbb{Z}$: \begin{equation} \Gamma\left(z\right)\Gamma\left(1 - z\right) = \frac{\pi}{\sin\left(\pi z\right)} \longrightarrow \Gamma\left(\frac{1}{4}\right)\Gamma\left(1 - \frac{1}{4}\right) = \frac{\pi}{\sin\left(\frac{\pi}{4}\right)} = \frac{\pi}{\frac{1}{\sqrt{2}}} = \sqrt{2}\pi \end{equation} Thus, \begin{equation} \int_{0}^\infty\cos\left( t^2 \right)\:dt = \frac{1}{4} \frac{\sqrt{2}\pi}{\sqrt{\pi}} = \frac{\sqrt{2\pi}}{4} \end{equation} For the remaining integral with sine integrand we rely on the result from Section \ref{int_sin_x^n_0_to_inf}: \begin{equation} Q\left(n\right) = \int_0^\infty \sin\left(x^n\right)\:dx= \frac{1}{2n}\left[\frac{\Gamma\left(1 - \frac{n+1}{2n}\right)\Gamma\left(\frac{n+1}{2n}\right)}{\Gamma\left(1 - \frac{1}{n}\right)}\right] \end{equation} For convergence we require that $n > 1$. Thus, \begin{equation} \int_0^\infty \sin\left(x^2\right)\:dx = Q(2) = \frac{1}{2\cdot 2}\left[\frac{\Gamma\left(1 - \frac{2+1}{2\cdot 2}\right)\Gamma\left(\frac{2+1}{2\cdot 2}\right)}{\Gamma\left(1 - \frac{1}{2}\right)}\right] = \frac{1}{4}\left[ \frac{\Gamma\left(1 - \frac{3}{4}\right)\Gamma\left(\frac{3}{4}\right)}{\Gamma\left(\frac{1}{2}\right)}\right] \end{equation} We once again employ Euler's Reflection Formula (as given before) so that: \begin{equation} \int_0^\infty \sin\left(x^2\right)\:dx = \frac{1}{4}\cdot \frac{\pi}{\sin\left(\frac{3\pi}{4}\right)}\cdot \frac{1}{\sqrt{\pi}} = \frac{1}{4}\cdot \frac{\pi}{\frac{1}{\sqrt{2}}}\cdot \frac{1}{\sqrt{\pi}}= \frac{\sqrt{2\pi}}{4} \end{equation} We now may form our solution for $A(k)$: \begin{equation} A\left(k\right) = \cos\left(\frac{k^2}{4}\right)\left[\frac{\sqrt{2\pi}}{4} - C\left(\frac{k}{2}\right)\right] + \sin\left(\frac{k^2}{4}\right)\left[ \frac{\sqrt{2\pi}}{4} - S\left(\frac{k}{2}\right)\right] \end{equation} For $B(k)$, we first note: \begin{equation} u^2 - ku = \left(u - \frac{k}{2}\right)^2 - \frac{k^2}{4} \end{equation} And so, \begin{equation} \cos\left( u^2 - ku\right) = \cos\left( \left(u - \frac{k}{2}\right)^2 - \frac{k^2}{4}\right) \end{equation} Here we employ the Trigonometric Identity: \begin{equation} \cos\left(X - Y\right) = \cos\left(X\right)\cos\left(Y\right) + \sin\left(X\right)\sin\left(Y\right) \end{equation} And so, \begin{equation} \cos\left( \left(u - \frac{k}{2}\right)^2 - \frac{k^2}{4}\right) = \cos\left( \left(u - \frac{k}{2}\right)^2 \right)\cos\left(\frac{k^2}{4}\right) +\sin\left( \left(u - \frac{k}{2}\right)^2 \right)\sin\left(\frac{k^2}{4}\right) \end{equation} Thus: \begin{align} B\left(k\right) &= \int_0^\infty \left[ \cos\left( \left(u - \frac{k}{2}\right)^2 \right)\cos\left(\frac{k^2}{4}\right) +\sin\left( \left(u - \frac{k}{2}\right)^2 \right)\sin\left(\frac{k^2}{4}\right)\right]\:du \\ &= \cos\left(\frac{k^2}{4}\right)\int_0^\infty\cos\left( \left(u - \frac{k}{2}\right)^2 \right)\:du +\sin\left(\frac{k^2}{4}\right)\int_0^\infty\sin\left( \left(u - \frac{k}{2}\right)^2 \right)\:du \end{align} For both Integrals let $t = u - \frac{k}{2}$ and so $dt = du$: \begin{align} B\left(k\right) &= \int_0^\infty \left[ \cos\left( \left(u - \frac{k}{2}\right)^2 \right)\cos\left(\frac{k^2}{4}\right) +\sin\left( \left(u - \frac{k}{2}\right)^2 \right)\sin\left(\frac{k^2}{4}\right)\right]\:du \\ &= \cos\left(\frac{k^2}{4}\right)\int_{-\frac{k}{2}}^\infty\cos\left( t^2 \right)\:dt +\sin\left(\frac{k^2}{4}\right)\int_{-\frac{k}{2}}^\infty\sin\left( t^2 \right)\:dt \\ &= \cos\left(\frac{k^2}{4}\right)\left[\int_{0}^\infty\cos\left( t^2 \right)\:dt + \int_{-\frac{k}{2}}^0\cos\left( t^2 \right)\:dt\right] + \sin\left(\frac{k^2}{4}\right)\left[ \int_{0}^\infty\sin\left( t^2 \right)\:dt +\int_{-\frac{k}{2}}^0\sin\left( t^2 \right)\:dt\right] \\ &= \cos\left(\frac{k^2}{4}\right)\left[\int_{0}^\infty\cos\left( t^2 \right)\:dt + \int_{-\frac{k}{2}}^0\cos\left( t^2 \right)\:dt\right] + \sin\left(\frac{k^2}{4}\right)\left[ \int_{0}^\infty\sin\left( t^2 \right)\:dt +\int_{-\frac{k}{2}}^0\sin\left( t^2 \right)\:dt\right] \end{align} As covered in $A(k)$: \begin{equation} \int_{0}^\infty\cos\left( t^2 \right)\:dt = \int_{0}^\infty\sin\left( t^2 \right)\:dt = \frac{\sqrt{2\pi}}{4} \end{equation} For the remaining two Integrals, we make the substitution $t = -u$ And so $dt = -du$: \begin{align} B\left(k\right) &= \cos\left(\frac{k^2}{4}\right)\left[\frac{\sqrt{2\pi}}{4} + \int_{\frac{k}{2}}^0\cos\left( \left(-u\right)^2 \right)-\:du\right] + \sin\left(\frac{k^2}{4}\right)\left[\frac{\sqrt{2\pi}}{4} +\int_{\frac{k}{2}}^0\sin\left( \left(-u\right)^2 \right)-\:du\right] \\ &=\cos\left(\frac{k^2}{4}\right)\left[\frac{\sqrt{2\pi}}{4} + C\left(\frac{k}{2}\right)\right] + \sin\left(\frac{k^2}{4}\right)\left[\frac{\sqrt{2\pi}}{4} +S\left(\frac{k}{2}\right)\right] \end{align} We now may form our solution for $J'\left(k\right)$: \begin{align} J'\left(k\right) &= \frac{1}{2}\left[A\left(k\right) + B\left(k\right)\right] \\ &= \frac{1}{2}\bigg[ \left( \cos\left(\frac{k^2}{4}\right)\left[\frac{\sqrt{2\pi}}{4} - C\left(\frac{k}{2}\right)\right] + \sin\left(\frac{k^2}{4}\right)\left[ \frac{\sqrt{2\pi}}{4} - S\left(\frac{k}{2}\right)\right]\right) \\ &\quad + \left( \cos\left(\frac{k^2}{4}\right)\left[\frac{\sqrt{2\pi}}{4} + C\left(\frac{k}{2}\right)\right] + \sin\left(\frac{k^2}{4}\right)\left[\frac{\sqrt{2\pi}}{4} +S\left(\frac{k}{2}\right)\right]\right)\bigg] \\ &= \frac{1}{2}\cdot 2 \cdot \frac{\sqrt{2\pi}}{4}\left[\cos\left(\frac{k^2}{4}\right) + \sin\left(\frac{k^2}{4}\right) \right] = \frac{1}{2}\sqrt{\frac{\pi}{2}}\left[\cos\left(\frac{k^2}{4}\right) + \sin\left(\frac{k^2}{4}\right) \right] \end{align} We now integrate with respect to $k$ to yield our solution for $J\left(k\right)$: \begin{align} J\left(k\right) &= \int \left[\frac{1}{2}\sqrt{\frac{\pi}{2}}\left[\cos\left(\frac{k^2}{4}\right) + \sin\left(\frac{k^2}{4}\right) \right] \right]\:dk = \frac{1}{2}\sqrt{\frac{\pi}{2}}\left[\int \cos\left(\frac{k^2}{4}\right)\:dk + \int \sin\left(\frac{k^2}{4}\right)\:dk\right] \end{align} For both integrals, we make the substitution $u = \frac{k}{2}$ and so $dk = 2 \:du$: \begin{align} J\left(k\right) &= \int \left[\frac{1}{2}\sqrt{\frac{\pi}{2}}\left[\cos\left(\frac{k^2}{4}\right) + \sin\left(\frac{k^2}{4}\right) \right] \right]\:dk = \frac{1}{2}\sqrt{\frac{\pi}{2}}\left[\int \cos\left(u^2\right)\cdot 2 \:du+ \int \sin\left(u^2\right)\cdot 2 \:du\right]_{u = \frac{k}{2}} \\ &= \sqrt{\frac{\pi}{2}}\bigg[ C\left(u\right) + S\left(u\right)+ D\bigg]_{u = \frac{k}{2}} =\sqrt{\frac{\pi}{2}}\left[ C\left(\frac{k}{2}\right) + S\left(\frac{k}{2}\right)\right] + D \end{align} Where $D = $ Constant of Integration. To resolve $D$, we employ the initial condition given previously: \begin{equation} J\left(0\right) = 0 = \sqrt{\frac{\pi}{2}}\left[ C\left(\frac{0}{2}\right) + S\left(\frac{0}{2}\right)\right] + D = 0 + D \longrightarrow D = 0 \end{equation} And so, \begin{equation} J\left(k\right) = \sqrt{\frac{\pi}{2}}\left[ C\left(\frac{k}{2}\right) + S\left(\frac{k}{2}\right)\right] \end{equation} We now may form our solution for $I\left(a,b\right)$: \begin{equation} I\left(a,b\right) = \frac{1}{\sqrt{b}}J\left(\frac{a}{\sqrt{b}}\right) = \frac{1}{\sqrt{b}} \cdot \sqrt{\frac{\pi}{2}}\left[ C\left(\frac{\frac{a}{\sqrt{b}}}{2}\right) + S\left(\frac{\frac{a}{\sqrt{b}}}{2}\right)\right] =\sqrt{\frac{\pi}{2b}}\left[ C\left( \frac{a}{2\sqrt{b}}\right) + S\left( \frac{a}{2\sqrt{b}}\right) \right] \end{equation}
Your expression of $J$ seems correct (checked most of the proof, double checked with Maple).
Actually there is an error at the very beginning, in your change of variable $x=\frac{u}{\sqrt b}$ in $I(a,b)$: you forgot the factor $1/x$.
And the correct formula is thus:
$$I(a,b)=I(\frac{a}{\sqrt b},1)$$