I need to show $$I\equiv\int_0^\infty\int_0^\infty e^{-(x^2+y^2+2xy\cos\theta)} \, dx \, dy=\frac{\theta}{2\sin\theta}.$$
And the hint given is to consider oblique $xy$-coordinate.
I kinda can see how answers comes out. If I transform the coordinate system where integrand becomes $e^{((1-\cos\theta) x'^2+(1+\cos\theta) y'^2)}$ and the integral argument is from $0$ to $\theta,$ I would get the similar value. But I'm stuck on how to construct transformation.
You could do this via the Jacobian, but it's possible to transform more or less one variable at a time: $$\begin{align}\int_0^{\infty}\int_0^{\infty}e^{-(x^2+y^2+2xy\cos\theta)}dx\,dy&=\int_0^{\infty}e^{-y^2\sin^2\theta}\left\{\int_0^{\infty}e^{-(x+y\cos\theta)^2}dx\right\}dy\\ &=\int_0^{\infty}e^{-y^2\sin^2\theta}\left\{\int_{y\cos\theta}^{\infty}e^{-u^2}du\right\}dy\\ &=\int_0^{\infty}e^{-v^2}\left\{\int_{v\cot\theta}^{\infty}e^{-u^2}du\right\}\frac{dv}{\sin\theta}\\ &=\csc\theta\int_0^{\infty}\int_{v\cot\theta}^{\infty}e^{-u^2-v^2}dudv\\ &=\csc\theta\int_{0}^{\theta}\int_0^{\infty}e^{-r^2}r\,dr\,dphi\\ &=\csc\theta\int_{0}^{\theta}\left[-\frac12e^{-r^2}\right]_0^{\infty}d\phi\\ &=\frac12\csc\theta\int_{0}^{\theta}d\phi\\ &=\frac12\csc\theta\left(\theta\right)\end{align}$$ So first we transformed $u=x+y\cos\theta$, then $y=y\sin\theta$, and then we could go into polar coordinate $u=r\cos\phi$, $v=r\sin\phi$ and then we had elementary integrals.