I am trying to integrate this function:
$$ \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} \exp\{ -\lvert xy \rvert - \phi(x^2 + y^2) \} dx dy $$
A little bit of background: the negative logarithm of this function is used as a penalty term in a regularized regression framework called pathway lasso. I am trying to understand the probabilistic meaning of such a penalty function, i.e. if this corresponds to any valid probability density function etc.
$$I=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty} \exp\{ -\lvert xy \rvert - \phi(x^2 + y^2) \} dx dy$$
becomes, using a cartesian to polar change of variables :
$$I=\int_{\theta=0}^{2 \pi}\int_{r=0}^{\infty} \exp\{ -r^2 \lvert \sin \theta \cos \theta \rvert - \phi r^2 \} \color{'r'}{r} dr d\theta$$
(where the last $r$ is the Jacobian). Using formula $\sin 2\theta=2\sin \theta\cos \theta$ :
$$I=\int_{\theta=0}^{2 \pi}\int_{r=0}^{\infty} \exp\{ -\tfrac12 r^2( \lvert \sin 2\theta \rvert + 2 \phi)\} \color{'r'}{r} dr d\theta$$
$$I=\int_{\theta=0}^{2 \pi}\left(\int_{r=0}^{\infty} \exp\{ -\tfrac12 r^2 A \} \color{'r'}{r} dr \right) d\theta \ \ \text{with} \ \ A:=\lvert \sin 2\theta \rvert + 2 \phi$$
$$I=\int_{\theta=0}^{2 \pi}\left[-\dfrac{1}{A}\exp\{ -\tfrac12 r^2 A \} \right]_{r=0}^{\infty} d\theta$$
$$I=\int_{\theta=0}^{2 \pi}\dfrac{1}{A} d\theta=\int_{\theta=0}^{2 \pi}\dfrac{1}{2 \phi + |\sin 2\theta|} d\theta$$ $$I=\dfrac{1}{2}\int_{\alpha=0}^{\pi}\dfrac{1}{2 \phi + \sin \alpha} d\alpha$$
(using change of variable $\alpha=2\theta$, as advised by @Ryan Greyling, allowing to drop the absolute values).
Using the change of variable $t=\tan \tfrac{\alpha}{2}$ (therefore with $0\leq t \leq \infty$), one obtains $\sin \alpha=\dfrac{2t}{1+t^2}$ with $d\alpha=\dfrac{dt}{1+t^2}.$
I leave this final calculation up to you...