Suppose I have an integral that looks like:
$$I=\int_{r=0}^\infty\int_{\omega_1=-\infty}^\infty\int_{\omega_2=-\infty}^\infty F(\omega_1,\omega_2)e^{-jr(\omega_1\cos(\alpha)+\omega_2\sin(\alpha))}r \ \mathrm dr \ \mathrm d \omega_1 \mathrm d \omega_2$$
where $F(\omega_1,\omega_2)$ is a characteristic function of random variable.
And I want to change coordinates from $(\omega_1,\omega_2)$ to $(\Omega,\theta)$ according to the rule:
$$\begin{cases} \omega_1=\Omega\cos(\theta) \\ \omega_2=\Omega\sin(\theta) \end{cases}$$
With $\Omega\in (-\infty,\infty), \ \theta\in (-\pi/2,\pi/2)$. Is it possible?
So I should transform $F(\omega_1,\omega_2)\to F(\Omega,\theta)$ and $\mathrm d \omega_1 \mathrm d \omega_2\to \mathrm d\Omega \ \mathrm d \theta$
$$
\begin{eqnarray*}
\frac{\partial \left( \omega_1,\omega_2\right) }{\partial \left( \Omega,\theta\right) }
&=&\det
\begin{pmatrix}
\partial \omega_1/\partial \Omega & \partial \omega_1/\partial \theta \\
\partial \omega_2/\partial \Omega & \partial \omega_2/\partial \theta
\end{pmatrix}
\\
&=&\det
\begin{pmatrix}
\cos(\theta) & -\Omega\sin(\theta)\\
\sin(\theta) & \Omega\cos (\theta)
\end{pmatrix}
\\
&=&\Omega\cos ^{2}(\theta) +\Omega\sin ^{2}(\theta) \\
&=&\Omega.
\end{eqnarray*}
$$
So I have $\mathrm d \omega_1 \mathrm d \omega_2\to \Omega \mathrm d\Omega \ \mathrm d \theta $ and $F(\omega_1,\omega_2)= \frac{F(\Omega,\theta)}{\left|\frac{\partial \left( \omega_1,\omega_2\right) }{\partial \left( \Omega,\theta\right) }\right|}=\frac{F(\Omega,\theta)}{\Omega}$
$$
\require{cancel}
I=\int_{\Omega=-\infty}^\infty\int_{r=0}^\infty\int_{\theta=-\pi}^\pi F(\Omega,\theta)e^{-j\Omega r(\theta-\alpha)}\frac{r}{\cancel{\Omega}} \cancel{\Omega}\ \mathrm d\Omega \ \mathrm d \theta \mathrm dr$$
So finally I get:
$$I=\int_{\Omega=-\infty}^\infty\int_{r=0}^\infty\int_{\theta=-\pi}^\pi F(\Omega,\theta)e^{-j\Omega r\cos(\theta-\alpha)}r\ \mathrm d\Omega \ \mathrm d \theta \mathrm dr$$
Is that right?
When choosing a parametrization $\psi$ of the $(\omega_1,\omega_2)$-plane in the form $$\psi:\qquad\omega_1=\Omega\cos\theta,\quad \omega_2=\Omega\sin\theta$$ you have to choose a parameter domain such that $\psi$ is essentially bijective, and then stick to it. Both $$-\infty<\Omega<\infty,\quad -{\pi\over2}\leq\theta\leq{\pi\over2}\tag{1}$$ and $$0\leq\Omega<\infty,\quad -\pi\leq\theta\leq\pi\tag{2}$$ are possible choices.
You have computed the Jacobian $J_\psi(\Omega,\theta)=\Omega$ correctly; but there is one point you missed: When transforming the integral you have to introduce the absolute value $|J_\psi(\Omega,\theta)|=|\Omega|$ as a factor, and not $J_\psi$. When you use the parameter domain $(1)$ the Jacobian gets negative for negative $\Omega$; therefore you will have to split the integral into two parts according to the sign of $\Omega$. To cut a long story short: It's more convenient to work with $(2)$ throughout.
Doing this your integral would become $$I=\int_{r=0}^\infty\int_{\Omega=0}^\infty\int_{\theta=-\pi}^\pi \tilde F(\Omega,\theta)\ \exp(\ldots)\ d\theta\ \Omega\>d\Omega \ r\> dr$$ with $$\tilde F(\Omega,\theta)=F(\Omega\cos\theta,\Omega\sin\theta)\ ,$$ and not what you had in mind.