Consider the Cartesian coordinate system with a vector $$f=(a(x^2+y^2)^{N/2}\cos(N\theta),a(x^2+y^2)^{N/2}\sin(N\theta),bz)$$where $a,b\in\mathbb{R}$ fixed and $N$ is an integer.
For $f'=f/|f|$, I want to determine the integral
$$\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}f'\cdot\left(\frac{\partial f'}{\partial x}\times\frac{\partial f'}{\partial y}\right) dxdy.$$
To do that, we transform to "almost spherical" coordinates
$r=\sqrt{a^2(x^2+y^2)^N+b^2z^2}$
$\tan\theta=\frac{y}{x}$
$\tan\phi=\frac{a(x^2+y^2)^{N/2}}{bz}$
with $\theta\in[0,2\pi]$, $\phi\in[0,\pi]$.
How do I rewrite the integral to these new coordinates?
I find that $f=(r\sin\phi\cos(N\theta),r\sin\phi\sin(N\theta),r\cos\phi)$ and $f'=(\sin\phi\cos(N\theta),\sin\phi\sin(N\theta),\cos\phi)$, but how do I rewrite $\frac{\partial f'}{\partial x}\times\frac{\partial f'}{\partial y}$?
First, to answer your question, you need to find the Jacobian Matrix $\mathcal{J}$ between the 2 set of coodinates. And then $dxdy = |\mathcal{J}|drd\theta$, where $|\mathcal{J}|$ is the determinant of the Jacobian Matrix.
There are a couple more problems in your thinking
not sure which one you are referring to , but I guess it is the third one
So if you define the transformation between $x,y$ and $r,\theta$ as $$ r = a(x^2 + y^2) ^{N/2} \quad \tan\theta = \frac{y}{x} $$
Your Jacobian matrix is $$ \mathcal{J} = \begin{bmatrix} \frac{\partial x}{\partial r} & \frac{\partial x}{\partial \theta} \\ \frac{\partial y}{\partial r} & \frac{\partial y}{\partial \theta} \\ \end{bmatrix} = \begin{bmatrix} \frac{2}{Nr}(\frac{r}{a})^{2/N}\cos\theta & -(\frac{r}{a})^{2/N}\sin\theta \\ \frac{2}{Nr}(\frac{r}{a})^{2/N}\sin\theta & (\frac{r}{a})^{2/N}\cos\theta \\ \end{bmatrix} $$
so
$$ dxdy = |\mathcal{J}|drd\theta = \frac{2}{Nr}\left(\frac{r}{a}\right)^{4/N}drd\theta $$
Just a sanity check, when $N=2, a=1$, we can observer that it falls back to the form of polar cooridnate $$ dxdy = rdrd\theta $$