For which values of $a,b > 0$ is the function
$f : (0, \infty)$ x $\mathbb{R} \to \mathbb{R}: (x,y) \mapsto \dfrac{1}{x^a(1+x^2+y^2)^b}$ integrable?
I have tried to perform to change this function to a function in polar coordinates but then I get a $\int_{-\pi/2}^{\pi/2} \dfrac{1}{cos^a(\theta)}d\theta$ which is not integrable for any $a > 0$.
Does somebody have an idea how I can solve this problem with another method or substitution?
You haven't pursued this correctly yet by changing to polar coordinates.
Since the integrand is nonnegative we can apply Tonelli's theorem and evaluate as iterated integrals in any order, although the integral may be infinite. Changing to polar coordinates this becomes
$$\int_{-\pi/2}^{\pi/2} \frac{d \theta}{(\cos \theta)^a}\int_0^\infty \frac{dr}{r^{a-1}(1 + r^2)^b}.$$
Since
$$\int_{0}^{\pi/2} \frac{d \theta}{(\cos \theta)^a} = \int_{-\pi/2}^{0} \frac{d \theta}{(\cos \theta)^a} = \int_{0}^{\pi/2} \frac{d \theta}{(\sin \theta)^a}, $$
and $\sin \theta \sim \theta$ as $ \theta \to 0$, the $\theta-$integral converges if and only if $a <1$.
Note, for example, that $\displaystyle\int_0^{\pi/2} \frac{ d\theta}{\sqrt{\theta}} = 2 \sqrt{\pi/2}$ is convergent.
For the $r-$integral consider that the integrand is $\sim r ^{1-a}$ as $r \to 0$ and $\sim r^{1-a - 2b}$ as $r \to \infty$ to determine the conditions for convergence/divergence.