Let $a,b \in \mathbb R$ such that $a^2+b^2 = 1$
We wish to determine whether this integral converges, and if it does, calculate it:
$\int_D \frac{1}{\sqrt{1-ax-by}}\,dx\,dy$ where $D = \{(x,y) \in \mathbb R^2 \mid x^2+y^2<1\}$
Since the domain is a circle it would make a lot of sense to transform it to $$\int_0^1 \int_0^{2\pi}\frac{r}{\sqrt{1-ar\cos\theta-br\sin\theta}}\,d\theta \, dr$$
But this function does not have an anti derivative (where theta is the variable and $r$ is contant).
If we change the order of integration, we get something equally unpleasant. I tried integrating by parts but that didn't lead me anywhere.
How do we calculate this integral?
You are not exploiting the constraint $a^2+b^2=1$. You may assume $a=\cos\varphi$ and $b=\sin\varphi$, hence by switching to polar coordinates your integral becomes
$$ \int_{0}^{1}\int_{0}^{2\pi}\frac{\rho}{\sqrt{1-\rho\sin(\theta+\varphi)}}\,d\theta\,d\rho = \int_{0}^{1}\int_{0}^{2\pi}\frac{\rho}{\sqrt{1-\rho\sin\theta}}\,d\theta\,d\rho$$ or, by Fubini's theorem, $$ \int_{0}^{2\pi}\frac{2}{3\sin^2\theta}\left[2-\sqrt{1-\sin\theta}\,(2+\sin\theta)\right]\,d\theta = 2\int_{0}^{\pi}\frac{2}{3\cos^2\theta}\left[2-\sqrt{1-\cos\theta}\,(2+\cos\theta)\right]\,d\theta $$ which simplifies into $\frac{8}{3}\sqrt{2}$ through the tangent half-angle substitution. Anyway, it is much simpler to exploit the independence from $\varphi$ and just assume $(a,b)=(1,0)$.