How to solve the surface area of a sphere $x^2 + y^2 + z^2 = r^2$ cut out by $x^2 +y^2 = r^2 - (r \cos\theta)^2$ that is above $xy$-plane
$$\frac{dz}{dx} = \frac{-x}{\sqrt{r^2-x^2-y^2}}$$
$$\frac{dz}{dy} = \frac{-y}{\sqrt{r^2-x^2-y^2}}$$
$$\begin{align} Serface Area & = \iint \sqrt{1+(\frac{dz}{dx})^2 +(\frac{dz}{dy})^2} dx\,dy\\& = \iint \sqrt{1+(\frac{-x}{\sqrt{r^2-x^2-y^2}})^2 + (\frac{-y}{\sqrt{r^2-x^2-y^2}})^2 }dx\,dy\\ &= \iint \sqrt{\frac{r^2}{r^2-x^2-y^2}} dx\,dy \\ &= \int^{2\pi}_{0}\int^{\sqrt{r^2 - (r \cos\theta)^2}}_{0} \sqrt{\frac{r^2}{r^2-(r^2cos\theta+r^2sin\theta))}}r\,dr\,d\theta \\ & = \int^{2\pi}_{0}\int^{\sqrt{r^2 - (r \cos\theta)^2}}_{0} \sqrt{\frac{r^2}{r^2-r^2}}r\,dr\,d\theta \end{align}$$
After changing it to polar coordinate with $x=rcos\theta$ and $y=rsin\theta$, the denominator is $0$. What am I doing wrong here?
As to why you are having trouble with the double integral, in the work you have shown, there is some confusion regarding polar coordinates. $r$ and $\theta$ are already taken up in the specification of the problem. As I mention in the other answer, the plane is at $z=r \cos \theta$. So, if you want to parametrize the plane for the carrying out a double integral, I suggest another set of variables for that purpose, say $\rho$ and $\varphi$. As you have noted correctly, the double integral for the surface area is
$$S=\int \int \frac{r}{\sqrt{r^2-x^2-y^2}}dxdy $$
Making the change of variables $x=\rho \cos \theta$, $y=\rho \sin \theta$
$$S=\int_0^{2 \pi} \int_0^{r \sin \theta} \frac{r}{\sqrt{r^2-\rho^2}}\rho d\rho \varphi = 2 \pi r \int_0^{r \sin \theta} \frac{1}{\sqrt{r^2-\rho^2}}\rho d\rho$$
We can integrate that easily with a change of variables $t^2=r^2-\rho^2$ to get $$S=2\pi r^2(1- \cos\theta)$$