I am trying to find the area of a quadrant of an ellipse by double integrating polar coordinates but the answer I'm getting is incorrect.
ellipse : $ x^2/a^2 + y^2/b^2 =1 $
Any point on ellipse : $ ( a\cos(\theta), b\sin(\theta)) $
At $ \theta$, taking $ d\theta $ segment, Thus $ r^2 = a^2\cos^2(\theta) + b^2\sin^2(\theta) $ [Using pythagoras theorem] $$ Area = \int_{0}^{\pi/2} \int_{0}^{\sqrt{a^2\cos^2(\theta) + b^2\sin^2(\theta)}} rdrd\theta $$
$$ = 1/2 \int_{0}^{\pi/2} r^2 \Big|_{0}^{\sqrt{a^2\cos^2(\theta)+b^2\sin^2(\theta)}} d\theta $$
$$ = 1/2 \int_{0}^{\pi/2} (a^2\cos^2(\theta)+b^2\sin^2(\theta)) d\theta $$ $$ = 1/2 \int_{0}^{\pi/2} ((a^2 - b^2)\cos^2(\theta)+b^2) d\theta $$ $$ = 1/4 \int_{0}^{\pi/2} (a^2 - b^2)(1+ \cos(2\theta)) d\theta +2b^2 d\theta $$ I am getting $$ \pi/8 (a^2 + b^2).$$ But the correct answer is $ \pi ab/4 $
Your mistake is to believe that $\theta$ is the polar argument. It is not, because
$$\tan\phi=\frac yx=\frac ba\tan \theta\ne\tan\theta.$$
You can fix by taking the differential
$$(\tan^2\phi+1)\,d\phi=\frac ba(\tan^2\theta+1)\,d\theta$$
and substituting
$$\left(\frac{b^2}{a^2}\tan^2\theta+1\right)\,d\phi=\frac ba(\tan^2\theta+1)\,d\theta$$
so that the integral becomes
$$\frac12\int_0^{\pi/2}\frac{\dfrac ba(\tan^2\theta+1)}{\dfrac{b^2}{a^2}\tan^2\theta+1}(a^2\cos^2\theta+b^2\sin^2\theta)\,d\theta \\=\frac{ab}2\int_0^{\pi/2}\frac{\sin^2\theta+\cos^2\theta}{b^2\sin^2\theta+a^2\cos^2\theta}(a^2\cos^2\theta+b^2\sin^2\theta)\,d\theta.$$