Well I am stuck from the beginning. I have two methods:
Method 1:
$y=\sqrt{a^2−x^2}$ here $y$ varies from $\sqrt{a^2−x^2}$ to $0$ $x$ varies from $a$ to $0$ $\iint(x^2+y^2) dx\, dy= \int\left(x^2y+\frac{y^3}{3}\right) dx$ then I do not how to proceed
Method 2: Changing into polar coordinates. $$\text{Let}\ x=r\cosθ, y=r\sinθ,$$ $$x^2+y^2= r^2$$ hence $r$ varies from $a$ to $0$; $θ$ varies from $0$ to $90$
\begin{align*} J(x,\frac{y}{r},θ)&= r\\ \iint(x^2+y^2) dx\, dy&= \iint(x^2+y^2)J dr\, dθ\\ &= \iint (r^2\cdot r)\ dr\, dθ\\ &= \iint r^3 dr\, dθ\\ &= \int \frac{r^4}{4} dθ\\ &= \int \frac{a^4}{4} dθ\\ &= \frac{a^4}{4}\cdot\frac{\pi}{2} \end{align*}
But that is not the answer. The answer is actually said to be $\frac{a^5}{5}$. Could anyone please help me?
If you make the change of variable $x = r\cos(\theta)$ and $y = r\sin(\theta)$, the proposed integral reduces to \begin{align*} \iint_{R}(x^{2} + y^{2})\mathrm{d}x\mathrm{d}y = \int_{0}^{\pi/2}\int_{0}^{a}r^{3}\mathrm{d}r\mathrm{d}\theta = \frac{\pi a^{4}}{8} \end{align*}
So your calculations are correct!