Find the center of mass of that part of the sphere
$x^2+y^2+z^2 \le a^2$
having $x,y,z \ge 0$ (that is, the part in the first octant)
With density given by $\rho(x,y,z)=(x^2+y^2+z^2)^{3/2}$
It should be solved using a triple integral.
I get either $\frac{2a}3$ or $\frac{6a}7$, but I don't know whether any of these are correct. I used spherical coordinates. The problem is that the density function has a power of $\frac{3}{2}$.
You first need to find the total mass, which you could find by integrating over the whole sphere and dividing by $8$, but it is better to set up the problem as you will need for the coordinates. The first octant has intervals $\theta \in [0,\pi/2]$ and $\phi \in [0,\pi/2]$, so that the mass is
$$M=\int_0^a dr \, r^2 \rho(r) \, \int_0^{\pi/2} d\theta \, \sin{\theta} \, \int_0^{\pi/2} d\phi$$
The coordinates are then given by
$$\bar{x} = \frac1{M} \int_0^a dr \, r^3 \rho(r) \, \int_0^{\pi/2} d\theta \, \sin^2{\theta} \, \int_0^{\pi/2} d\phi\, \cos{\phi}$$
$$\bar{y} = \frac1{M} \int_0^a dr \, r^3 \rho(r) \, \int_0^{\pi/2} d\theta \, \sin^2{\theta} \, \int_0^{\pi/2} d\phi\, \sin{\phi}$$
$$\bar{z} = \frac1{M} \int_0^a dr \, r^3 \rho(r) \, \int_0^{\pi/2} d\theta \, \sin{\theta} \cos{\theta} \, \int_0^{\pi/2} d\phi$$
I get $$\left (\bar{x},\bar{y},\bar{z}\right ) = \left (\frac{3 a}{7},\frac{3 a}{7},\frac{3 a}{7} \right )$$
To compare, the analogous result for uniform density is
$$\left (\bar{x},\bar{y},\bar{z}\right ) = \left (\frac{3 a}{8},\frac{3 a}{8},\frac{3 a}{8} \right )$$
which is closer to the center.