I want to find the moment of inertia about the axis perpendicular to the base and passing through the center of mass (z-axis?). Mass is M and height is H and base has sides $a$.
I know I can take a slab of the pyramid and take the volume of the slab dv = $b^2$ x dH where $b$ is the length of the slab and dH is the height and then get the mass dm = $\rho$ x dv = $\rho$ x $b^2$ x dh. I'm stuck at this point and would appreciate any help.
On
let $\rho$ be the mass per unit volume, so that $M=\frac 13\rho a^2H$
Set the origin at the vertex and let the axis through the vertex and the centre of the base be the $x$ axis.
Consider an element in the form of a thin square lamina of thickness $\delta x$ which is parallel to the base and at a distance $x$ from O. Let the side of this square be $2y$.
Then $$\frac yx=\frac {a}{2H}.$$
The volume of the element is $\delta V=4y^2\delta x$
The mass of the element is $\delta m=4\rho y^2 \delta x$
Now using the standard result for the moment of inertia of a square lamina about an axis perpendicular to the plane of the lamina and through the centre, we have the MI of the element as $$\frac 16\delta m (2y)^2=\frac 83\rho y^4\delta x$$
So the MI is given by $$\frac 83\frac {3M}{a^2H}\int_0^H\frac{a^4x^4}{16H^4}dx=\frac{1}{10}Ma^2$$
The basic integral you need is $\displaystyle I=\int_Q r^2 \, dm$. The problem with your approach so far is that the slab is not equidistant from the axis of rotation, so that $r^2$ varies over your slab. You need to take a chunk where you can approximate the $r^2$ as constant over your chunk. That means a chunk $dV=dx\,dy\,dz$. So here, $dm=\rho\,dx\,dy\,dz$, where you can get $\rho=M/V$, assuming constant density. Then you need to set up your integral as $$I=\rho\iiint_Qr^2\,dx\,dy\,dz.$$ If you draw some careful pictures, you can find out how $r^2$ varies with respect to $x,y,$ and $z$, and you can get your limits for each of the variables. Can you proceed from here?