Suppose I want to find the center of mass of a cone, from its vertex. It is a right circular solid cone of radius $a$.
About the $z$ axis, the center of mass is given by : $$z_{com}=\frac{\int dm\,z}{\int dm}$$
In cylindrical coordinates, let $dm=rdrd\phi dz$
Moreover, we know that for a cone, we have the following relation :
$$\frac{z}{r}=\frac{h}{a}$$
Hence, I can have $z=\frac{hr}{a}$ or $r=\frac{az}{h}$.
Normally this is how we do the integral :
$$\frac{\int_{0}^{h}\int_{0}^{\frac{az}{h}}rdrzdz}{\int_{0}^{h}\int_{0}^{\frac{az}{h}}rdrdz} = \frac{3h}{4}$$
(I've ignored the integral over the angle and the density, since that cancels out)
Here, $r$ ranges from $0$ to $\frac{az}{h}$. Then we integrate over $z$ from $0$ to $h$. However, I could have first integrated over $z$ and then integrated over $r$ - this should give me the same answer but it doesn't. In this case, $z$ should range from $\frac{hr}{a}$, and then $r$ should range from $0$ to $a$.
Hence we would have : $$\frac{\int_{0}^{a}\int_{0}^{\frac{hr}{a}}zdz\,rdr}{\int_{0}^{a}\int_{0}^{\frac{hr}{a}}dz\,rdr} = \frac{3h}{8}$$
There is an extra factor of $1/2$ coming from somewhere. For some reason, the order of integration seems to matter here. However, how would one know which is the correct order, and why exactly does the order even matter ?
Your work considers a cone with vertex at the origin.
Now if you are integrating wrt $dz$ first then for all values of $r$, the upper bound of $z$ is $h$ and the lower bound is a function of $r$. Please see the diagram at the end of the answer that shows your mistake. The integral in the numerator should be,
$ \displaystyle 2\pi\int_0^a \int_{(r h) /a}^{h} z ~r ~dz~ dr = 2\pi \cdot \frac{a^2h^2}{8}$
and in the denominator it should be,
$\displaystyle 2\pi\int_0^a \int_{(r h) /a}^{h} ~r ~dz~ dr = 2 \pi \cdot \frac{a^2h}{6}$
Below is a diagram that shows $2$D projection of the cone from side. Based on your bounds, for every value of $r \in (0, a)$, you are integrating over the vertical strip marked in red. That gives you center of mass of the cylinder of radius $a$ and height $h$ with the given cone cut out of it. That is the unshaded region in the diagram. You should instead be integrating over the blue strip.