My attempt: I computed the mass and the $x$ coordinate. The solution contains the final coordinates and on checking, I observed that my answer is wrong.

The answer states that $$x=\frac{3a}{4}.$$ Where am I going wrong?
Copyright © 2021 JogjaFile Inc.

I consider sections or lines for the integration limits. In this case if I try to integrate first wrt $y$, I imagine vertical lines for some values of $x$ in between and the limits are clear: down, $0$ and up, the line $y=(b/a)x$
$x$ varies between $0$ and $a$ and the upper line limiting $y$ is $y=(b/a)x$
$$m=\int_0^a\int_0^{(b/a)x}x\mathbb dy\mathbb dx=\int_0^a\left[y\right]_0^{(b/a)x}x\mathbb dx=(b/a)\int_0^ax^2\mathbb dx$$
$$m=\dfrac{a^2b}{3}$$
$$x_{cm}=\dfrac{1}{m}\int_0^a\int_0^{(b/a)x}\rho x\mathbb dy\mathbb dx=\dfrac{3}{a^2b}\int_0^a\int_0^{(b/a)x}x^2\mathbb dy\mathbb dx=$$
$$=\dfrac{3}{a^2b}\int_0^a\left[y\right]_0^{(b/a)x}x^2\mathbb dx=\dfrac{3}{a^2b}\dfrac{b}{a}\int_0^ax^3\mathbb dx$$
$$x_{cm}=\dfrac{3}{a^2b}\dfrac{b}{a}\dfrac{a^4}{4}=\dfrac{3a}{4}$$
As expected.