Given the joint pdf as $f_{X,Y}(x,y)=3x$ where $0\le y\le x\le1$ , I'm trying to calculate the covariance
My attempt:
First calculate $E\left(X\right)$ , $E\left(Y\right)$ and $E\left(XY\right)$ separately
$$f_X\left(x\right)=\int_{0}^{x}3x\>dy$$
$$E\left(X\right)=\int_{0}^{1}x\cdot f_X\left(x\right)dx$$
$$f_Y\left(y\right)=\int_{y}^{1}3xdx$$
$$E\left(Y\right)=\int_{0}^{1}y\cdot f_Y\left(y\right)dy$$
$$E\left(XY\right)=\int_{0}^{1}\int_{0}^{x}xy\left(3x\right)dydx$$
then substitute the above values in the equation $Cov\left(X,Y\right)=E\left(XY\right)-E\left(X\right)E\left(Y\right)$
My main question is regarding selecting the correct upper and lower bounds when intergrating. For instance, when finding $E(XY)$, why is it wrong if we use $$E\left(XY\right)=\int_{0}^{1}\int_{0}^{1}xy\left(3x\right)dydx$$
instead of
$$E\left(XY\right)=\int_{0}^{1}\int_{0}^{x}xy\left(3x\right)dydx$$
Always remember the support.
Your support is $0\leqslant y\leqslant x\leqslant 1$. The joint density function only equals $3x$ when $x$ and $y$ exist inside the support, elsewhere the density is $0$. That is, it is a piecewise function.$$\begin{align}f_{X,Y}(x,y) & = \begin{cases} 3x &:& 0\leqslant y\leqslant x\leqslant 1\\0&:&\textsf{otherwise}\end{cases}\\[2ex] &= 3x\,\mathbf 1_{0\leqslant y\leqslant x\leqslant 1}\end{align}$$
When finding $f_X(x)$ you "integrate out" $y$, which lives between $0$ and $x$, leaving $x$ between $0$ and $1$.
$$\begin{align}f_X(x) &= \int_{0\leqslant y\leqslant x}3x\,\mathbf 1_{0\leqslant x\leqslant 1}\,\mathrm d y\\[2ex]&= 3x\,\mathbf 1_{0\leqslant x\leqslant 1}\,\int_0^x\,\mathrm d y\\[2ex]&= 3x^2\,\mathbf 1_{0\leqslant x\leqslant 1}\end{align}$$
When finding $f_Y(y)$ you "integrate out" $x$, which lives between $y$ and $1$, leaving $y$ between $0$ and $1$.
$$\begin{align}f_Y(y) &= \int_{y\leqslant x\leqslant 1}3x\,\mathbf 1_{0\leqslant y\leqslant 1}\,\mathrm d x\\[2ex]&= 3\,\mathbf 1_{0\leqslant y\leqslant 1}\,\int_y^1 x\,\mathrm d x\\[2ex]&= \tfrac 32(1-y^2)\,\mathbf 1_{0\leqslant y\leqslant 1}\end{align}$$
When performing a double integral, you "interate out" the inner variable, then integrate out the outer. $$\begin{align}\mathsf E(g(X,Y)) &=\iint_{0\leqslant y\leqslant x\leqslant 1} 3x\,g(x,y)\,\mathrm d (x,y)\\&=\int_{0\leqslant x\leqslant 1}\int_{0\leqslant y\leqslant x} 3x\,g(x,y)\,\mathrm d y\,\mathrm dx&&=3\int_0^1 x\int_0^x g(x,y)\,\mathrm d y\,\mathrm dx\\&=\int_{0\leqslant y\leqslant 1}\int_{y\leqslant x\leqslant 1} 3x\,g(x,y)\,\mathrm d x\,\mathrm d y&&=3\int_0^1\int_y^1 x\,g(x,y)\,\mathrm d x\,\mathrm d y \end{align}$$
So $$\begin{align}\mathsf E(X) &= \int_0^1 x \int_0^x 3x\,\mathrm d y\,\mathrm d x&&=\int_0^1 x\, f_X(x)\,\mathrm d x\\[2ex]\mathsf E(Y) &= \int_0^1 y\int_y^1 3x\,\mathrm d x\,\mathrm d y &&= \int_0^1 y\,f_Y(y)\,\mathrm d y\\[3ex]\mathsf E(XY) &= \int_0^1\int_0^x xy\cdot 3x\,\mathrm d y\,\mathrm d x \\[1ex]& = \int_0^1\int_y^1 xy\cdot 3x\,\mathrm d x\,\mathrm d y \end{align}$$
And so forth.