So the problem is $p(x,y) = 120xy(1-x-y)I \{x \geq0,y \geq0,x+y \leq 1 \}$
Now that $Z = Y - E(Y|X)$ What is the correlation coefficient of $Z$ and $X$
So here I First tried to get $E(Z)$, which
$E(Z) = E(Y) - E(E(Y|X)) = E(Y) - E(Y) = 0$
Also $E(Z^2) = E(Y^2 - 2YE(Y|X) - E(Y)^2) = E(Y^2) - E(Y)^2 = Var(Y)$
Hence that
$\sqrt{Var(Z)} = \sqrt{Var(Y)}$
Also for covariance
$Cov(X,Z) = E(XZ) - E(X)E(Z) = E(XY - XE(Y|X)) = E(XY) - E(X)E(Y) = Cov(X,Y)$
Now that I got $\rho_{XZ} = \rho_{XY} = \frac{Cov(X,Y)}{\sqrt{Var(X)}\sqrt{Var(Y)}}$
Here now I tried to get $E(X),E(Y),E(X^2)E(Y^2)$
But the calculation gets really messy that I can't continue
$P_X(X) = \int_{0}^{1-x}120xy(1-x-y)dy = 20(1-x)^3$
$E(X) = \int_{0}^{1-y}20x(1-x)^3dx = 10(1-y)^2-20(1-y)^3+15(1-y)^4-4(1-y)^5$
$E(X^2) = \int_{0}^{1-y}20x^2(1-x)^3dx = \frac{20}{3}(1-y)^3-15(1-y)^4+12(1-y)^5-\frac{20}{6}(1-y)^6$
More it goes more messy it gets, thinking that I am doing something wrong, it will get more messy when I find $E(X)^2$
If you have any idea what I am doing wrong can you help??
Here is a no calculation approach.
Consider $\langle X \rangle$ the linear span of $X$ in $\mathscr{L}^2,$ and $\mathrm{F}_X = \{\mathbf{E}(Z \mid X) \mid Z \in \mathscr{L}^2\},$ which is also a vector space. Clearly $\langle X \rangle \subset \mathrm{F}_X.$ It is well known that $Z \mapsto \mathbf{E}(Z \mid X)$ is the orthogonal projection onto $\mathrm{F}_X,$ a fortiori $Z - \mathbf{E}(Z \mid X) \perp \langle X \rangle.$ The covariance between $Z$ and $X$ is just the inner product, so the covariance between $Z - \mathbf{E}(Z \mid X)$ and $X$ is zero.