The joint pdf is as follows:
$f(x,y) = 5040x^3y^5$ ($0≤x, 0≤y, x+y≤1$)
I have worked out: $f_{X}(x)=840x^3(1-x)^6$ for ($0≤x≤1$) $f_{Y}(y)=1260(1-y)^4y^5$ for ($0≤y≤1$)
However, when working out E(X), E(Y) and E(XY) I am unsure what limits to use which is making me get an answer of $\cfrac{17400}{121}$ for the covariance and hence, a massively wrong answer for the correlation.
Can anyone help? It would be massively appreciated
Note that the support of $f_{X,Y}(x,y)$ is the triangle with vertices $$(0,0), (1,0), (0,1).$$ For a given $x \in [0,1]$, the values of $y$ range over $[0, 1-x]$.
It is instructive to consider the general case
$$\begin{align*} \operatorname{E}[X^k Y^m] &= \int_{x=0}^1 \int_{y=0}^{1-x} x^k y^m \cdot 5040 x^3 y^5 \, dy \, dx \\ &= 5040 \int_{x=0}^1 x^{3+k} \int_{y=0}^{1-x} y^{5+m} \, dy \, dx \\ &= 5040 \int_{x=0}^1 x^{3+k} \left[\frac{y^{6+m}}{6+m}\right]_{y=0}^{1-x} \, dx \\ &= \frac{5040}{6+m} \int_{x=0}^1 x^{3+k} (1-x)^{6+m} \, dx \\ &= \frac{5040}{6+m} \frac{\Gamma(4+k)\Gamma(7+m)}{\Gamma(11+k+m)} \\ &= \frac{7! (k+3)!(m+5)!}{(k+m+10)!}, \end{align*}$$ where I have used the beta integral identity (as found in the density of a beta distribution) $$\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)} \int_{x=0}^1 x^{a-1} (1-x)^{b-1} \, dx = 1.$$ From this, we are able to immediately compute all relevant moments through appropriate selection of $k$, $m$:
$$\operatorname{E}[XY] = \frac{7!4!6!}{12!} = \frac{2}{11} \\ \operatorname{E}[X] = \frac{7!4!5!}{11!} = \frac{4}{11} \\ \operatorname{E}[Y] = \frac{7!3!6!}{11!} = \frac{6}{11} \\ \operatorname{E}[X^2] = \frac{7!5!5!}{12!} = \frac{5}{33} \\ \operatorname{E}[Y^2] = \frac{7!3!7!}{12!} = \frac{7}{22}. $$ The remainder of the calculation I leave as a straightforward exercise.
Note that this method of solution works because the joint PDF is of the form $C x^p y^q$ for suitable $C, p, q$, and is readily amenable to such an approach. We would not perform such a generalization in cases where the resulting integrand would become needlessly complicated.