I was given a function $f(x,y)=1120x^{3}y^{3}$ for $0\leq x, 0\leq y, $ and $ x+y \leq 1$
I went ahead and calculated the marginal PDF's for X and Y
$f_{X}(x) = \int_{-\infty}^{\infty} f_{x,y}(x,y)$ dy = $1120\int_{0}^{1-x} x^{3}y^{3}dy$ = $280x^{3}(1-x)^{4}$
Similarly for $f_{Y}(y) = 280(1-y)^{4}y^{3}$
I was then tasked with calculating Corr(x,y)
Im aware r=Corr(x,y)= ${Cov(x,y)}\over{(Var(x)Var(y))^{1/2}}$
I've went ahead and calculated the Variances by making use of the suitable formula of $E[x^{2}]-(E[x])^{2}$ but in integral, with E[x] using Marginal function of x, and E[y] using marginal function for Y where necessary.
Can somebody give me a pointer how to calculate Corr(x,y). I have a large integral formula in mind but i think i may be over complicating it.
Let us generalize a little. Define the density as $f(x,y)=cx^py^p$, where $p\geq 1$ and $c$ is the normalizing constant. This density is defined in the region $R=\{(x,y):x\geq 0,y\geq 0,x+y\leq 1\}$ (of course $f(x,y):=0$ outside $R$).
Let $g$ be a function defined on $R$. To calculate $E[g(X,Y)]$ one uses $$ E[g(X,Y)]=\int_Rg(x,y)f(x,y)\,dxdy,\quad(\text{here, the integral is double}). $$ For example, if $g(x)=x^2$ you get $E[g(X,Y)]=E[X^2]$.
I affirm that all you need to do is adapt the function $g$ in each calculation and complete a Beta density. For example, if $g(x,y)=xy$, then \begin{align*} E[g(X,Y)] &= \int_Rxyf(x,y)\,dxdy\\ &=c\int_0^1\int_0^{y-1}x^{p+1}y^{p+1}\,dxdy\\ &=\dfrac{c}{p+2}\int_0^1(1-y)^{p+2}y^{p+1}\,dy. \end{align*} Here is the trick: the function $y\mapsto (1-y)^{p+2}y^{p+1}$ is a Beta density except for some missing constant terms. Even more, is being integrated in $(0,1)$, so if you put those missing constant terms, the integral will be one. Define as $K>0$, the missing constant terms in the previous example. Then, $$ E[g(X,Y)] = \dfrac{c}{K(p+2)}\int_0^1K(1-y)^{p+2}y^{p+1}\,dy = \dfrac{c}{K(p+2)}. $$ You just need to see what is $K$: https://en.wikipedia.org/wiki/Beta_distribution.
Let $g(X,Y)=X^nY^m$, $n,m\geq 0$. Then, by the above display, \begin{align*} E[X^nY^m] &= c\int_0^1\in_0^{1-y}x^{p+n}y^{p+m}\,dxdy\\ &= \dfrac{c}{p+n+1}\int_0^1(1-y)^{p+n+1}y^{p+m}\,dy\\ &=\dfrac{c}{p+n+1}\dfrac{\Gamma(p+n+2)\Gamma(p+m+1)}{\Gamma(2p+n+m+3)}\\ &=\dfrac{c(p+n)!(p+m)!}{(2p+n+m+2)!}. \end{align*}
To obtain each term of $Corr(X,Y)$ just substitute the values of your problem, i.e., $p=3$, $c=1120$, and vary over $n$ an $m$.