The random variables $X$ and $Y$ have joint PDF $$f(x,y)= \frac{\Gamma(a+b+c)}{\Gamma(a)\Gamma(b)\Gamma(c)}x^{a-1}y^{b-1}(1-x-y)^{c-1} $$ where $0 \leq x \leq 1 , 0 \leq y \leq 1, x+y < 1 $ where $a , b,c$ are positive real. Note that $\Gamma$ is the gamma function.
Find the correlation coefficient of $X$ and $Y$.
The final answer is supposed to be $$- \sqrt{\frac{ab}{(b+c)(a+c)}}$$
I first tried to find $$E(X)=\frac{\Gamma(a+b+c)}{\Gamma(a)\Gamma(b)\Gamma(c)} \int_0^1\int_0^1 x^{a}y^{b-1}(1-x-y)^{c-1} dxdy.$$ But I have no clue what the integral on the RHS is.
$\begin{align} \text{You domain is:} \\ \left.\begin{array}{c} 0\leq x\leq 1 \\ 0\leq y\leq 1 \\ x+y< 1 \end{array}\right\} & \equiv x\in [0,1], y\in [0, x) \\[2ex]\text{So the expectation is:} \\[2ex] \mathsf E[X] & = \frac{\Gamma(a+b+c)}{\Gamma(a)\Gamma(b)\Gamma(c)}\int_0^1 \int_0^{1-x} x\; f_{X,Y}(x,y)\,\operatorname{d}y \operatorname{d} x \\[1ex] & = \frac{\Gamma(a+b+c)}{\Gamma(a)\Gamma(b)\Gamma(c)}\int_0^1 \int_0^{1-x} x^a y^{b-1}(1-x-y)^{c-1}\operatorname{d}y\operatorname{d}x \\[2ex] & = \frac{\Gamma(a+b+c)}{\Gamma(a)\Gamma(b)\Gamma(c)}\int_0^1 x^a \int_0^{1-x}y^{b-1}((1-x)-y)^{c-1}\operatorname{d}y\operatorname{d}x \\[4ex] \text{Then use the definition of a p.d.f.} \\[2ex] 1 & = \frac{\Gamma(a+b+c)}{\Gamma(a)\Gamma(b)\Gamma(c)}\int_0^1 x^{a-1} \int_0^{1-x} y^{b-1}(1-x-y)^{c-1}\operatorname{d}y\operatorname{d}x \\[2ex]\text{By substitution} \\[2ex] \therefore \mathsf E[X] & = \frac{\Gamma(a+b+c)}{\Gamma(a)\Gamma(b)\Gamma(c)}\frac{\Gamma(1+a)\Gamma(b)\Gamma(c)}{\Gamma(1+a+b+c)} \\ & = \frac{a}{a+b+c} \\[2ex]\text{And so on...} \end{align}$