Its my solution for this question
$$ f(x,y)= \begin{cases} x^2+\frac{xy}{3}, & 0<x<1, \quad 0<y<2 \\ 0, &\textrm{otherwise}\\ \end{cases}$$ then calculate $\rho(X,Y)$ ?
Note: $\rho$ is the Rho. I solved this question but I find the $\rho(X,Y) = -0.051$ but my teacher says it's the wrong answer? How can I solve this problem correctly.
Well, your calculations are correct, but the final answer is not equal to -0.5132 . That is only an approximation.
$\begin{align} f(x,y)&=(x^2+xy/3)\mathbf 1_{0\leq x\leq 1, 0\leq y\leq 1} \\[2ex]\mathsf E(X) &= \int_0^1\int_0^2 x f(x,y)~\mathrm dy~\mathrm d x \\[1ex] &= 13/18 \\[2ex]\mathsf E(X^2) &= \int_0^1\int_0^2 x^2 f(x,y)~\mathrm dy~\mathrm d x \\[1ex] &= 17/30 \\[2ex]\mathsf {Var}(X) &= \mathsf E(X^2)-\mathsf E(X)^2 \\[1ex] &= 73/1620 \\[2ex]\mathsf E(Y) &= \int_0^1\int_0^2 y f(x,y)~\mathrm dy~\mathrm d x \\[1ex] &= 10/9 \\[2ex]\mathsf E(Y^2) &= \int_0^1\int_0^2 y^2 f(x,y)~\mathrm dy~\mathrm d x \\[1ex] &= 14/9 \\[2ex]\mathsf {Var}(Y) &= \mathsf E(Y^2)-\mathsf E(Y)^2 \\[1ex] &= 26/81 \\[2ex]\mathsf E(XY) &= \int_0^1\int_0^2 xy f(x,y)~\mathrm dy~\mathrm d x \\[1ex] &= 43/54 \\[2ex]\mathsf {Cov}(X,Y) &= \mathsf E(XY)-\mathsf E(X)\mathsf E(Y) \\[1ex] &= -1/162 \\[2ex]\mathsf{Corr}(X,Y) &= (-1/162)/(\surd[(73/1620)(26/81)]) \\[1ex]&=-\surd(5/1898) \\[1ex]&\approx -0.051325938368630364195576486370191526370379503588767652076654564... \end{align}$