I want to compute $$\rho(X,Y)=\frac{\text{Cov}[X,Y]}{\sqrt{\text{Var}[X]\text{Var}[Y]}}=\frac{E[XY]-E[X]E[Y]}{\sqrt{\text{Var}[X]\text{Var}[Y]}}.$$
I know that $E[X]=1/2$, $\text{Var}[X]=1/12$. To complete this problem I need to find $E[Y]$ and $E[XY]$. For $E[Y]$ the mean should be is $(0+x)/2=x/2$ but the book says $1/4$, why?
For $E[XY]$ I have that
$$E[XY]=\int_0^1\int_0^xxy\cdot f(x,y) \ dydx,$$
but how do I find the joint distribution $f(x,y)?$ Sure if they stated that $X$ and $Y$ are independent then I could use that $f_X(x)f_Y(x)=1\cdot 1/x=1/x$ but I can't assume independence.
What you found was: $$\mathbb E[Y\mid X=x]=\frac12 x$$
That means that: $$\mathbb E[Y|X]=\frac12 X$$ and applying the general rule: $$\mathbb EY=\mathbb E[\mathbb E[Y|X]]$$ we then find that: $$\mathbb EY=\mathbb E\frac12X=\frac12\mathbb EX=\frac12\frac12=\frac14$$
Applying that same rule in this form:$$\mathbb E[XY]=\mathbb E[\mathbb E[XY\mid X]]$$and exploiting that: $$\mathbb E[XY\mid X]=X\mathbb E[Y|X]$$ we find: $$\mathbb E[XY]=\mathbb E[\mathbb E[XY\mid X]]=\mathbb E[X\mathbb E[Y\mid X]]=\mathbb E\frac12X^2=\frac12\mathbb EX^2=\frac12\frac13=\frac16$$