We know that (X,Y) have density $g(x,y)=6xy$ for $x,y \in T$ where $T=((x,y); 0 \leq x \leq 1, 0 \leq y\leq \sqrt x )$.
The task is to calculate $E(X|Y=y)$ and $E[YE(X|Y)]$.
Now I find joint density of x and y
$g(x|y)=\frac{6xy}{\int_{0}^{1} 6xy dx}=\frac{6xy}{3y}=2x$
And $E(X|Y=y)=\int_{0}^{1} 2x^2 dx=\frac{2}{3}$
Is this solution corect? What can be done with the $E[YE(X|Y)]$?
First we have for $y\in (0,1)$ $$g(x\vert y) = 1_{\{y^2 \leq x \leq 1\}}\frac{6xy}{\int_{y^2}^1 6xydx} = 1_{\{y^2 \leq x \leq 1\}}\frac{6xy}{3y - 3y^5} = 1_{\{y^2 \leq x \leq 1\}}\frac{2x}{1 - y^4}$$ resulting in $$E[X\vert Y= y] = \frac 1 {1-y^4} \int_{y^2}^1 x 2x dx = \frac{2(1 - y^6)}{3(1 - y^4)}$$
Further $$E[YE[X\vert Y]] = E[XY] = 6 \int_0^1 y^2 \int_{y^2}^1 x^2 dx dy = 2 \int_0^1 y^2 (1 - y^6) dy \\= 2\int_0^1 y^2dy - 2 \int_0^1y^8dy = \frac 2 3 - \frac2 9 = \frac 4 9$$