I need to calculate $\operatorname{E}( X_2 \mid X_1=x, Y=y)$, where $Y=\max\{X_2,X_3\}$ and joint density of $X_1$, $X_2$ and $X_3$ is given by:
$$f_{X_1,X_2,X_3}(x,y,z)=\frac{24}{25}(1+\frac{xyz}{3})$$ $x,y,z\in [0,1]$.
I completely don't know how can I tackle this problem. Please help me.
One way to approach it is as follows. Firstly, $Y$ is either $X_2$ or $X_3$ and with equal probability because $f_{X_1,X_2,X_3}$ is symmetric wrt those two variables. So we begin by conditioning on the event that $X_2=Y$:
\begin{eqnarray*} E(X_2\mid X_1=x, Y=y) &=& E(X_2\mid X_1=x, Y=y, X_2=y)P(X_2=y\mid X_1=x, Y=y) + \\ && E(X_2\mid X_1=x, Y=y, X_2\lt y)P(X_2\lt y\mid X_1=x, Y=y). \qquad\text{(1)} \end{eqnarray*}
Now to evaluate these components. That $E(X_2\mid X_1=x, Y=y, X_2=y) = y\;$ is easy. And again, by symmetry of $f_{X_1,X_2,X_3}$, we have
$$P(X_2=y\mid X_1=x, Y=y) = \dfrac{1}{2} = P(X_2\lt y\mid X_1=x, Y=y).$$
It remains to find $E(X_2\mid X_1=x, Y=y, X_2\lt y)$ and we do so by firstly finding $P(X_2 = u\mid X_1=x, Y=y, X_2\lt y)$ for any $u \lt y$:
\begin{eqnarray*} P(X_2 = u\mid X_1=x, Y=y, X_2\lt y) &=& \dfrac{P(X_2 = u, X_1=x, Y=y, X_2\lt y)}{P(X_1=x, Y=y, X_2\lt y)} \\ && \\ &=& \dfrac{P(X_2 = u, X_1=x, X_3=y)}{P(X_1=x, X_3=y, X_2\lt y)}. \\ && \\ P(X_2 = u, X_1=x, X_3=y) &=& f_{X_1,X_2,X_3}(x,u,y) \\ &=& \dfrac{24}{25}\left( 1 + \dfrac{xuy}{3} \right). \\ && \\ P(X_1=x, X_3=y, X_2\lt y) &=& \int_{v=0}^{y} \dfrac{24}{25}\left( 1 + \dfrac{xvy}{3} \right)\;dv \\ &=& \dfrac{24}{25}\left[ v + \dfrac{xv^2y}{6} \right]_{v=0}^{y} \\ &=& \dfrac{24}{25}\left( y + \dfrac{xy^3}{6} \right). \\ && \\ \therefore\quad P(X_2 = u\mid X_1=x, Y=y, X_2\lt y) &=& \dfrac{\dfrac{24}{25}\left( 1 + \dfrac{xuy}{3} \right)}{\dfrac{24}{25}\left( y + \dfrac{xy^3}{6} \right)} \\ && \\ &=& \dfrac{6 + 2xuy}{6y + xy^3}. \\ && \\ \therefore\quad E(X_2\mid X_1=x, Y=y, X_2\lt y) &=& \int_{u=0}^{y} u P(X_2 = u\mid X_1=x, Y=y, X_2\lt y)\;du \\ && \\ &=& \int_{u=0}^{y} \dfrac{6u + 2xu^2y}{6y + xy^3}\;du \\ && \\ &=& \left[ \dfrac{3u^2 + \dfrac{2xu^3y}{3}}{6y + xy^3} \right]_{u=0}^{y} \\ &=& \dfrac{3y^2 + \dfrac{2xy^4}{3}}{6y + xy^3} \\ &=& \dfrac{9y + 2xy^3}{18 + 3xy^2}.\end{eqnarray*}
Returning to equation $(1)$, we now have,
\begin{eqnarray*} E(X_2\mid X_1=x, Y=y) &=& \dfrac{1}{2} y + \dfrac{1}{2} \dfrac{9y + 2xy^3}{18 + 3xy^2} \\ && \\ &=& \dfrac{18y + 3xy^3 + 9y + 2xy^3}{36 + 6xy^2} \\ && \\ &=& \dfrac{27y + 5xy^3}{36 + 6xy^2}. \end{eqnarray*}