$A, B, C$ are three variables, uniform distribution on $[0,1]$. Given that $C<0.8$, and $C>A>B$, find the expected value of $C$.
I try to solve this conditional prob question, while not sure that I'm right or wrong. My answer is:
$P(C<0.8, C>A>B)=(0.8)^3\cdot1/6$. $P(C< x)=x^3\cdot1/6$, so the probability density function of $x$ is $ \frac{\text{3x}^2}{0.8^3} $, so the result is $ \int\limits_0^{0.8}{\frac{\text{3x}^2}{0.8^3}}\text{xdx}=\frac{3}{4}\cdot 0.8=0.6 $
Your answer is correct but unfortunately I can't really make sense of your method. Here is how one might compute the answer systematically.
We have independent variables $X_1, X_2, X_3 \sim U(0,1)$ and $Y = \max(X_1, X_2, X_3).$ We wish to compute $\mathbb{E}[Y \ | \ Y < 0.8].$ In general, for any an event $A$ with non-zero probability we have $$\mathbb{E}[Y \ |\ A] = \frac{ \mathbb{E}[\mathbb{1}_A Y]}{\mathbb{P}(A)}$$
where $1_A$ is the indicator function of the event $A$ (that is, it is equal to $1$ if $A$ occurs, $0$ otherwise).
We have $\mathbb{P}(Y \leq y) = \mathbb{P}(X_1 \leq y, X_2 \leq y, X_3 \leq y) = y^3$ for $y\in [0,1],$ so $\mathbb{P}(Y < 0.8) = 0.8^3$ and the density of $Y$ is given by $p(y) = \mathbb{1}_{y\in [0,1]} 3y^2.$ We also have
$$ \mathbb{E}[\mathbb{1}_A Y] = \int^1_0 \mathbb{1}_{y < 0.8} \ y \ p(y) dy = \int^{0.8}_0 y \cdot 3y^2 dy = \frac{3}{4} \cdot 0.8^4$$
which confirms your computation that $$\mathbb{E}[Y\ | \ Y < 0.8] = \frac{3}{4} \cdot 0.8 = \frac{3}{5}.$$