We have that $$X \sim\mathcal U(0,1)$$ and given X = x then $$Y\mid X=x \sim \mathcal{Binomial}(n=5,p=x)$$.
Given that $E[Y] = 5/2$, I am asked to calculate $E[Y^2]$.
I have decided to use the computational formula for variance, to get the expression: $$E[Y^2] = E[Y]^2 + V[Y]$$.
Where $$V[Y] = E[(Y-E[Y])^2]$$.
I have decided to use LOUTS together with the unconditional expectation $$E[Y] = \int_{0}^{1} E[g(y)|X=x] *f_X(x) dx$$ Such that $$E[Y] = \int_{0}^{1} E[(Y-E[Y])^2|X=x] *f_X(x) dx = \int_{0}^{1} E[(Y-E[Y])|X=x] dx$$
Now, the term $E[(Y-E[Y])^2|X=x]$ is simply the variance of a binomial distribution with parameters n = 5 and p = x.
So the full expression is:
$$V[Y] = \int_{0}^{1} 5*x(1-x) dx = 5/6$$.
Combining this, I get:
$$E[Y^2] = 75/12 + 10/12 = 85/12$$, which is wrong.
Another method would be to use, the fact that:
$$E[E[Y^2|X]] = E[Y^2]$$, which gives me the correct answer. However, I would like to know, why my original method didn't work? Am I not using LOTUS correctly or have I misunderstood something and started calculating a different statistic than $E[Y^2]$?
The fundamental reason why you cannot say $$\operatorname{Var}[Y] = \int_{x=0}^1 \operatorname{Var}[Y \mid X = x]f_X(x) \, dx,$$ is because this doesn't take into account an additional source of variation in $Y$ as a result of the variability in $X$. All that the integral on the RHS is doing is calculating a weighted average of the conditional variance of $Y$ given $X$, weighted by the density of $X$. As such, what it is really calculating is $$\operatorname{E}[\operatorname{Var}[Y \mid X]] = \int_{x=0}^1 \operatorname{Var}[Y \mid X = x]f_X(x) \, dx. \tag{1}$$
However, the law of total variance states that $$\operatorname{Var}[Y] = \operatorname{E}[\operatorname{Var}[Y \mid X]] + \operatorname{Var}[\operatorname{E}[Y \mid X]]. \tag{2}$$ The missing second term is the variance of the conditional expectation of $Y$ given $X$. A proof of this law follows:
$$\begin{align} \operatorname{Var}[Y] &= \operatorname{E}\left[(Y - \operatorname{E}[Y])^2\right] \\ &= \operatorname{E}\left[Y^2\right] - \operatorname{E}[Y]^2 \\ &= \operatorname{E}\left[\operatorname{E}[Y^2 \mid X]\right] - \operatorname{E}[\operatorname{E}[Y \mid X]]^2 \\ &= \operatorname{E}\left[\operatorname{E}[Y^2 \mid X] - \operatorname{E}[Y \mid X]^2 + \operatorname{E}[Y \mid X]^2\right] - \operatorname{E}[\operatorname{E}[Y \mid X]]^2 \\ &= \operatorname{E}[\operatorname{Var}[Y \mid X]] + \operatorname{E}\left[\operatorname{E}[Y \mid X]^2\right] - \operatorname{E}[\operatorname{E}[Y \mid X]]^2 \\ &= \operatorname{E}[\operatorname{Var}[Y \mid X]] + \operatorname{Var}[\operatorname{E}[Y \mid X]]. \tag{3} \end{align}$$
What is this second term in your case? It is $$\operatorname{Var}[\operatorname{E}[Y \mid X]] = \int_{x=0}^1 \left(\operatorname{E}[Y \mid X = x] - \operatorname{E}[\operatorname{E}[Y \mid X]]\right)^2 f_X(x) \, dx. \tag{4}$$ Notice that the double expectation $\operatorname{E}[\operatorname{E}[Y \mid X]]$ in the integrand is not a function of $x$. It is a constant, and in fact, is simply $\operatorname{E}[Y]$.
So after having computed $\operatorname{E}[\operatorname{Var}[Y \mid X]] = \frac{5}{6}$, from $(1)$, we can compute from $(4)$ that $$\operatorname{E}[Y \mid X] = 5X, \quad \operatorname{E}[Y] = \operatorname{E}[\operatorname{E}[Y \mid X]] = \operatorname{E}[5X] = \frac{5}{2},$$ hence $$\operatorname{Var}[\operatorname{E}[Y \mid X]] = \int_{x=0}^1 \left(5x - \tfrac{5}{2}\right)^2 \, dx = \frac{25}{12}. \tag{5}$$ Therefore $$\operatorname{Var}[Y] = \frac{5}{6} + \frac{25}{12} = \frac{35}{12}, \tag{6}$$ hence $$\operatorname{E}[Y^2] = \operatorname{Var}[Y] + \operatorname{E}[Y]^2 = \frac{35}{12} + \frac{25}{4} = \frac{55}{6}. \tag{7}$$
The same answer could also have been obtained without the law of total variance by writing $$\begin{align} \operatorname{E}[Y^2] &= \operatorname{E}\left[\operatorname{E}[Y^2 \mid X]\right] \\ &= \operatorname{E}\left[\operatorname{Var}[Y \mid X] + \operatorname{E}[Y \mid X]^2\right] \\ &= \operatorname{E}\left[5X(1-X) + (5X)^2\right] \\ &= \operatorname{E}[20X^2 + 5X] \\ &= \operatorname{E}[20(X - 1/2)^2 + (20X - 5) + 5X] \\ &= 20 \operatorname{E}[(X - \operatorname{E}[X])^2] + 25 \operatorname{E}[X] - 5 \\ &= 20 \operatorname{Var}[X] + \frac{25}{2} - 5 \\ &= \frac{20}{12} + \frac{25}{2} - 5 \\ &= \frac{55}{6}. \tag{8} \end{align}$$