let be $Z$ random variable, $Y_i$ iid random variable, $E(Y_i)=\mu$, $\operatorname{var}(Y_i)=\sigma^2$; $Z$ and $Y_i$ are independent. $X=\sum_{i=1,\ldots,Z} Y_i$
show that $E(X^2\mid Z)=\mu^2Z^2+\sigma^2Z$.
I compute $E(X^2\mid Z=n)=nE(Y^2)+n\mu^2$ and i remplace the quantity $E(Y^2)$ by $\operatorname{var}(Y)+(E(Y))^2=\sigma^2+\mu^2$, but i dont get the result.
Some help
Assuming $X= \sum\limits_{i=1}^Z Y_i$, then using $\sigma^2 = \mathsf E(Y_i^2)-\mu^2$ gives the desired result.
$$\begin{align} \mathsf E(X^2\mid Z) & = \mathsf E((\sum_{i=1}^Z Y_i)^2 \mid Z) \\[1ex] & = \sum_{i=1}^Z\mathsf E(Y_i^2)+\raise{1ex}{\mathop{\sum_{i=1}^Z \sum_{j=1}^Z}_{i\neq j}}\;\mathsf E(Y_i)\mathsf E(Y_j) & \star \\[1ex] & = Z (\sigma^2+\mu^2)+Z(Z-1)\mu^2 \\[1ex] & = Z\sigma^2+Z^2\mu^2\end{align}$$
Your main error seems to be in miscounting $Y_iY_j$ terms in the expansion at $\star$.