I am reading Montgomery book for DOE. this is part of his proofs. I could figure out what he does until this last line, how did he get the second line from the first line. I know E(x) formulas for continuous and discrete functions, but I still have difficulty understanding this.
There is one squared summation and one squared double summation, for which you will need the appropriate formulae.
The formulae are as follows: $\underbrace{\left(\sum_{i=1}^n a_i\right)^2}_{\text{n$^2$ terms}}=\underbrace {\sum a_i^2}_{\text{n terms}}+\underbrace {\sum_{i\lt j} a_ia_j}_{\text{$\frac{n(n-1)}2$ terms}}+\underbrace{\sum_{i>j}a_ia_j}_{\text{$\frac{n(n-1)} 2$ terms}}=\sum a_i^2+\underbrace{\sum_{i\ne j}a_ia_j}_{\text{n(n-1) terms}}$.
For the double sum, the square is $\underbrace{\left(\sum_{j=1}^m\sum_{i=1}^n a_{ij}\right)^2}_{\text{$n^2m^2$ terms}}=\underbrace{\sum_{j=1}^m\sum_{i=1}^n a_{ij}^2}_{\text{nm terms}}+\underbrace{\sum_{l=1}^m\sum_{j=1}^m\sum_{i\ne k} a_{ij}a_{kl}}_{\text{$m^2n(n-1)$ terms}}+\underbrace{\sum_{i=1}^n\sum_{j\ne l}a_{ij}a_{il}}_{\text{mn(m-1) terms}}$
To get you started, the left sum is $E\left[\sum_{i=1}^a\frac 1 {n_i}\left(\sum_{j=1}^{n_i}(\mu+\tau+\epsilon_{ij})\right)^2\right]=E\sum_{i=1}^a\frac 1 {n_i}\left[\sum_{j=1}^{n_i}(\mu+\tau_i+\epsilon_{ij})^2+\sum_{j\ne k}(\mu+\tau_i+\epsilon_{ij})(\mu+\tau_i+\epsilon_{ik})\right]=...$
Eventually you'll need to use the fact that $E(\epsilon)=0,E(\epsilon^2)=var(\epsilon)-[E(\epsilon)]^2=\sigma^2$, and further, that $\mu$ is constant, also don't know what "$\sum \tau_i$" is.