I can't quite follow the solution for the part highlighted.
Writing $U(x,t)=\phi(x)e^{\lambda t}$ casts $(2)$ into $$\lambda\phi=\phi^{\prime\prime}-(\overline u\phi)^\prime,\tag3 \\ \phi(0)=\phi(1)=0.$$ Now multiply $(3)$ by $\phi$ and integrate between $0$ and $1$ to find (using integration by parts) $$\lambda\int_0^1\phi^2dx=[\phi\phi^\prime]^1_0-\int_0^1(\phi^\prime)^2dx-[\phi\overline u\phi]^1_0+\color{green}{\boxed{\color{black}{\displaystyle\int_0^1\overline u\phi\phi^\prime dx.}}}$$ The boundary terms are zero due to the conditions on $\phi$, and a further integration by parts of the last term on the rhs yields $$\lambda\int_0^1\phi^2dx=-\int_0^1(\phi^\prime)^2dx+\color{green}{\boxed{\color{black}{\displaystyle\left[\frac12\phi^2\overline {u}\right]_0^1-\dfrac12\int_0^1\overline u^\prime\phi^2dx .}}}$$
Let $dv=\phi\phi' \; dx$ and use $\frac{d}{dx}(\phi^2)=2\phi\phi'$.