Given that $E(u) = \int \mathcal{L}(u,u') dx$, \begin{align} \frac{dE(u)}{du} \bigg |_{h} &= lim_{\epsilon \to 0} \frac{1}{\epsilon}(E(u) + \epsilon h) - E(u))\\ &= \lim_{\epsilon \to 0} \frac{1}{\epsilon} \int \big((\mathcal{L}(u + \epsilon h ,u' + \epsilon h') - \mathcal{L}(u,u')\big) dx \\ &= \lim_{\epsilon \to 0} \frac{1}{\epsilon} \int \big(\mathcal{L}(u,u') + \frac{\partial \mathcal{L}}{\partial u} \epsilon h + \frac{\partial \mathcal{L}}{\partial u'} \epsilon h' + o(\epsilon ^{2})) - \mathcal{L}(u,u')\big) dx \\ &= \int \big( \frac{\partial L}{\partial u} h + \frac{\partial \mathcal{L}}{\partial u'}h' \big) dx\\ &=\int \big( \frac{\partial L}{\partial u} h - \frac{d}{dx} \frac{\partial \mathcal{L}}{\partial u'}h \big) dx\\ &= \int \big( \frac{\partial L}{\partial u} - \frac{d}{dx} \frac{\partial \mathcal{L}}{\partial u'} \big) h(x) dx \end{align}
How did you from step 4 step 5? \begin{align} &= \int \big( \frac{\partial L}{\partial u} h + \frac{\partial \mathcal{L}}{\partial u'}h' \big) dx\\ &=\int \big( \frac{\partial L}{\partial u} h - \frac{d}{dx} \frac{\partial \mathcal{L}}{\partial u'}h \big) dx \end{align}
It was mentioned to be integration by parts. But I can't seem to figure it out.
Note that, by the chain rule $$ \frac{d}{dx}\Big(\frac{\partial\mathcal{L}}{\partial u'}h\Big)=\frac{d}{dx}\Big(\frac{\partial\mathcal{L}}{\partial u'}\Big)h+\frac{\partial\mathcal{L}}{\partial u'}h'. $$ And so acting with $\int dx$ on both sides and using $$ \frac{\partial\mathcal{L}}{\partial u'}h\Big|_{\text{boundary}}=0 $$ you will find $$\frac{\partial\mathcal{L}}{\partial u'}h'=-\frac{d}{dx}\Big(\frac{\partial\mathcal{L}}{\partial u'}\Big)h$$ as always.