$\int ^{5}_{1}\int ^{\sqrt{x-1}}_{y=0}{ye^{(x-1)^2}}dydx$
First, I changed the integration from $x$ to $y$.
However, I get $\int ^{5}_{1}\dfrac{\left( x-1\right) }{2}e^{\left( x-1\right) ^{2}}dx$ and I get $\int \dfrac{1}{4}\sqrt{u}e^{u}du$.
I can't go from here. How to integrate this?
You are making the substitution wrong. If $u=(x-1)^2$, then $du=2(x-1)dx$, and the factor $(x-1)/2$ becomes simply $1/4$, not $\frac14\sqrt u$.