My approach to compute $\int_0^1 xe^{2x}\, dx$ is via integration by parts by setting $$ u'(x)=x^2,\qquad v(x)=e^{2x} $$ which gives me $$ \int_0^1xe^{2x}\, dx=\frac{e^2}{2}-\int_0^1x^2e^{2x}\, dx. $$ Then, doinf again integration by parts by setting $$ u'(x)=x^2,\qquad v(x)=e^{2x}, $$ I get $$ \int_0^1 x^2e^{2x}\, dx=\frac{e^3}{3}-\frac{2}{3}\int_0^1x^3e^{2x}\, dx $$ which gives me $$ \int_0^1 xe^{2x}\, dx=\frac{e^2}{6}+\frac{2}{3}\int_0^1 x^3e^{2x}\, dx. $$
I guess now I have to do another integration by parts and so on but this won't come to an end. So what am I doing wrong?
There is a rule of thumb, known as the ILATE or the LIATE rule, where
I - Inverse trigonometric, L - Logarithmic, A - Algerbraic, T - Trigonometric, E - Exponential,
It suggests that the type of function that appears first in the acronym be taken as $v(x)$ and the other as $u'(x)$.
In this case, $x$ is algebraic and $e^{2x}$ is exponential. This suggests a reverse order from what you chose.
$$I=\int_0^1xe^{2x}dx=\frac{e^2}{2}-\frac12\int_0^1e^{2x}dx$$
which can be easily simplified.
Also see this question and this question.