if $F(x)=\int_0^x(x-t)^nu(t)dt$ then find $F^{(n+1)}(x)$
from Leibniz rule $a(x)=0,b(x)=x,a'(x)=0,b'(x)=1$ and $G'(x)=n(x-t)^{n-1}\\$
so $F^{(1)}(x)=n\int_0^x(x-t)^{n-1}u(t)dt$ and $F^{(n)}(x)=n.(n-1).(n-2)...1\int_0^x(x-t)^{n-n}u(t)dt=n!\int_0^xu(t)dt$. How can I continue from here? is it $F^{(n+1)}(x)=0$ because applying Leibniz rule one more time gives $G'(x)=0$?
Leibniz's rule tells us that we have
$$\frac{d}{dx}\big(\int_{a(x)}^{b(x)}f(x,t)dt\big)=f(x,b(x))\frac{d}{dx}b(x)-f(x,a(x))\frac{d}{dx}a(x)+\int_{a(x)}^{(b(x)}\frac{\partial }{\partial x}f(x,t)dt.$$
So if $F(x)=\int_{0}^{x}(x-t)^{n}u(t)dt$ then
$$F^{(k)}(x)=n(n-1)\cdot\cdot\cdot(n-k+1)\int_{0}^{x}(x-t)^{n-k}u(t)dt$$
for $1\leq k\leq n$. You then correctly found that $F^{(n)}(x)=n!\int_{0}^{x}u(t)dt$. Note that since $a(x)=0$, $b(x)=x$ we have $a'(x)=0$, but $b'(x)=1$ so that the first doesn't vanish. Applying Leibniz's rule to $F^{(n)}(x)$ we obtain
$$F^{(n+1)}(x)=n!u(x)\frac{d}{dx}b(x)-n!u(0)\frac{d}{dx}a(x)+n!\int_{0}^{x}\frac{\partial}{\partial x}u(t)dt$$ $$=n!u(x).$$