I want to compute this integral $$I=\sum_{k=0}^{+\infty} H^k \int_0^t \frac{ u^{(k+1)\alpha-1}}{\Gamma(1-\alpha)\Gamma((k+1)\alpha)}(t-u)^{-\alpha}du$$ With $\alpha\in(0,1)$, $H\in\mathbb{R}^{m\times m}$, $t>0$ and $\Gamma$ is a gamma function $(n!=\Gamma(n+1))$.
So I said if I can put $v=(t-u)^{-\alpha}$, then,
If $u=0\Rightarrow v=t^{-\alpha}$. $\hspace{2em}$ If $u=t \Rightarrow v=0$.
The problem is as follows:
$$v=(t-u)^{-\alpha} \quad \Rightarrow \quad dv=\alpha(t-u)^{-\alpha-1}du\quad\Rightarrow\quad du=\frac{1}{\alpha} v^{\alpha(\alpha+1)}dv $$ or $$v^\alpha=(t-u)\quad \Rightarrow \quad d(v^\alpha)=d(t-u)\quad\Rightarrow\quad du=-\alpha v^{\alpha-1}dv $$ Thank you to share with me your knowledge. best regards;
Your problem results from a simple mistake: Note that $(x^{-\alpha})^\alpha = x$ is wrong and you have to use $(x^{-\alpha})^{-1/\alpha} = x$ instead. Then you will find $$ v = (t-u)^{-\alpha} ~ \Rightarrow ~ \mathrm{d} v = \alpha (t-u)^{-\alpha -1} \mathrm{d} u ~ \Rightarrow ~ \mathrm{d} u = \frac{1}{\alpha} (t-u)^{\alpha + 1} \mathrm{d}v ~ \Rightarrow ~ \mathrm{d} u = \frac{1}{\alpha} v^{\color{red}{-} \frac{\alpha+1}{\color{red}{\alpha}}} \mathrm{d} v $$ and $$ v = (t-u)^{-\alpha} ~ \Rightarrow ~ v^{\color{red}{-\frac{1}{\alpha}}} = t - u ~ \Rightarrow ~ -\frac{1}{\alpha} v^{-\frac{1}{\alpha} - 1} \mathrm{d} v = - \mathrm{d} u ~ \Rightarrow ~ \mathrm{d} u = \frac{1}{\alpha} v^{-\frac{\alpha + 1}{\alpha}} \mathrm{d} v \, . $$ Now both ways lead to the same, correct result.
One of your new limits of integration is not correct as well: If you put $u = t$ in $v = (t-u)^{- \alpha}$, you get $v = \infty$ and not $v = 0$ .
As for the integral itself, using Shashi's substitution is a lot easier. If you set $u = t x$, you will obtain \begin{align} \int \limits_0^t u^{(k+1)\alpha - 1} (t-u)^{-\alpha} \, \mathrm{d} u &= t^{\alpha k} \int \limits_0^1 x^{(k+1)\alpha - 1} (1-x)^{-\alpha} \, \mathrm{d} x = t^{\alpha k} \operatorname{B} ((k+1) \alpha, 1 - \alpha) \\ &= t^{\alpha k} \frac{\Gamma ((k+1) \alpha) \Gamma(1-\alpha)}{\Gamma(1 + \alpha k)} \end{align} with the beta function $\operatorname{B}$. Your series now becomes $$ I = \sum \limits_{k = 0}^\infty \frac{(t^{\alpha} H)^k}{\Gamma(1+\alpha k)} = \operatorname{E}_{\alpha,1} (t^\alpha H) \, .$$ $\operatorname{E}_{\alpha,\beta}$ is called Mittag-Leffler function.