Letting $t=xy\;\color{blue}{\Rightarrow}\;dt=x\ dy$, the integral turns out to be
$$
\frac1{\Gamma(r)}\int_0^1(x-xy)^{\alpha-1}(xy)^\lambda x\ dy=\frac{x^{\alpha+\lambda}}{\Gamma(r)}\int_0^1(1-y)^{\alpha-1}y^\lambda \ dy,
$$
then you may refer to Beta function.
$$
\frac1{\Gamma(r)}\int_0^x(x-t)^{\alpha-1}t^\lambda\ dt=\frac{x^{\alpha+\lambda}}{\Gamma(r)}\cdot\text{B}\left(\alpha,\lambda+1\right)=\frac{x^{\alpha+\lambda}}{\Gamma(r)}\cdot\frac{\Gamma(\alpha)\Gamma(\lambda+1)}{\Gamma(\alpha+\lambda+1)}.
$$
Letting $t=xy\;\color{blue}{\Rightarrow}\;dt=x\ dy$, the integral turns out to be $$ \frac1{\Gamma(r)}\int_0^1(x-xy)^{\alpha-1}(xy)^\lambda x\ dy=\frac{x^{\alpha+\lambda}}{\Gamma(r)}\int_0^1(1-y)^{\alpha-1}y^\lambda \ dy, $$ then you may refer to Beta function. $$ \frac1{\Gamma(r)}\int_0^x(x-t)^{\alpha-1}t^\lambda\ dt=\frac{x^{\alpha+\lambda}}{\Gamma(r)}\cdot\text{B}\left(\alpha,\lambda+1\right)=\frac{x^{\alpha+\lambda}}{\Gamma(r)}\cdot\frac{\Gamma(\alpha)\Gamma(\lambda+1)}{\Gamma(\alpha+\lambda+1)}. $$