I am reading through the book Special Functions written by Roy, et al. At page $5$, the limit is justified by Lebesgue dominated convergence theorem (LDCT). I didn't know this theorem so spent about a month learning graduate analysis so that includes the theorem but again it doesn't solve my confusion!
LDCT in graduate real analysis texts is stated for when both upper and lower bounds of integral $f_n(x)$ being constants, or in Lebesgue sense the space on which integration is done is same : so to solve this situation I introduced characteristic function inside the integral. But again in $$\lim_n \int_0^n t^{x-1} (1-t/n)^{y+n-1} dt = \int_0^{\infty} t^{x-1} e^{-1} dt$$ how the three conditions of LDCT are satisfied :
i. $|f_n|=|\mathbb{1}_{[0,n]} t^{x-1} (1-t/n)^{y+n-1}| \le t^{x-1} e^{-1}$,
ii. $\lim f_n = t^{x-1} e^{-1}$ (there is $y$ in $f_n$ but there is none in the limit!),
iii. $f_n \in L^1$.
Thanks in advance.
PS: I read pages of MSE on how to post a good question. So uploading the below is only for a copy of my statement as a reference not an intention of replacement.
$$B(x,y)=\frac{(x+y)_n}{n!}\frac{n!}{y_n}\int^n_0\left(\frac{t}{n}\right)^{x-1}\left(1-\frac{t}{n}\right)^{y+n-1}\frac{dt}{n}$$ $$=\frac{(x+y)_n}{n!n^{x+y-1}}\frac{n!n^{y-1}}{y_n}\int^n_0 t^{x-1}\left(1-\frac{t}{n}\right)^{y+n-1}dt.$$ As $n\to\infty$, the integral tends to $\int^\infty_0t^{x-1}e^{-t}dt$. This may be justified by the Lebesgue dominated convergence theorem. Thus $$B(x,y)=\frac{\Gamma(y)}{\Gamma(x+y)}\int^\infty_0t^{x-1}e^{-t}dt.$$
Note that for fixed $t$, $$\lim_{n \to \infty} t^{x-1}(1-t/n) ^{y+n-1}= t^{x-1}\underbrace{\lim_{n \to \infty}(1-t/n)^n}_{= \,\,e^{-t}}\underbrace{\lim_{n \to \infty}(1-t/n)^{y+1}}_{= \,\,1} = t^{x-1} e^{-t},$$
This limit also serves as an integrable upper bound to justify switching the limit and integral according to LDCT.
Since for $t < n$,
$$-\log(1-t/n) = \int_{1-t/n}^1 \frac{ds}{s} \geqslant \frac{1 - (1-t/n)}{1} = t/n,$$
we have $n \log (1- t/n) \leqslant -t$ which implies $(1-t/n)^n \leqslant e^{-t},$ and, hence,
$$\left|\mathbf{1}_{[0,n]} t^{x-1}(1-t/n) ^{y+n-1} \right| \leqslant t^{x-1}(1- t/n)^n (1- t/n)^{y+1} \leqslant t^{x-1}\cdot e^{-t} \cdot 1$$