I need to find correct proof that β-function is smooth on its domain.
Is there some feature of such functions, I guess that we need to prove the continuity of all n-derivatives, or their partial derivatives, but how it will be eventually I hadn't got yet. I will very glad your help.
You have more than enough to differentiate through the integral sign (the Leibniz rule.) For example, thinking of $x,y>0,$ we have
$$\frac{d}{dx}\int_0^1t^{x-1}(1-t)^{y-1}\,dt = \int_0^1(\ln t)t^{x-1}(1-t)^{y-1}\,dt.$$
You can keep going, piling up factors like $(\ln t)^m \ln (1-t)^n$ in the integral. None of these factors will destroy integrability.