I've tried to find a similar formula of following.
I've heard about this formula \begin{align} \prod_{i=1}^n \dfrac{1}{a_i} = \Gamma(n)\left[ \prod_{i=1}^n \int_0^1 \mathrm{d }x_i\right]\dfrac{\delta(1-\sum_{i=1}^n x_i)}{(\sum_{i=1}^n a_i x_i)^n} \end{align} in a class. ($\delta(x)$ is a Dirac-delta function. I use this for convenience.)
I checked that it's okay when $n=2$.
I tried to prove this but I've noticed when $n=3$, it holds that \begin{align} \int_0^1 \int_0^1 \mathrm{d}x_1 \mathrm{d}x_2 \dfrac{1}{(m_1 x_1 + m_2 x_2 + m_3)^3} &= \int_0^1 \int_{m_1 x_1 + m_3}^{m_1 x_1 + m_2 + m_3} \dfrac{\mathrm{d}x_1 \mathrm{d} u}{m_2 u^3}\\ &= -\dfrac{1}{2m_2}\int_{0}^1 \dfrac{\mathrm{d}x_1}{(m_1 x_1 + m_2 + m_3)^2} - \dfrac{\mathrm{d}x_1}{(m_1 x_1 + m_3)^2} \\ &=\dfrac{1}{2}\dfrac{m_1 + m_2 + 2 m_3}{m_3 (m_1 + m_3)(m_2 + m_3)(m_1 + m_2 + m_3)}\\ \end{align} Obviously, it doesn't coincide with what I want to show.(When substitute $m_1$ to $a-c$, $m_2$ to $b-c$ and $m_3$ to $c$.)
Therefore, I think the formula suggested in the class is wrong. However, it seems to be used in several calculation and so, I think there is a similar formula rather than this.
Could someone tell me what it might be?
The formula you cite is correct; in fact, it can be generalized. Your error is assuming $x_2$ integrates from $0$ to $1$; it should be $0$ to $1-x_1$. So the correct result for$$a:=a_1,\,b:=a_2,\,c:=a_3,\,m_1:=a-c,\,m_2:=b-c,\,m_3:=c$$is$$\int_0^1dx_1\int_{m_1x_1+m_3}^{m_1x_1+m_2(1-x_1)+m_3}\frac{du}{m_2u^3}\\=\frac{1}{2m_2}\int_0^1dx_1\left(\frac{1}{(m_1x_1+m_3)^2}-\frac{1}{((m_1-m_2)x_1+m_2+m_3)^2}\right)\\=\frac{1}{2m_2}\left(\frac{1}{m_3(m_1+m_3)}-\frac{1}{(m_1+m_3)(m_2+m_3)}\right)\\=\frac{1}{2(m_1+m_3)m_3(m_2+m_3)}=\frac{1}{2acb}.$$