I'm trying to get an equation for a solid angle of a segment of octahedron in the same vein as described in this article cubemap-texel-solid-angle. I ended up having to integrate $$\int \int \frac{1}{(x^2+y^2+(1-x-y)^2)^\frac{3}{2}} \,\mathrm{d}x \,\mathrm{d}y$$ where $0 \leq x \leq 1$ and $0 \leq y \leq 1-x$. That is, integral over a segment of a triangle mapped onto the sphere(one octant). Does anyone know how to integrate that? Subdivided polyhedrons by courtesy of Gavin Kistner
Update Thanks to the general formula from this answer: $$\omega=\cos^{-1}\left(\frac{\cos\alpha-\cos\beta\cos\gamma}{\sin\beta\sin\gamma}\right)-\sin^{-1}\left(\frac{\cos\beta-\cos\alpha\cos\gamma} {\sin\alpha\sin\gamma}\right)-\sin^{-1}\left(\frac{\cos\gamma-\cos\alpha\cos\beta}{\sin\alpha\sin\beta}\right)$$
We can calculate a solid angle for all the triangles. Here is the gist and the shadertoy. The naive implementation is not numerically stable at small angles.
Update See this answer
The general problem I think you have described is to find the solid angle subtended at one vertex of a tetrahedron.
If we label the vertex in question $O$ and put it at the center of a unit sphere, then project the opposite face onto the sphere, we get a spherical triangle. The lengths of the "sides" of that triangle are the angles $\alpha,$ $\beta,$ and $\gamma$ between the edges of the tetrahedron that meet at $O.$ The angles at the vertices of the spherical triangle are the dihedral angles $A,$ $B,$ and $C$ between the faces of the tetrahedron that meet at $O.$ The usual convention is we use the name $A$ for the angle between the sides of length $\beta$ and $\gamma,$ the name $B$ for the angle between the sides of length $\alpha$ and $\gamma,$ and the name $C$ for the angle between the sides of length $\alpha$ and $\beta.$
The solid angle at $O$ is then the area of the spherical triangle, which in turn is equal to the spherical excess of that triangle, defined as $$ E = A + B + C - \pi. $$
But the information that you seem to be assuming is that you know the three angles $\alpha,$ $\beta,$ and $\gamma.$ So the question becomes how to find $E$ in terms of those angles.
The spherical law of cosines says that $$ \cos\alpha = \cos\beta \cos\gamma + \sin\beta \sin\gamma \cos A. $$ Solving for $A$ we get $$ A = \arccos \left(\frac{\cos\alpha - \cos\beta \cos\gamma} {\sin\beta \sin\gamma}\right) .$$
There are similar formulas involving the angles $B$ and $C,$ with the results $$ B = \arccos \left(\frac{\cos\beta - \cos\alpha \cos\gamma} {\sin\alpha \sin\gamma}\right) $$ and $$ C = \arccos \left(\frac{\cos\gamma - \cos\alpha \cos\beta} {\sin\alpha \sin\beta}\right) .$$
As a result, one formula for the spherical excess is \begin{align} E &= \arccos \left(\frac{\cos\alpha - \cos\beta \cos\gamma} {\sin\beta \sin\gamma}\right) \\ &\qquad + \arccos \left(\frac{\cos\beta - \cos\alpha \cos\gamma} {\sin\alpha \sin\gamma}\right) \\ &\qquad + \arccos \left(\frac{\cos\gamma - \cos\alpha \cos\beta} {\sin\alpha \sin\beta}\right) - \pi. \end{align}
The formula shown in the question is a variation of this formula that can be obtained using the identity $\arccos(x) = \frac\pi2 - \arcsin(x).$
I would be suspicious of the numerical stability of this formula for very small spherical angles (that is, when you have divided your sphere into a very large number of triangular facets), because neither $\arccos(x)$ nor $\arcsin(x)$ is very accurate when $x$ is close to $1.$ You might be better off with another formula such as $$ E = 2 \arctan\left(\frac{\tan\frac\alpha 2 \tan\frac\beta 2 \sin C} {1 + \tan\frac\alpha 2 \tan\frac\beta 2 \cos C}\right), $$ (from here) using formulas such as $$ \cos C = \frac{\cos\gamma - \cos\alpha \cos\beta}{\sin\alpha \sin\beta} $$ and $$ \sin C = \sqrt{1 - \cos^2 C}. $$
This should be fine if the three angles of the triangle are approximately equal (as seems to be the case in your "octahedron"-based construction). If one of the angles is almost $180$ degrees and the other two are almost zero you might want to compute $\sin C$ differently.