Consider a spherical triangle with vertices $A, B$ and $C$, respectively. How to determine its area?
I know the formula:
$A = E R^2$,
where $R$ is radius of sphere, and $E$ is the excess angle of $(a + b + c - \pi)$, but how to determine the angles between $ABC, ACB$ and $BAC$?
On
The area of a spherical triangle can also be calculated using the lengths of its sides, as in this Dr.Math link.
On
If you know the angular distance between the points, L'Huilier's Formula (cited in Derek Jennings' answer) gives $$ \tan\left(\frac{E}{4}\right)=\sqrt{\tan\left(\frac{s}{2}\right)\tan\left(\frac{s-a}{2}\right)\tan\left(\frac{s-b}{2}\right)\tan\left(\frac{s-c}{2}\right)}\tag{1} $$ where $a=\operatorname{ang}(B,C)$, $b=\operatorname{ang}(C,A)$, $c=\operatorname{ang}(A,B)$, and $s=\frac{a+b+c}{2}$.
If $A$, $B$, and $C$ are given as points in $\mathbb{R}^3$, then you can use $(1)$ with $$ \operatorname{ang}(A,B)=\cos^{-1}\left(\frac{A\cdot B}{|A||B|}\right)\tag{2} $$ or, to answer the question you asked, you can compute $$ \angle CAB=\cos^{-1}\left(\frac{(C\;A\cdot A-C\cdot A\;A)\cdot(B\;A\cdot A-B\cdot A\;A)}{|C\;A\cdot A-C\cdot A\;A||B\;A\cdot A-B\cdot A\;A|}\right)\tag{3} $$ and simply compute $E=\angle CAB + \angle ABC + \angle BCA - \pi$.
If $A$, $B$, and $C$ are given in other formats, there are probably ways to handle those, too, but more specific information would be needed.
Addition: Using cross products simplifies $(3)$ a bit: $$ \angle CAB=\cos^{-1}\left(\frac{(C\times A)\cdot(B\times A)}{|C\times A||B\times A|}\right)\tag{4} $$
The angle between two great circles is equal to the angle between the planes containing them. Such a plane always passes through the centre of the sphere, say $O$. So, for example, angle $a$ is the angle between two planes, one of which passes through $A$, $B$, and $O$, and the other passes through $A$, $C$, and $O$.
The angle between two planes, in turn, equals the angle between their normals, and you can determine the normal from the three points by using the cross product.