I have 3 points in 3D and I want to calculate $ABC$ angle, which could be reflex angle. I also know the normal of the plane they are in. Is there a way to calculate the angle?
EDIT:
Take a look at this picture. The two vectors create two angles one would be $a$ and the other $360 - a$. Now I also know the direction of normal between the two vectors. With that information how can I determine if they construct angle $a$ or $360-a$.
On
You can use scalar product $$ \langle \mathbf{u},\mathbf{v}\rangle = \lVert \mathbf{u}\rVert \lVert \mathbf{v}\rVert \cos\alpha $$ where $\alpha$ is the angle between $\mathbf{u}$ and $\mathbf{v}$.
Hence $$ \alpha=\arccos \frac{\langle \mathbf{u},\mathbf{v}\rangle}{\lVert \mathbf{u}\rVert \lVert \mathbf{v}\rVert} $$
In your case \begin{align} \mathbf{u}=A-B\\ \mathbf{v}=C-B \end{align}
Notice that you'll get $\alpha\le180°$.
I finally found a solution for this case in here.
The part that is relevant for this question:
When your vectors are not placed arbitrarily, but lie within a plane with a known normal vector $n$. Then the axis of rotation will be in direction $n$ as well, and the orientation of $n$ will fix an orientation for that axis. In this case you can calculate the angle via:
$$dot = x_1x_2 + y_1y_2 + z_1z_2$$ $$det = x_1y_2z_n + x_2y_nz_1 + x_ny_1z_2 - z_1y_2*x_n - z_2y_nx_1 - z_ny_1*x_2$$ $$angle = arctan(det, dot)$$
One condition for this to work is that the normal vector n has unit length. If not, you'll have to normalize it.