How to find the orientation of three vertices A, B and C of a plane in 3D are in clockwise, anticlockwise or collinear? I find this below link in 2D
http://www.geeksforgeeks.org/orientation-3-ordered-points/
I have tried in 3D, but I am not sure about this. dot(n, cross(A-C, B-C)) where n is normal to a plane.
If the result is positive, B is counterclockwise from A; if it's negative, B is clockwise from A. Any help would be much appreciated.
Let's say your three points are $\vec{p}_1 = ( x_1 , y_1 , z_1 )$, $\vec{p}_2 = ( x_2 , y_2 , z_2 )$, and $\vec{p}_3 = ( x_3 , y_3 , z_3 )$.
Let $$\vec{n} = \left ( \vec{p}_2 - \vec{p}_1 \right ) \times \left ( \vec{p}_3 - \vec{p}_1 \right )$$ i.e. $$\begin{cases} x_n = (y_2 - y_1) (z_3 - z_1) - (z_2 - z_1) (y_3 - y_1) \\ y_n = (z_2 - z_1) (x_3 - x_1) - (x_2 - x_1) (z_3 - z_1) \\ z_n = (x_2 - x_1) (y_3 - y_1) - (y_2 - y_1) (x_3 - x_1) \end{cases} $$ If the three points are collinear, then $\vec{n} = ( 0 , 0 , 0 ) = 0$. Otherwise, the three points are on a plane, with $\vec{n}$ being normal (perpendicular) to the plane.
As mentioned in a comment, if we look at the triangle from the side the normal vector $\vec{n}$ points to, the points are in counterclockwise order; but, if we look at the triangle from the other side, they are in clockwise order.
If we know that the three points are on a plane with normal vector $\vec{k}$, then $$\begin{cases} \vec{k} \cdot \vec{n} = \vec{k} \cdot \left ( \left ( \vec{p}_2 - \vec{p}_1 \right ) \times \left ( \vec{p}_3 - \vec{p}_1 \right ) \right ) \gt 0, & \text{ counterclockwise } \\ \vec{k} \cdot \vec{n} = \vec{k} \cdot \left ( \left ( \vec{p}_2 - \vec{p}_1 \right ) \times \left ( \vec{p}_3 - \vec{p}_1 \right ) \right ) \lt 0, & \text{ clockwise } \\ \vec{k} \cdot \vec{n} = \vec{k} \cdot \left ( \left ( \vec{p}_2 - \vec{p}_1 \right ) \times \left ( \vec{p}_3 - \vec{p}_1 \right ) \right ) = 0, & \text{ oops } \end{cases}$$
The "oops" case covers several possible situations. For example, if the three points are collinear, then $\vec{n} = 0$. Or, if the three points are on a plane parallel to $\vec{k}$, the result is zero also.