I'm currently trying to work out if an angle is a clockwise or anti-clockwise rotation about a point. I used the equation: a.b = ||a|| ||b|| cos(A) to calculate the angle between the two vectors in 3D space, but I'm now unsure as to how to tackle the clockwise/anti-clockwise problem and would really appreciate some help on how to approach it.

Thank you!
In three dimensions, this requires a choice of orientation of the plane $P$ spanned by ${\bf a}, {\bf b}$. We can specify such an orientation by choosing a vector $\bf n$ transverse to $P$; then, the basis (${\bf a}, {\bf b}$) is positively oriented with respect to the orientation---equivalently, the angular displacement from $\bf a$ to $\bf b$ is anticlockwise from the p.o.v. of the endpoint of $\bf n$---if $$\det \begin{pmatrix} {\bf a} & {\bf b} & {\bf n} \end{pmatrix} > 0$$ and the reverse is true, and in particular the displacement is clockwise, if $$\det \begin{pmatrix} {\bf a} & {\bf b} & {\bf n} \end{pmatrix} < 0.$$
NB in two dimensions, there is a conventional choice of orientation of the plane, and if we write ${\bf a} = (a_1, a_2)^T$ and ${\bf b} = (b_1, b_2)^T$, then we have $$\det \begin{pmatrix} {\bf a} & {\bf b} \end{pmatrix} = a_1 b_2 - a_2 b_1 > 0$$ if the displacement is anticlockwise and the reverse inequality if it is clockwise.