Suppose I have a camera in position $p$ whose look vector is $v$.
Given a target point $t$, I want the camera to smoothly rotate and look at $t$. How can I determine if it is faster to rotate clockwise or anti-clockwise?
In the image, clearly it's faster to rotate clockwise.
Vector camera to current target = $\;\vec v$
Vector camera to new target = $\;\vec t$
The cross product $\;\vec v \times \vec t= \vec r\;$ gives the axis of rotation.
The module of this product $\;\lvert \vec r \rvert = \lvert \vec v \rvert · \lvert \vec t \rvert · sin \theta \;$ gives the angle $\theta\;$ of rotation from $\vec v\;$ to $\vec t\;$. This angle is always $\le 180º$.
In 2D, to tell the clockwise or counter-clockwise rotation see this answer