I'm looking for the formula to calculate the cartesian coordinates of a point A(Xa, Ya, Za) after a rotation of an angle δ (in radians) around an axis B(Xb, Yb, Zb)C(Xc, Yc, Zc).
I found how to do it using rotation matrices or quaternions. But how can I do that using the cartesian coordinates of the A, B and C points?
$\newcommand{\Vec}[1]{\mathbf{#1}}$Here is an algorithm (not a formula per se, but can be turned into one):
Geometrically, $(\Vec{e}_{1}', \Vec{e}_{2}', \Vec{e}_{3})$ is the result of rotating the "original" orthonormal basis $(\Vec{e}_{1}, \Vec{e}_{2}, \Vec{e}_{3})$ through angle $\delta$ about $\Vec{e}_{3}$. The formula in item 4. decomposes $A - B$ (the displacement of point $A$ from the "origin" $B$) into components with respect to the original basis, and uses those as components with respect to the rotated basis.
Note: If $\Vec{e}_{3} = (0, 0, \pm 1)$ we can pick $\Vec{e}_{1} = (1, 0, 0)$ and $\Vec{e}_{2} = (0, 1, 0)$. Otherwise, we have $\Vec{e}_{3} = (X, Y, Z)$ with $X^{2} + Y^{2} \neq 0$, and may pick $$ \Vec{e}_{1} = \frac{(-Y, X, 0)}{\sqrt{X^{2} + Y^{2}}}. $$ This is not numerically robust if the segment $\overline{BC}$ is nearly parallel to the third Cartesian axis. In practice, find the two largest (in absolute value) components of $\Vec{e}_{3}$ and pick $\Vec{e}_{1}$ using those instead of the first two.