My problem involves finding a formula for this:
You're given a line that passes through the origin in 3D space (more specifically, a point on the line). You are then given a point and an angle. The task is to find where the point ends up after a counterclockwise rotation around the line by the angle.
I managed to do this by reformulating the problem as an intersection of three spheres, but I got two results: one clockwise and one counterclockwise.
At this point I have no idea how to figure out which is which, because "counterclockwise" isn't a sufficient definition. You must know which way the line is oriented.
Is there a way to choose from the two destination points such that rotating objects (such as a cube, given its vertices) aren't deformed (because the points rotate in different directions)?
Let $p$ be the first point lying on the line passing through the origin. Since the line passes through the origin $0$ also lies on it. We can form the unit vector $\frac{p-0}{||p-0||_2} = \hat{p}$, this will be our rotation axis. Let the point we want to rotate around the axis be $v$. After that find the axis-angle rotation matrix $M(\hat{p},\theta)$ (https://en.wikipedia.org/wiki/Rotation_matrix#Axis_and_angle) and the rotate point is $v' = Mv$.