Given 3 unit vectors $\vec{a}$, $\vec{b}$, and $\vec{c}$ in $\mathbb{R}^3$.
I would like to compute the rotation between $\vec{a}$ and $\vec{b}$.
Then apply that rotation to $\vec{c}$ to obtain a new vector $\vec{d}$.
$\vec{c}$ is not coplanar with $\vec{a}$ and $\vec{b}$.
I attempted this with both rotation matrices and quaternions, computing the rotation between $\vec{a}$ and $\vec{b}$, but when I apply it to $\vec{c}$ the dot product is not equal, e.g.:
$\vec{a} \cdot \vec{b} \ne \vec{c} \cdot \vec{d}$
Because $\vec{c}$ is not coplanar with $\vec{a}$ and $\vec{b}$.
Should I compute the rotation for each axis independently and then recombine them? I'm not sure what the correct approach is here.
Context:
I'm trying to correlate the rotation from the center pixel of an image $\vec{a}$ to a point of interest on the image $\vec{b}$. Given that I know the cameras focal length I can compute the unit vectors in pixel space. I have the cameras angle of focus in world coordinates $\vec{c}$ and want to angle the camera by the same amount such that it is pointing directly at the point of interest, new vector $\vec{d}$
Of course, you use Rodrigues rotation formula. I'm pretty certain there's an equivalent quaternion formulation, but... well, I'm not Professor Quaternion. =D In this case, $\alpha$ and $\theta$ are equal, and since $\vec{a}$ and $\vec{b}$ are unit vectors, we have $\sin\theta=|\vec{a}\times\vec{b}|$ and $\cos\theta=\vec{a}\cdot\vec{b}$, giving the final result $$\vec{d}=(\vec{a}\cdot\vec{b})\,\vec{c}+|\vec{a}\times\vec{b}|\,\vec{k}\times\vec{c}+(1-\vec{a}\cdot\vec{b})\,(k\cdot\vec{c})\,\vec{k},$$ where $$\vec{k}=\frac{\vec{a}\times\vec{b}}{|\vec{a}\times\vec{b}|}$$ is your (normalized) axis of rotation.
Edit: the formulation $$\vec{d}=(\vec{a}\cdot\vec{b})\,\vec{c}+(\vec{a}\times\vec{b})\times\vec{c}+\frac{((\vec{a}\times\vec{b})\cdot\vec{c})\,\vec{a}\times\vec{b}}{1+\vec{a}\cdot\vec{b}}$$ avoids the problem worrying @John Huges... but it doesn't resolve another one: $\vec{a}=-\vec{b}$. In that case, it's really not clear what should happen.