I am trying to solve the following problem- originally I'd posted this on stackoverflow, but it was suggested that I try posting it here instead.
I need to figure out how to align a ray origin and direction so that they aim at a specific target in 3D space by rotating a matrix that the ray origin and direction are parented to. I have prepared a series of graphical examples to illustrate what I'm trying to do:
1) In this picture, the orange cube represents the parent matrix the ray origin (yellow sphere) and ray direction (blue arrow) are parented to. The parent matrix has an offset associated with it but no rotation.
2) In this picture, the parent matrix has been rotated so that the ray origin and direction are guaranteed to intersect with the target point. The entire setup has then been aligned around the axis formed between the parent matrix offset and the target point so that the ray origin falls upon a plane defined by the parent matrix offset, up vector, and target point.
3) Another view of #2, but this time showing how the ray direction is now pointing directly at the target point.
4) This picture shows how the entire system is aligned to the up vector (otherwise there would be no singular solution)
How can I calculate the rotation of the parent matrix to solve the system mentioned above? Note that the positions of the vectors and matrices described above are not constant and will shift around, so I need to be able to solve this system regardless of where things are in 3D space.
I would break the problem down into three rotations, pretty much as you’ve got the process broken down in your diagrams: first bring the ray onto the ray-origin/target plane, then rotate about this plane’s normal to point the ray at the target, and finally, rotate about the origin-target axis to bring the whole mess onto the target/up-vector plane. Each of these rotations is quite simple to construct. I won’t explain how to do that here. There are plenty of resources both on this SE and elsewhere on the Internet that explain ways to do that.
W.l.o.g. assume that the orange box is at the origin (zero offset). If it isn’t you can throw in a couple of translations. Let $P$ be the ray origin, $T$ the target, $d$ the ray’s initial direction and $u$ the up vector. The first and last rotations align pairs of planes, so you can determine the correct rotation by comparing normals. For the first rotation $\mathbf R_1$, rotate about the axis through $P$ to align $P\times r$ with $P\times T$. For the last rotation $\mathbf R_3$, the axis goes through $T$ and we’re rotating $P\times T$ onto $u\times T$. For the second rotation, find a nonnegative scalar $\lambda$ such that $\|P+\lambda r\|=\|T\|$. This is a matter of solving a quadratic equation for $\lambda$. The rotation is about the axis $P\times T$ and brings $P+\lambda R_1r$ onto $T$. The required rotation matrix is then $\mathbf R=\mathbf R_3\mathbf R_2\mathbf R_1$.
Working through this for your example, I get $$\mathbf R_1 = \left( \begin{array}{ccc} 0.726864 & 0.0259659 & -0.68629 \\ -0.116034 & 0.989562 & -0.0854536 \\ 0.676908 & 0.141746 & 0.72229 \\ \end{array} \right) \\ \mathbf R_2 = \left( \begin{array}{ccc} 0.986999 & 0.136551 & 0.0847708 \\ -0.0458694 & 0.744811 & -0.665697 \\ -0.15404 & 0.653155 & 0.741391 \\ \end{array} \right) \\ \mathbf R_3 = \left( \begin{array}{ccc} 0.865045 & -0.498 & 0.0607611 \\ 0.50091 & 0.864103 & -0.0491426 \\ -0.0280309 & 0.0729464 & 0.996942 \\ \end{array} \right) $$ and $$\mathbf R = \left( \begin{array}{ccc} 0.959663 & -0.124591 & -0.252041 \\ -0.128137 & 0.604122 & -0.786523 \\ 0.250258 & 0.787092 & 0.563788 \\ \end{array} \right).$$ This yields $$\mathbf RP = (-298.358, 521.806, 771.946) \\ \mathbf Rr = (0.429653, -0.299892, 0.852126),$$ which you can verify are coplanar with $u$ and $T$, and that $T$ lies on the transformed ray:
You’ll need to watch out for and handle a few degenerate cases, which occur when some of these vectors are colinear so that their cross products vanish.