If I have a camera C at the origin. I have a set of three points in space - A, B, and C. There are other points that I am interested in as well. I want to define a transformation matrix (either for the camera or all the points) so that the camera is facing the plane ABC directly.
Is there a standard way of doing this?
In the long run, I am trying to flatten a 3D space to a 2D space, but where the dimension that I'm throwing away is based on three arbitrary points in space. So, I have to rotate everything such that there is a dimension to throw away.
As johnnyb notes in a comment, in order for the camera to face this plane directly, its axis must be perpendicular to, or normal to, the plane. So, your first task is to compute such a normal vector. If you don’t know how to do that, a quick web search will find that one way to compute one is $N=(A-C)\times(A-B)$ or a similar expression.
There’s still a sign ambiguity to deal with. We want a normal that points away from the camera so that when the camera points in this direction, it faces toward the plane. So, compute $N\cdot A$, which is the scalar projection of $A$ onto $N$. If this is positive, then $N$ points from the origin toward the plane, so all is good. If negative, $N$ points away from the plane, so the desired camera direction is $-N$ instead. (If it’s zero, then the camera lies on the plane and you’re out of luck.)
Your problem is now reduced to one that’s been asked and answered many, many times here: search for “align vectors.” Note, though, that there’s an infinite number of rotations that meet your criteria. Once the camera is facing in the desired direction, you can rotate it arbitrarily about its axis and it will still face the plane. You’ll need to decide on some other constraints that you might like to have on the camera motion in order to have a unique solution.