I am trying to back-project points in (homogeneous) 2D space to rays in 3D space.
For the 2D points I have the intrinsic (image-to-camera) matrix and extrinsic (pose) matrix of the viewpoint as
$$ K_{3\times3}:= \text{3x3 intrinsic matrix}$$ $$ P_{3\times4}:= \left[ R_{3\times3} | -R_{3\times3}C_{3\times1} \right]$$
where $R$ is an orthonormal world-to-image rotation matrix and $C$ is the center of projection in world coordinates (i.e. the "camera center").
I am backprojecting using a slightly simplified (but identical) version of the standard 2-step process of projecting camera-to-image, and image-to-world. That's usually done as
$$X_{cam} = \left[x,y,1\right]^T$$ $$X_{img} = K^{-1} X_{cam}$$ $$X_{world} = P^{-1}\left[X_{img} 1\right]^T $$
where $P^{-1}$ is a 3x4 pseudoinverse of $P$.
To skip over the need to augment my image points with the homogeneous $1$, I just do
$$X_{world} = C + \lambda R^TK^{-1}X_{cam}$$
where $\lambda$ is some distance along the ray.
However, when I take this approach, I always end up behind my camera. In fact, the result is not the negative of the desired output, but the exact reflection.
Why am I ending up with a reflection?
Update: I checked the determinant of $R$ and it is +1, so it is a rotation matrix rather than a reflection matrix.
If it helps... Here is a visualization showing the output. The green are the computed camera viewing frustums computed this way (using the corners of an image). Those other dots you see are known 3D points in a scene. They should be fully encompassed by the viewing frustum. The frustum pointing left is the output when I just take the negative depth along the ray. The frustum pointing right is what is computed above (note it's pointing in the exact opposite direction than it should).
