In Hartley & Zisserman's Multiple View Geometry in Computer Vision, in the chapter on trifocal tensors (section 14.1.5 in the linked copy), the authors take extra steps to ensure that the three retrieved cameras are in the same projective frame.
Can someone explain what this buys me, in terms of camera models and reconstruction? In particular, if I "upgrade" camera pair $\{P, P'\}$ to an affine or metric reconstruction, can I then use this common frame to upgrade pair $\{P, P''\}$ to a corresponding reconstruction? [I have to admit that I don't precisely know what I mean by "corresponding"; intuitively, I want both reconstructions to represent the same 3D structure.]