How do I detect if a 4x4 transformation matrix contains reflection?

Question

We currently check if the determinant of the upper left 3x3 values is negative to detect reflection in a 4x4 transformation matrix but we are unsure that it works in all cases (any arbitrary 3D reflection plane).

What do you think about it? Do you know a different approach?

Thanks.

EDIT: For closed objects (e.g. a Tetrahedron) it's easy to check if the 4x4 transformation altered it because it volume becomes negative. We are trying to detect this alteration even for open objects like a dome. Checking the 3x3 determinant seems to work fine but we wanted to be sure that it was the correct approach.

Answer 1

If your matrix is the augmented matrix representing an affine transformation in 3D, then yes, the proper thing to do to see if it switches orientation is checking the sign of the top $3\times 3$ determinant. This is easy to see: if your transformation is $Ax+b$, then the $+b$ part is a translation and does not affect orientation, and $x\mapsto Ax$ switches orientation iff $\det A<0$ (you probably know this fact already).

If your matrix is a proper projective transformation (i.e., if it does not leave the point at infinity fixed), then I am not sure but my guess would be the determinant of the whole $4\times 4$ matrix.

Answer 2

If $R$ is a $4 \times 4$ matrix representing a reflection with fixed points on a 3D hyperplane, then his spectrum is $\sigma(R)=(1,1,1,-1)$, i.e. the eigenvalues are $1$, with eigenvectors on the 3D hyperplane and $-1$ with an eigenvector orthogonal to such hyperplane.

So this can be a test. But I don't understand how your test works.