Given distinct proper rigid transformations $A, B \in \operatorname{SE}(n)$, what is the maximum dimension of the nullspace of $A - B$? That is, what is the maximum dimension of $\operatorname{Eq}(A, B) = \{x \in \mathbb{R}^n\ |\ A(x) = B(x)\}$?
I conjecture that the answer is $n - 2$ by observing simple cases:
When $n = 2$, proper rigid transformations are "defined" by $2$ affinely independent points, so $\operatorname{Eq}(A, B)$ can contain at most one point and has a dimension of $0$.
When $n = 3$, proper rigid transformations are "defined" by $3$ affinely independent points, so $\operatorname{Eq}(A, B)$ is at most the affine hull of two affinely independent points and has a dimension of $1$.
As this line of reasoning is not quite rigorous, I cannot tell if it easily extends to higher dimensions (also hampered by my lack of four-dimensional imagination!). Is my conjecture true for all integers $n \geq 2$?
Yes. WLOG, suppose $A$ and $B$ are linear (so there are no translations). Let $A,B$ also denote the matrix representations of the two transofrmations. As they belong to $SE(n)$, $A^TB\in SO(n)$. Hence $A^TB=UDU^\ast$ for some unitary matrix $U$ and some complex diagonal matrix $D$ whose diagonal entries have unit moduli. Therefore $$\operatorname{nullity}(A-B)=\operatorname{nullity}(I-D).$$ If $\operatorname{nullity}(A-B)=n-1$, $D$ must have $n-1$ eigenvalues equal to $1$. However, as $A^TB$ is real orthogonal and $A\not=B$, the remaining eigenvalue of $D$ must be equal to $-1$. But then $\det A^TB=-1$, which is a contradiction. Therefore the nullity of $A-B$ is at most $n-2$.