Given N points represented by a $N\times 3$ matrix: $$ P= \begin{bmatrix} P_{11} & P_{12} & P_{13} \\ P_{21} & P_{22} & P_{23} \\ P_{31} & P_{32} & P_{33} \\ \cdots & \cdots & \cdots \\ P_{N1} & P_{N2} & P_{N3} \\ \end{bmatrix} $$ where the $i$th row $P_i$ contains three coordinates of the $i$th point.
A rigid transformation preserves the Euclidean distance between every pair of points. The well-known rigid transformations include translations (along the x,y, and z-axis) and rotations (around x,y,and z-axis).
I am looking for six orthogonal vectors (dimension: $3N$ ) as the displacements resulted from applying rigid transformations (three translations & three rotations) to N points. I tried two approaches, the first is analytically false, and the second is not solvable by numeric method.
1. Trivial approach
Applying along x-axis translations and subtracting the original values results in $P^X$, each row $$ P^X_i = [P_{i1} +1,P_{i2},P_{i3} ]- P_i = [1,0,0] $$ y-axis translations results in $P^Y$, each row $P^Y_i = [0,1,0 ]$
z-axis translations results in $P^Z$, each row $P^Z_i = [0,0,1 ]$
One can rewrite matrix $P^X$ into a vector $\mathbf{p}^X$ (representing the displacements after transformation), $$ \mathbf{p}^X=\begin{bmatrix} P^X_{11} \\ P^X_{12}\\ P^X_{13}\\ P^X_{21} \\ P^X_{22}\\ P^X_{23}\\ \dots \\ P^X_{N1} \\ P^X_{N2}\\ P^X_{N3}\\ \end{bmatrix} $$
while $\mathbf{p}^Y$ and $\mathbf{p}^Z$ are defined likewise. Then $ \mathbf{p}^X \cdot \mathbf{p}^Y = \mathbf{p}^X \cdot \mathbf{p}^Z=\mathbf{p}^Y \cdot \mathbf{p}^Z=0$ holds, i.e., the three vectors are orthogonal.
Applying around x-axis translations and subtracting the original values results in $P^A$, each row $$ P^A_i = P_i \begin{bmatrix} 1 & 0 & 0 \\ 0 & \cos\theta & \sin\theta \\ 0 & -\sin\theta & \cos\theta\\ \end{bmatrix}- P_i $$ where $\theta$ denotes a fixed, very small angle.
while $P^B$ (around y-axis) and $P^C$ (around z-axis) can be defined likewise. Rewrite the matrices into vector form $\mathbf{p}^A, \mathbf{p}^B, \mathbf{p}^C$. However, it is easy to find out that the three vectors are generally NOT orthogonal.
2. General approach
One can generalize the three translations along any orthogonal frame specified by three Euler angles, resulting in $$ \mathbf{p}^X , \mathbf{p}^Y , \mathbf{p}^Z $$ associated with 3 parameters.
The three rotations can be also generalized along a frame specified by three Euler angles, around any point for each rotation respectively. It leads to $$ \mathbf{p}^A , \mathbf{p}^B , \mathbf{p}^C $$ associated with 9 parameters (3 Euler angels + 3 points).
There are totally 15 unknowns and 15 equations (any pair of dot product between the six vectors equals zero). But numeric method gives no solution.
So I am wondering why there is no solution for the general approach. Or is there any theory about such mutual orthogonal transforms?