I am trying to formalize mathematically the current problem I am facing. Current issue is:
Working in the medical field, we receives MR images that have been acquired along the same direction, when reading the direction cosine matrix from file, the values are slightly different (up to a certain precision). It is this impossible to do a simple substraction to check for equality. We need to define a distance function to check if two direction cosine matrix are below a particular $\epsilon$ value.
So if I try to reformulate my original issue it seems to be:
given two oriented planes $P_1$ & $P_2$ define a space metric in between them.
For vectors the function $d(x,y)=\arccos(x\cdot y)$ is a metric on the unit sphere, i.e. it's symmetric, positive-definite, and satisfies the triangle inequality. It's basically just the great circle distance between the two points.
My naive assumption was that it should be possible to extend that with the angle in between two planes. With $\vec{N_1}$ and $\vec{N_2}$ the normals of resp. planes $P_1$ & $P_2$ we have the angle $\theta$ defined as:
$$\theta = \arccos( \vec{N_1} \cdot \vec{N_2})$$
We can verify this metric works for planes defined with the following direction cosine matrix:
$$P_1: \vec{U_1} = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \vec{V_1} = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}$$
$$P_2: \vec{U_2} = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}, \vec{V_2} = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}$$
lead to: $$ \theta = \arccos( \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}) = \frac{\pi}{2} > 0$$
And it gives the correct result when $P_1 = P_2$ (=> $\theta = 0$), which satifies my current need for my original problem.
However it does not gives the correct result when we craft a specific orientation of the plane $P_2$ (along same infinite plane as $P_1$, ie. same normal):
$$P_1: \vec{U_1} = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \vec{V_1} = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}$$
$$P_2: \vec{U_2} = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}, \vec{V_2} = \begin{bmatrix} -1 \\ 0 \\ 0 \end{bmatrix}$$
lead to: $$ \theta = \arccos( \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}) = 0$$
What would be a right space metric function for two oriented planes ?
To give an actual numerical example of direction cosine matrix that I have to compare:
$$ P_1 : \begin{pmatrix} 0.997704 & -0.0673549 \\ 0.0677201 & 0.992439 \\ 6.10347\text{e-}5 & -0.102604 \end{pmatrix}$$
while
$$ P_2 : \begin{pmatrix} 0.997704 & -0.0673549 \\ 0.0677202 & 0.992439 \\ 6.10347\text{e-}5 & -0.102604 \end{pmatrix}$$