I am working in homogeneous coordinates to represent affine transforms in 3-d. As such, my points in $\mathbb{R^3}$ get a fourth coordinate, and my transforms are 4x4 matrices. To be clear, I transform a point $X \in \mathbb{R}^3 = (x, y, z)^T$ into a point $X' = (x', y', z')^T$ via a 4x4 matrix $M$ by computing $$Y = M. (x, y, z, 1)^T = (x'', y'', z'', w'')^T$$ and then $$X' = (x''/w'', y''/w'', z''/w'')^T.$$ Let denote $f$ the mapping $X' = f(X)$
In practice, my matrices represent rotations in their 3x3 upper left corner, translations in their rightmost column, and scaling factor in their bottom-right value. They are hence invertible (assuming the scaling factor is non zero).
$$M = \begin{bmatrix} R&t\\0&1/s\end{bmatrix}$$ with $R$ a 3x3 rotation matrix, $t$ a 3x1 vector and $s$ a scaling factor.
I would like to define a meaningful distance between two such matrices $M_1$ and $M_2$. That would mean, for instance, computing something like $\sup_{X \in \mathbb{R}^3, ||X||=1} ||f_1f_2^{-1} X - X||$, with $f_1$ and $f_2$ the maps corresponding to the transforms by $M_1$ and $M_2$
Does such a norm exist in the litterature, and how can I compute it in practice given two 4x4 matrices ?
The matrix itself is again homogeneous, i.e. a multiple of the matrix describes the same operation. So you'd need something that does yield distance zero for that, which includes the special case of the matrix with all values negated.
So in a way you are dealing with $\mathbb{RP}^{15}$, a $15$-dimensional space of homogeneous coordinate vectors, the $16$ elements of which correspond to your $16$ matrix entries.
You could pick one hyperplane as the plane at infinity, and then compute something like (relative) Euclidean distances from that. But I see no strong reason to distinguish one of the directions in that space over the others. And I'm worried about the infinities such a choice would bring with it. Could be that for the affine sup-problem you're concerned with, an affine choice here would indeed make sense, as I might be biased towards considering the generic projective setup.
So instead of doing affine geometry, I'd prefer elliptic geometry: consider the $15$-sphere you get by normalizing all your $16$-dimensional vectors to unit length. There are two choices of sign, so antipodal points on the sphere are identified. The natural distance measure in Elliptic geometry is the angle between two lines.
As a first step towards computing that angle, I'd compute the dot product of the normalized matrices, with all operations treating them as vectors not operations.
$$\left\langle\frac{M_1}{\lVert M_1\rVert},\frac{M_2}{\lVert M_2\rVert}\right\rangle$$
That quantity corresponds to the cosine of the angle. You might need to normalize it to the $\left[-\frac\pi2,+\frac\pi2\right]$ range by adding multiples of $\pi$, in order to deal with the opposite sign is same operation scenario (which is equivalent to moving from the angle between directed vectors to that between undirected lines).
Another way to get rid of the opposite sign for opposite vectors would be taking the square of the expression. You can even use that to avoid the square roots for the denominators:
$$\frac{\langle M_1,M_2\rangle^2}{\langle M_1,M_1\rangle\,\langle M_2,M_2\rangle}$$
This expression is $1$ for same operations and $0$ for maximally different (orthgonal) operations. You could take the square root and then the cosine to turn this back into angles. But if all you need is a monotonic function of the angle, you might get away with
$$1-\frac{\langle M_1,M_2\rangle^2}{\langle M_1,M_1\rangle\,\langle M_2,M_2\rangle}$$
which is $0$ for same operation and becomes larger the more different they are, with respect to the 16-dimensional angle definition of difference outlined above.
I haven't gone to the trouble yet of checking which potentially desirable properties of a metric these definitions satsify. I'm certain that they are not the only possible definitions, and for many applications probably not the most appropriate ones. But they do feel to me as though they would make sense in a wide and generic class of problems, so I suggest you give them a try for your specific application.