Let's say I have the following homogenous transform
$$H_1 = \begin{bmatrix} 1 & 0 & 0 & 1 \\ 0 & 1 & 0 & 2 \\ 0 & 0 & 1 & 3 \\ 0 & 0 & 0 & 1 \end{bmatrix}$$
and another one
$$H_2 = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}$$
I want to find the relative transform $H$ such that $H_2 = HH_1$.
$$H = H_2H_1^{-1}= \begin{bmatrix} 1 & 0 & 0 & -1 \\ 0 & 1 & 0 & -2 \\ 0 & 0 & 1 & -3 \\ 0 & 0 & 0 & 1 \end{bmatrix}$$
which makes sense because I want to subtract $\begin{bmatrix} 1 & 2 & 3 \end{bmatrix}^T$ from the translation component of $H_1$ to get $H_2$.
However, let's say I introduced some rotation.
$$H_1 = \begin{bmatrix} 1 & 0 & 0 & 1 \\ 0 & 0 & -1 & 2 \\ 0 & 1 & 0 & 3 \\ 0 & 0 & 0 & 1 \end{bmatrix}$$
$$H_2 = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}$$
$$H_2H_1^{-1} = \begin{bmatrix} 1 & 0 & 0 & -1 \\ 0 & 0 & 1 & -3 \\ 0 & -1 & 0 & 2 \\ 0 & 0 & 0 & 1 \end{bmatrix}$$
I no longer get the same type of behavior. I was wondering why this is the case.
The inverse of a matrix product is the product of the individual inverses, but reversed. After introducing a rotation, you effectively have the product of a translation and rotation. In block form, $$\left[\begin{array}{c|c}R & \mathbf t \\ \hline \mathbf 0^T & 1\end{array}\right] = \left[\begin{array}{c|c}I & \mathbf t \\ \hline \mathbf 0^T & 1\end{array}\right] \left[\begin{array}{c|c}R & \mathbf 0 \\ \hline \mathbf 0^T & 1\end{array}\right].$$ Inverting this gives you $$\begin{align} \left[\begin{array}{c|c}R & \mathbf t \\ \hline \mathbf 0^T & 1\end{array}\right]^{-1} &= \left[\begin{array}{c|c}R & \mathbf 0 \\ \hline \mathbf 0^T & 1\end{array}\right]^{-1} \left[\begin{array}{c|c}I & \mathbf t \\ \hline \mathbf 0^T & 1\end{array}\right]^{-1} \\ &= \left[\begin{array}{c|c}R^Y & \mathbf 0 \\ \hline \mathbf 0^T & 1\end{array}\right] \left[\begin{array}{c|c}I & -\mathbf t \\ \hline \mathbf 0^T & 1\end{array}\right] \\ &= \left[\begin{array}{c|c}R^T & -R^T\mathbf t \\ \hline \mathbf 0^T & 1\end{array}\right]. \end{align}$$ What this says is that when you undo the combined rotation and translation, you can either undo the translation first and then undo the rotation, reversing the order of the operations, or you can first undo the rotation and then translate by the un-rotated translation vector.