I have been trying to digest the concept of similarity transformation, but the more I study it, the more I get confused.
To start with, let's have two reference frames 0 and 1 with a homogeneous transformation $T^0_1$ relating the two frames (the superscript is the frame we convert to, so $T^0_1$ takes a vector in frame 1 and transforms it to frame 0).
Now, if $A$ is a linear transformation expressed in frame 0 and $B$ is the similar transformation to $A$, but expressed in frame 1, then $A$ and $B$ are related as follows:
$$ A=(T^0_1) B (T^0_1)^{-1} $$
However, I have also learned how to compound transformations. So, since $B$ is expressed relative to frame 1, then, if I premultiply it by $T^0_1$, I get an equivalent transformation (call it $C$) but relative to frame 0, isn't that right? But wait a second, shouldn't $C$ be our $A$ above? However, the two quantities don't match. So, please tell me what I'm missing here.
You need to be very careful to separate the vector from its representation in a particular basis. Also, compounding transforms doesn't correspond to expressing a transformation in a different basis -- infact this is precisely what similarity transformations are for. So maybe you want to know how the similarity transformation arises.
Say that expressing $B$ in frame 1, relative to frame 0, gives you a matrix $C$. Then what property must $C$ satisfy? It should hold that vectors (the little pointed arrows, "geometric entities", or whatever you imagine them to be) transform independent of the frame. Colloquially speaking, a length of 10 m should be represented as the number 10 if you are using a scale of length 1 m, and the number 5 if you are using a scale of length 2 m, but, the vector corresponding to both the representations is the same -- it is just being expressed in two different basis.
Writing out what is decided on above formally, you have that if you take a vector represented in frame 1 as $v$ and multiply it by the matrix $B$, then the vector that you get is the same as the vector you obtain by multiplying the representation of the vector in frame 0 by matrix $C$. But, the representation of the vector in frame $0$ is, in your notation, given by $T_1^0 v$.
So you have: $V(CT_1^0v) = V(Bv)$, where $V(\cdot)$ denotes the vector corresponding to its expression in a particular frame. But, you know you can transform the expression of the transformed vector $CT_1^0v$ (the important part, and where you are possibly slipping up -- this is an expression in frame 0) into an expression of the vector in frame 1 by $(T_1^0)^{-1}(CT_1^0v)$. But, the vector is also expressed in frame 1 as $Bv$. So, these expressions must be the same. This is true for all vectors, and so: $(T_1^0)^{-1}CT_1^0 = B$.