This question is related, or maybe is better to say inspired, to/by this other one about quantum mechanics.
From what I currently understand the action of a transformation $T$ on another matrix, or to be more general on another operator $A$, is obtained by applying the following formula:
$$TAT^{-1}$$
Just to put some reference this formula can be found on this wikipedia article about change of basis.
The result of this calculation should be another operator $A'$
$$A'=TAT^{-1} \tag{1}$$
that does exactly what $A$ was doing but on the transformed space (right?). Through the course of my studies I was always struggling to remember this formula; I was always making confusion between $TAT^{-1}$ and $T^{-1}AT$, and I was always forced to double check. But then I found some nice physical/mathematical interpretation of this formula that made remembering it a lot easier: of course the formula must be $TAT^{-1}$, because we want the action of $A'$ to be the same of $A$ but in the transformed space, so what we do to accomplish this is
- Using $T^{-1}$ to turn $\vec{v}'$ into $v$
- Now that we are dealing with a vector in the non-transformed space we can apply $A$ to get the desired action
- Then we put back the result in the transformed space by applying $T$
This interpretation allows to clearly understand and remember why $T^{-1}$ must come before $T$.
I was really convinced by my own reasoning, but then I realized that there is a huge problem with my line of though: lets take $T$ to be a rotation, and lets also take $A$ to be some operator involving rotations. Problem now is that rotations do not commute, rotating in one way and then in another way is not the same as doing the opposite, and so my interpretation on why (1) works breaks apart badly! So is my interpretation of (1) wrong? Or does (1) not work with rotations? Seems strange..
In the case of my explanation of (1) being wrong: is there some other useful interpretation of (1)?
This topic has also huge physical implications: for example in quantum mechanics we apply the parity operator $P$ to other hermitian operators all the time, and it would be nice to understand why the formula should be $PAP^{-1}$ and not $P^{-1}AP$. (Maybe using parity as an example is not a great idea since $P=P^{-1}$, but come on you get what I am saying here!)
Let $T\colon V\to W$ be a vector space isomorphism and $A\colon V\to V$ some endomorphism of $V$. Let $B = TAT^{-1}$. (I changed your notation since writing $'$ in $A'$ might be a bit cumbersome.) It is immediate that $B$ is an endomorphism of $W$. In a sense, $B$ does to $W$ what $A$ does to $V$. In a sense, they are "the same". I will try to explore what "the same" means and as we will see, it will depend on what $T$ itself is.
Are $A$ and be $B$ the same same?
No, $A$ and $B$ are not literally the same: it is not true that $A=B$. The primary reason is that $V\neq W$. Even when $V = W$, they are still not the same most of the time because $T$ did do something to $V$, and it is likely that $A$ "cares" about that change. In fact, $A = B$ if and only if $A = TAT^{-1}$, if and only if $AT = TA$, i.e. $A$ and $T$ commute. Since they won't do that in general, $A\neq B$ in general.
So, if they are not the same same, how "same" are they?
It is easy to check that $Av_1 = v_2 \implies B(Tv_1) = Tv_2$. Let me rephrase that in plain English: if $A$ sends $v_1$ to $v_2$, then $B$ sends $Tv_1$ to $Tv_2$. So, if we identify vector spaces $V$ and $W$ to be the same via the isomorphism $T$, then we can identify $A$ and $B$ to be the same via isomorphism $T$ as well. Let me express this idea formally. I will define a relation $\sim$ on $V$ and $W$. Let $v\in V$ and $w\in W$. Then, $$v\sim w \iff w = Tv.$$
Since $T$ is linear, $\sim$ will respect the linear structure of vector spaces $V$ and $W$, i.e. $$v_1\sim w_1,\ v_2\sim w_2 \implies \alpha v_1 + \beta v_2 \sim \alpha w_1 + \beta w_2.$$
Since $T$ is an isomorphism, in particular, it is a bijection, so for any $v\in V$ there is exactly one $w\in W$ such that $v\sim w$, and vice versa. The above construction is what it means to identify vector spaces $V$ and $W$ via $T$.
This identification let's us formally express how $A$ and $B$ are "the same" and it's the following: $$v\sim w \implies Av\sim Bw.$$ In plain English, if $v$ and $w$ are "the same", then applying $A$ and $B$ to them will give us "the same" thing.
This is easy to verify: $$v\sim w \implies w = Tv \implies Bw = BTv = TAv \implies Av \sim Bw.$$
Furthermore, we can also easily see that the implication $v\sim w\implies Av\sim Bw$ is true for all $v\in V$ and $w\in W$ if and only if $B = TAT^{-1}$. This tells us that if we idenitfy $V$ and $W$ via $T$, the correct way to idenitfy the corresponding spaces of endomorphisms $\operatorname{End}(V)$ and $\operatorname{End}(W)$ via $T$ is by the formula $B = TAT^{-1}$.
But, what does this actually mean for the operators $A$ and $B$?
Well, the first thing they have in common is that the matrix representations of $A$ and $B$ will be the same, but let me be more precise about this. Let $S=\{e_1,\ldots,e_n\}$ be a basis for $V$. Since $T$ is an isomorphism, $T(S)=\{Te_1,\ldots,Te_n\}$ is a basis for $W$. Then the matrix coefficients of $A$ in basis $S$ will be the same as matrix coefficients of $B$ in basis $T(S)$. Let us show this. Assume that $Ae_j = \sum_{i=}^n \alpha_{ij}e_i$. Then, $$B(Te_j) = TAe_j = T(\sum_{i=}^n \alpha_{ij}e_i) = \sum_{i=}^n \alpha_{ij}(Te_i).$$ In plain English, $B$ does to the basis vectors $T(S)$ literally the same thing as $A$ does to the basis vectors $S$. However, doing "the same thing" doesn't mean the result will literally be the same, it will be the same in the sense of the above relation $\sim$.
Another important thing to notice, whether through above consideration of matrix representations or directly, $A$ and $B$ have the same eigenvalues. However, $A$ and $B$ don't have the same eigenvectors. But, again, it is easy to establish the connection between their eigenvectors: $$Av = \lambda v \implies B(Tv) = TAv = \lambda Tv,$$ so, if $v$ is an eigenvector of $A$ with eigenvalue $\lambda$, then $Tv$ is an eigenvector of $B$ with the same eigenvalue $\lambda$. In conclusion, if $V_\lambda(A)\leq V$ and $W_\lambda(B)\leq W$ are the eigenspaces of $A$ and $B$ with eigenvalue $\lambda$, then $$W_\lambda(B) = TV_\lambda(A).$$
In particular, if $v\in V$ is a fixed point of $A$, i.e. $Av = v$, then $Tv$ is a fixed point of $B$, i.e. $B(Tv) = Tv.$
Another direct consequence is that $A$ and $B$ will have the same determinant and trace, characteristic polynomial, minimal polynomial etc.
A concrete example.
All of the above might look like just some formalistic considerations, but it is quite useful to identify different, but isomorphic spaces $V$ and $W$ since we might be more familiar with one than the other.
Let $V = \mathcal P_n$, the real vector space of polynomials in one variable of degree less than or equal to $n$. Let us consider endomorphism $A$ of $V$ given by differentiation, i.e. $$A(a_0+a_1x+a_2x^2+\ldots+a_nx^n) = a_1 + 2a_2x + \ldots + na_nx^{n-1}.$$ Now, let $W = \mathbb R^{n+1}$. Let $T\colon V\to W$ be given by $$x^k\mapsto (0,\ldots,0,1,0,\ldots,0),$$ where $1$ occurs at the $(k+1)$-th coordinate, and all the others are $0$. $T$ maps the basis $\{1,x,x^2,\ldots, x^n\}$ to the canonical basis for $\mathbb R^{n+1}$, so it is an isomorphism. If we define $B = TAT^{-1}$, then we can easily see that $$B(a_0,a_1,\ldots,a_n) = (a_1,2a_2,\ldots,na_n,0).$$
Hopefully, comparing the formulas for $A$ and $B$ will tell you how "same" $A$ and $B$ are.
A confusing (?) example.
Let $V = W = \mathbb R^2$. Let $A(a,b) = (2a+b,b-a)$ and $T(a,b) = (a,a+b).$ Quick check gives us that $T^{-1}(a,b) = (a,b-a)$. If we let $B = TAT^{-1}$, then we can quickly calculate that $B(a,b) = (a+b,2b-a).$
Now, clearly, $A$ and $B$ look completely different. However, let $e_1 = (1,0)$, $e_2 = (0,1)$, and let $f_1 = Te_1$ and $f_2 = Te_2$. I'll leave it to you to confirm for yourself that $$A(ae_1 + be_2) = (2a+b)e_1 + (b-a)e_2,\\ B(af_1 + bf_2) = (2a+b)f_1 + (b-a)f_2.$$
So, really, $A$ and $B$ are the same as much as $e_1$ and $e_2$ is the same as $f_1$ and $f_2$.
Hopefully, the above two examples show that thinking $A$ and $B$ to be the same is only as useful as isomorphism $T$ is useful for indentifying vector spaces $V$ and $W$.