In many books and articles I sometimes see similar matrix constructs, multiplication of the matrix transpose (or inverse), some other matrix, and then the first matrix itself, like this: $M^T AM$ or $M^{-1} A M$.
For square matrices $M$ I've found an example of definition of matrix congruence, that is, existence of certain matrices $A,B,P$ related so that $P^T A P = B$ names $A$ and $B$ congruent. However, this only applies to square $A,B$ and invertible $P$. Also I'm not aware of practical usage of this concept.
Again, for square $A, B, P$ there exists a concept of matrix similarity, that is, relation of $B=P^{-1}AP$ names $A$ and $B$ similar. I'm more familiar with this concept since it's frequently used in computer graphics when $B$ acts as "base" model-view matrix, and series of $P_i$ act as a set of modelling matrices for series of objects in a scene.
However, these "tricks" for non-square matrices are also frequently used in literature, for example, in normal equations for non-linear least squares methods and algorithms.
The question is: what are the informal explanations of $M^T AM$ or $M^{-1} A M$ matrix "tricks" and how they are used in applications?
There’s no “trick” here. These all represent a change of basis (coordinate system) for various objects that the matrix $A$ might represent. The matrix $M$ converts coordinates of a vector from one basis to another.
When $A$ represents a linear transformation, the appropriate formula is $M^{-1}AM$. The matrix $M$ converts from some coordinate system into the one that $A$ is expecting, and $M^{-1}$ converts its output back into the original coordinate system. Algebraically, if we have $x=Mx'$ and $y=My'$, then from $y=Ax$ we have $My'=AMx'$, and multiplying both sides by $M^{-1}$ produces $y'=(M^{-1}AM)x'$.
Similarly, $M^TAM$ is a change of basis when $A$ represents a bilinear or quadratic form. Here, we’re working with an expression of the form $x^TAy$, from which $x^TAy = (Mx')^TA(My) = x'^T(M^TAM)y$. You might also see $M^TAM$ when $A$ represents a linear transformation, but that’s a special case of $M^{-1}AM$ when $M^{-1}=M^T$, which occurs when the change of basis is achieved via an orthogonal transformation.