There is this rather hard linear algebra problem in my notes that the professor left as an exercise during the holidays if we have free time, and it goes:
Problem: Let $M_n(\mathbb{R})$ be the vector space of all $n \times n$ matrices over the real number line. For $A \in$ $M_n(\mathbb{R}),$ let $A^t$ denote the transpose of matrix $A$. Define the linear operators $\tau_A: M_n(\mathbb{R}) \rightarrow M_n(\mathbb{R})$ by $\tau_A(X) = AXA^t$. Prove that $\text{Tr}(\tau_A)$ is $\text{Tr}(A)^2$ and $\text{Det}(\tau_A)$ is $\text{Det}(A)^{2n}.$
Can anyone give me some start so I can get going with this problem? I would appreciate the help.
There are two basic approaches to this problem. One is to find the eigenvalues/eigenvectors of the transformation, and then to compute the trace/determinant using the product/sum (I think this is what Qiaochu has in mind). The other is to try to interpret the trace/determinant in the context of maps on $M_n(\Bbb R)$.
Hint: (For approach 1) Suppose that $A$ is diagonal. What do the eigenvalues/eigenvectors of $A$ look like? Consider in particular the case where all eigenvalues of $A$ are distinct. Now, what do the eigenvectors of $A$ look like if $A$ is diagonalizable?
Once you have the result for diagonalizable matrices, you can get the general result either using the continuity of trace/determinant, or by extending our argument above using Jordan form.
Hint: (For approach 2) We may compute the trace of a map $\tau:M_n(\Bbb R)\to M_n(\Bbb R)$ as $$ \sum_{i,j = 1}^n \operatorname{trace}(E_{ij}^T \tau(E_{ij})) $$ where $E_{ij}$ denotes the matrix with a $1$ in the $i,j$ entry and zeros elsewhere.
To compute the determinant, it is helpful to consider this map as a composition. Namely, $\tau_A = \tau^{(1)}_A \circ \tau^{(2)}_A$, where $$ \tau^{(1)}(X) = AX, \qquad \tau^{(2)}(X) = XA^T $$ If you find the matrices of $\tau^{(1)}$ and $\tau^{(2)}$ relative to the basis $\{E_{ij}: 1 \leq i,j \leq n\}$ (taken in lexicographical order), then you will find that it is easy to compute $\det \tau^{(1)}$ and $\det \tau^{(2)}$ by exploiting the block-structure of these matrices.