How do I find the characteristic polynomial and the eigenvalues and eigenvectors of the linear operator which defined by: $$T\left(A\right)=\:A^t$$
where T: $T\::\:R^{n\cdot n}\:->\:R^{n\cdot n}$
I tried to find the matrix which represents the transformation, but without any success: find such M which will give me the result: $$M*A=\:A^t$$
so I could do:
$$\left|M-\lambda I\right|=0$$
On
I tried to find the matrix which represents the transformation, but without any success: find such M which will give me the result: $$M∗A=A^t$$
I want to drill down on this, as this is a common error. You need to remember that the $n \times n$ matrices are the vectors here. The space $M_{n \times n}(\Bbb{R})$ is $n^2$-dimensional. The space of linear transformations on $M_{n \times n}(\Bbb{R})$ has $(n^2)^2 = n^4$ dimensions, and such transformations are represented by $n^2 \times n^2$ matrices.
How do we multiply such an $n^2 \times n^2$ matrix to an $n \times n$ matrix? We don't. Remember, linear operators associate with matrices, once you've chosen a basis. The matrix then multiplies not to the vector, but to the corresponding coordinate column vector with respect to the basis. The result will be the coordinate column vector with respect to the basis of the transformed vector (in this case the transposed matrix).
As a quick illustration, we can consider the $2 \times 2$ case. Nominate the standard basis: $$B = \left(\begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}, \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}, \begin{pmatrix} 0 & 0 \\ 1 & 0 \end{pmatrix}, \begin{pmatrix} 0 & 0 \\ 0 & 1 \end{pmatrix}\right) = (e_{11}, e_{12}, e_{21}, e_{22}).$$ Then, if we let $T$ be the transpose operator, \begin{align*} T(e_{11}) &= \begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}^\top = \begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix} = 1e_{11} + 0e_{12} + 0e_{21} + 0e_{22} \\ T(e_{12}) &= \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}^\top = \begin{pmatrix} 0 & 0 \\ 1 & 0 \end{pmatrix} = 0e_{11} + 0e_{12} + 1e_{21} + 0e_{22} \\ T(e_{21}) &= \begin{pmatrix} 0 & 0 \\ 1 & 0 \end{pmatrix}^\top = \begin{pmatrix} 0 & 0 \\ 1 & 0 \end{pmatrix} = 0e_{11} + 1e_{12} + 0e_{21} + 0e_{22} \\ T(e_{22}) &= \begin{pmatrix} 0 & 0 \\ 0 & 1 \end{pmatrix}^\top = \begin{pmatrix} 0 & 0 \\ 0 & 1 \end{pmatrix} = 0e_{11} + 0e_{12} + 0e_{21} + 1e_{22}. \end{align*} Putting these coordinates as columns, we get our matrix: $$\begin{pmatrix}1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1\end{pmatrix}.$$ If you want to use this matrix to transpose a matrix, say, $$A = \begin{pmatrix} 1 & 3 \\ 2 & 4 \end{pmatrix},$$ you would first express $A$ in terms of coordinates with respect to $B$: $$A = 1e_{11} + 3e_{12} + 2e_{21} + 4e_{22} \implies [A]_B = \begin{pmatrix} 1 \\ 3 \\ 2 \\ 4\end{pmatrix}.$$ We would then multiply this coordinate vector with the $4 \times 4$ matrix, to get $[T(A)]_B$: $$[T(A)]_B = \begin{pmatrix}1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1\end{pmatrix} \begin{pmatrix} 1 \\ 3 \\ 2 \\ 4\end{pmatrix} = \begin{pmatrix} 1 \\ 2 \\ 3 \\ 4\end{pmatrix},$$ so $$T(A) = 1e_{11} + 2e_{12} + 3e_{21} + 4e_{22} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix},$$ as expected. All this is a complicated way of saying, the real matrix you want to analyse, vis-a-vis eigenvalues/characteristic polynomials/determinant, is the given $4 \times 4$ matrix (in the case where $n = 2$).
So, that's not going to be fun to do in general. You'd need to find a pattern for the $n^2 \times n^2$ matrices, and somehow compute the determinant. Fortunately, there's an easier. You can follow Robert Israel's suggestion, and simply guess and check the eigenvalues and eigenvectors.
Because $T^2(A) = A = I(A)$ for all matrices $A$ (where $I$ is the identity transformation on $M_{n \times n}(\Bbb{R})$), we have that $T^2 - I = 0$. That is we have a polynomial $p(x) = x^2 - 1 = (x - 1)(x + 1)$ such that $p(T) = 0$, which implies $T$ is diagonalisable, and that $-1$ and $1$ are the only possible eigenvectors. (Moreover, because $p$ is square-free, we can actually deduce that $T$ is diagonalisable.)
You just need to figure out the dimensions of these eigenspaces. Once you know this, your characteristic polynomial will be $$(x - 1)^{d_1}(x + 1)^{d_{-1}},$$ where $d_1$ and $d_{-1}$ are the dimensions of the eigenspaces corresponding respectively to $1$ and $-1$.
Hint: Eigenvectors are the symmetric and antisymmetric matrices.