Let $f: \mathfrak{gl}(n,\mathbb{R})\to \mathbb{R}$ be a polynomial such that $f(AXA^{-1})=f(X)$ for all $A\in GL(n,\mathbb{R})$. Prove that there is a polynomial $g(x)\in\mathbb{R}[x_1,..., x_n]$ such that $f(X) = g(a_0, . . . , a_{n−1})$ where $a_i$ is the coefficient of $λ^i$ in the characteristic polynomial of $X$.
I have been thinking about this problem for a long time and I still can't think of anything to do, I would like someone to give me a suggestion or tell me how I can easily solve this problem.
Assuming that $f$ is a polynomial mapping, this statement is closely related to the theory of invariant polynomials. If instead of $\mathfrak{gl}_n$ we were looking at a (split) semisimple $\mathfrak g$, this would be a classical description of the $\mathfrak g$-invariant elements of the symmetric algebra $S(\mathfrak g)$ often credited to Chevalley (cf. MO 182871) which further would be shown to be the elements of the symmetric algebra $S(\mathfrak h)$ invariant under the Weyl group: and in the case of $\mathfrak{sl}_n$, this are the polynomials in $n-1$ variables which are invariant under permutation of the variables, i.e. the symmetric polynomials, i.e. what comes out here modulo the centre. (And Harish-Chandra linked that further to the centre of the universal enveloping algebra of $\mathfrak g$, but that's not important right now.)
But here's a handwavy proof for the case at hand. First notice that via conjugation it suffices to see how $f$ looks like on matrices in Jordan normal form. Then notice that e.g. the Jordan block
$\pmatrix{\lambda &1\\0&\lambda}$ via conjugation with $A:= \pmatrix{\varepsilon &0\\0&1}$ is brought to $\pmatrix{\lambda &\varepsilon\\0&\lambda}$, so by letting $\varepsilon \to 0$ via continuity (polynomial mapping!) we can further restrict to defining $f$ on diagonal matrices. (There's a complication here because over $\mathbb R$, we cannot even assume that every eigenvalue $\lambda \in \mathbb R$, so there are some technicalities to be filled in. However, one can prove quite generally for any Jordan-Chevalley decomposition $s+n$ that polynomial mappings are $0$ on the nilpotent part $n$.)
So now we assume our polynomial mapping $f$ is defined on the diagonal matrices, i.e. it is basically a polynomial in $n$ variables (one for each entry of the diagonal). But via conjugation with permutation matrices, the entries get permuted, i.e. our polynomial has to be invariant under all permutations of the $n$ variables. (Note that this is exactly the same as looking at $S(\mathfrak h)^W$, with $\mathfrak h$ here being the diagonal matrices, as in the general theory.) By definition, these are the symmetric polynomials, and it's known at least since Newton that these are all polynomials in the elementary symmetric polynomials, which evaluated on the eigenvalues of our matrix are also just the coefficients occuring in the characteristic polynomial.