According to terminologies in "Invariant theory" a true definition for an invariant function $f:M_{n}(\mathbb{R})\to \mathbb{R}$ is the following:
Definition 1: A continuous function $f$ is invariant if $f(A)=f(P^{-1}AP)$ for every $P\in Gl_{n}(\mathbb{R})$
But I found the following alternative definition in some papers:
Definition 2: A continuous function $f$ is invariant if $f(AB)=f(BA)$ for all $A,B \in M_{n}(\mathbb{R})$
Are these two properties equivalent for a continuous $f$?
Yes, they are.
$1\implies 2$:
If $B$ is invertible, then it is a member of $GL_n(\mathbb C)$, therefore according to definition $1$, $$f(BA) = f(B^{-1}BAB) = f(AB)$$
If $B$ is not invertible, then you can find a sequence $B_n$ of invertible matrices that converges to $B$ (for example, $B-\alpha_n I$ where $\alpha_n$ converges to zero and avoids the eigenvalues of $B$), and then due to continuity and the result above, $$f(BA) = \lim_{n\to\infty} f(B_nA) = \lim_{n\to\infty} f(AB_n) = f(AB)$$
$2\implies 1$:
Here we don't even need continuity: $$f(P^{-1}AP) = f(APP^{-1}) = f(A)$$