I am reading the following paper: https://arxiv.org/pdf/math/0211450.pdf (p.7) (Reading that page from "In several application domains" is pretty enough.)
In particular, the following part:
My understanding is:
$\rho(g)$ is an element in $GL(\mathbb{R}^N)$, $\rho(g)$ should act on a vector $v\in \mathbb{R}^N$, also can be written as $\rho_g(v)$. However, $X\in \mathcal{S}^N$, which is a symmetric matrix not a vector.
I am confused about this concept.
You have that $\rho$ is a homomorphism from $G \to \mathrm{GL}(\mathbb{R}^{N})$. Now $\mathrm{GL}(\mathbb{R}^{N})$ is a group, which can give many different group actions. The natural action is on $\mathbb{R}^{N}$ by left multiplication, but it can also act on the set $\mathcal{S}^{N}$ of symmetric matrices by conjugation. You can verify that this does indeed provide an example of a group action.