I'm working this problem from Howe's An Invitation to Representation Theory:
Let $G = (\mathbb{R}^*, \times)$ be the group of non-zero real numbers under multiplication, and let $P_2$ denote the vector space of polynomials (in one variable) of degree two or less. Verify that the action of $G$ on $P_2$ given by $a\mathbin{.}p(x) = p(a^{-1}x)$ is a representation of $G$ on $P_2$.
I'm not sure where to start because I'm having trouble differentiating the group action from the representation. Any advice?
A representation of a group $G$ on a $\mathbb{k}$-vector space $V$ is a homomorphism $$ \rho: G \to \operatorname{GL}(V), $$
Explicitly, the conditions that make this map a homomorphism are $$ \rho(1)(v) = v \quad\text{and}\quad \rho(gh)(v) = \bigl(\rho(g) \circ \rho(h)\bigr) (v). \tag{A} $$ for all $g, h \in G$ and all $v \in V$ or in coordinates,
In your example, $G = \mathbb{R}^*$ and $V = P_2 = P_2(\mathbb{R})$ and we can check that the proposed map $\rho: \mathbb{R}^* \to \operatorname{GL}(P_2)$, given by $a \mapsto \rho_a$, where $\rho_a: P_2 \to P_2$ for each $a \in \mathbb{R}^*$ is the scaling transformation $p(x) \mapsto p(a^{-1}x)$, satisfies the required properties:
(0) $\rho_a$ is linear and has appropriate source and target. Certainly, it's well-defined since $a \neq 0$, and we can verify that $p(a^{-1}x)$ is still a polynomial in $x$ of degree at most $2$. Then you have to check linearity.
(1) $\rho_a$ is a group action. These are the properties in $(A)$. The only potential subtlety is with the second property:
Since $G$ is abelian, writing $(ab)^{-1}$ as $b^{-1}a^{-1}$ might seem superfluous, but it's what makes the calculation work, which also reveals that it's the right kind of formula for similar situations where the group isn't abelian.
By the way, if we have a basis $\{v_1, \dots, v_n\}$ chosen for $V$, we can do this all in coordinates. The representation is equivalent to a homomorphism $$ \rho: G \to \operatorname{GL}_n(\mathbb{k}), $$ producing invertible $n \times n$ matrices for each $g \in G$. Let's use the notation $\rho_g := \rho(g)$, so we require the matrices to satisfy $$ \rho_1 = I_n \quad\text{and}\quad \rho_{gh} = \rho_g\,\rho_h. $$
We can also work out the example in coordinates since $P_2$ has a nice basis given by monomials: $\{1, x, x^2\}$. We need to construct for each $a \in \mathbb{R}^*$, the $3 \times 3$ matrix representing pre-multiplication by $a^{-1}$. The action on each basis vector is
hence for each $a \in \mathbb{R}^*$, the matrix $\rho_a$ is
and we can easily check that these matrices satisfy the required identities.