Let $G$ be a Lie Group with a continuous representation $(V,\rho)$. Consider the corresponding Lie Algebra $\mathfrak{g}$. Also consider the space of smooth vectors $V^\infty$ of $V$. Now let $\rho'$ be given (for all $v\in V^\infty$) by
$\rho'(g)=\frac{d}{dt}|_{t=0}\rho(e^{tb})v$.
I am asked to prove that $\rho':\mathfrak{g}\longrightarrow V^\infty$ is linear and an action of $\mathfrak{g}$.
Questions:
1) To show $\rho'$ is an action of $\mathfrak{g}$, I have to show that $\rho'([a,b])=\rho'(a)\rho'(b)-\rho'(b)\rho'(a)$, where $[a,b]$ is the Lie bracket on $\mathfrak{g}$. How am I supposed to show this without explicitly knowing how $[a,b]$ is defined on $\mathfrak{g}$? Or can one find this explicitly?
2) Is the linearity of $\rho'$ trivial given the linearity of $\rho$? I am struggling to show this due to the former linearity being on $\mathfrak{g}$.
3) How does one show that $V^\infty$ is invariant under $\rho'$? Similar to the previous question, I feel this should be immediate.
I would appreciate any answers or hints to any questions.
Edit: 4) Where is/is continuity used here?
Question (1): When the representation is a Hilbert representation, we can do as follows:
We know the space $C^{\infty}(G)$ of smooth functions on $G$ is a representation of the Lie algebra $\mathfrak{g}$. Now we try to reduce your problems to $C^{\infty}(G)$.
For any vector $\varphi_{0}\in V$, we can define a map $I_{\varphi_{0}}:V^{\infty}\rightarrow C^{\infty}(G), \varphi\mapsto \langle \rho(g)\varphi,\varphi_{0}\rangle$. Notice that $I_{\varphi_{0}}(\varphi)$ is a smooth function since $\varphi$ is a smooth vector.
For each element $g\in G$ and $\varphi\in V^{\infty},X\in\mathfrak{g}$, \begin{align} (\mathrm{d}X\circ I_{\varphi_{0}})(\varphi)(g)&=\frac{\mathrm{d}}{\mathrm{d}t}\Bigg|_{t=0}I_{\varphi_{0}}(\varphi)(g\cdot e^{tX})\\ &=\left\langle\frac{\mathrm{d}}{\mathrm{d}t}\Bigg|_{t=0}\rho(g\cdot e^{tX})\varphi,\varphi_{0}\right\rangle\\ &=\langle\rho(g)\rho^{\prime}(X)\varphi,\varphi_{0}\rangle=I_{\varphi_{0}}\circ\rho^{\prime}(X)(\varphi)(g). \end{align} This shows that $I_{\varphi_{0}}$ is $\mathfrak{g}$-equivariant.
In order to show that $\rho^{\prime}([X,Y])=[\rho^{\prime}(X),\rho^{\prime}(Y)]$, it suffices by dual to show that $$I_{\varphi_{0}}(\rho^{\prime}([X,Y])(\varphi))=I_{\varphi_{0}}(([\rho^{\prime}(X),\rho^{\prime}(Y)])(\varphi))$$ holds for each $\varphi_{0}\in V,\varphi\in V$. This is obviously true because we know the action of $\mathfrak{g}$ on $C^{\infty}(G)$ commutes with the Lie bracket.
Question (2): Yes, the linearity follows from that of $\rho$.
Question (3): $V^{\infty}$ is invariant under $\mathfrak{g}$ by the definition of $V^{\infty}$: $\varphi$ is smooth if it is $C^{k}$ for each $k$, then $\rho^{\prime}(X)\varphi$ is $C^{k}$ because $\varphi\in V^{\infty}$ is $C^{k+1}$.
Question (4): Of course you can still define $V^{\infty}$ and the action of $\mathfrak{g}$ when $\rho$ is not continuous, but in this case $V^{\infty}$ will not be dense in $V$.