I'm pretty much stuck on the following question (taken from the book Lie groups and introduction to linear groups by Rossman W.):
I've found some clues, but I think I lack proper understanding of what needs to be done here.
If I recall correctly, the relation between lie groups homomorphism $T$ and its lie algebra homomorphism $\tau$ is $\tau(X) = \frac{d}{dt} T(e^{tX})|_{t=0}$ when in this case $ X \in gl(n,\mathbb{R})$. But then, some factor involving the exponent function should appear. On the other hand, the derivative of $f(x)$ is $$\displaystyle \sum_{i} \xi_i ' \frac{\partial f}{\partial\xi_i}$$, so it should be some kind of chain rule?
In (b) I have similar problem - it seems like I need to "go down" to $so(3)$ and use its basis matrices, but I'm not sure how to justify it.
Any assistance will be highly appreciated.
It is exactly all about chain rule : $\tau(Y)f(x) = \dfrac{d}{dt}_{\vert t=0}T(e^{tY})f(x)=\dfrac{d}{dt}_{\vert t=0}f(e^{-tY}x)$.
If you write $\phi(t)=e^{-tY}x$ then $$\phi'(t)=-Yx=-\sum_{i,j=1}^nY_{ij}\xi_je_i$$ and you get $$\tau(Y)f(x) = df_{x}(-Yx)=-\sum_{i,j=1}^n\frac{\partial f}{\partial \xi_i}(x)Y_{ij}\xi_j.$$
You can see $\tau$ as the differential of $T:GL(n,\mathbb R)\rightarrow GL(F)$ at the identity element $I_n$.
Saying that $f\in \mathbb R[\xi_1,\xi_2,\xi_3]$ is invariant under the action of $SO(3,\mathbb R)$ means that the restriction of $\tau$ to the Lie algebra of $SO(3,\mathbb R)$ is the null linear map since $$\forall X\in so(3), \forall t\in \mathbb R, \quad T(e^{tX})f(x)=f(x).$$
And as you said, the Lie algebra $so(3)$ has a natural basis $$so(3)=<\begin{pmatrix} 0&0&1\\0&0&0\\-1&0&0\end{pmatrix},\begin{pmatrix} 0&1&0\\-1&0&0\\0&0&0\end{pmatrix},\begin{pmatrix} 0&0&0\\0&0&1\\0&-1&0\end{pmatrix}>=<A,B,[A,B]>$$ If you check fomula of question $(a)$ with one these three matrices, you will get exactly the expected three relations.