There is natural Lie group representation of $GL(n)$ on $C^\infty(\mathbb{R}^n)$ given by \begin{align} \rho: GL(n) & \rightarrow \text{End}(C^\infty(\mathbb{R}^n)) \\ A & \rightarrow \left( \rho(A):f \rightarrow f\circ A^{-1} \right) \end{align} What is the associated lie algebra representation?
It should be tangent map of $\rho$ at unit element. So take path $A(t)$ in $GL(n)$, that $A(0)=I$, $A'(0)= X$. And start differentiating $$ \frac{d}{dt}\Bigg|_{t=0} f(A^{-1}(t)x) = -f'(x)Xx $$
This gives me $\rho^*(X)(f(x)) = - f'(x)Xx$. But than $\rho([X,Y]) \neq [\rho(X),\rho(Y)]$ because $\rho([X,Y])(f)$ would contain first derivatives of $f$ but $[\rho(X),\rho(Y)](f)$ would contain second derivatives of $f$.
I have one more question. Do I get into the trouble when I consider representation on $C(\mathbb R^n)$(i.e. dropping the smoothness)? Is it still sensible to ask for Lie algebra representation?
Ok based on the encouragement from answer, that I got it right. I went through the calculation. Here it is:
\begin{align} \rho(X)(f(x)) &= - (\partial_i f(x)) X_{ij} x_j \\ \rho(Y)\rho(X)(f(x)) &= \partial_k ((\partial_i f(x)) X_{ij} x_j) Y_{kl}x_l \\ &= (\partial_k \partial_i f(x)) X_{ij} Y_{kl} x_j x_l + (\partial_i f(x)) X_{ik} Y_{kl} x_l \\ \rho(X)\rho(Y)(f(x)) &= (\partial_k \partial_i f(x)) Y_{ij} X_{kl} x_j x_l + (\partial_i f(x)) Y_{ik} X_{kl} x_l \\ &= (\partial_k \partial_i f(x)) X_{ij} Y_{kl} x_j x_l + (\partial_i f(x)) Y_{ik} X_{kl} x_l \end{align}
As you can see those second order term in $\rho(X)\rho(Y)(f(x))$ and $\rho(Y)\rho(X)(f(x))$ are the same and cancel out in the commutator $[\rho(X),\rho(Y)]$.
I had to use index notation in the proof because it seamed to me as the most reasonable choice, but still I do not like it. Another option is to write down the second order term as $$ (\partial_k \partial_i f(x)) Y_{ij} X_{kl} x_j x_l = x^T Y^T f''(x) X x $$ But this is not useful when I would do third derivatives.
Your computation is correct, but you were too pessimistic about the continuation: the second derivative terms in the bracket of $\rho(X)$ and $\rho(Y)$ cancel.
It becomes less reasonable to ask for a Lie algebra repn on not-smooth functions, since natural Lie algebra repns are differential operators. Nevertheless, once you start down that path, if you go all the way to the action on distributions and distributional derivatives, things become sensible again.