I made an observation last week, and it has bothered me ever since.
Recall the formulae $$\exp([X,Y])=\lim_{n\rightarrow\infty}\left(\exp\left(\frac{1}{n}X\right)\exp\left(\frac{1}{n}Y\right)\exp\left(\frac{-1}{n}X\right)\exp\left(\frac{-1}{n}Y\right)\right)^{n^2}$$ and $$[X,Y]_p(f)=\lim_{t\rightarrow 0}\frac{1}{t^2}\Bigg(f\Big(\!\exp(-tY)\exp(-tX)\exp(tY)\exp(tX)p\Big)-f(p)\Bigg).$$
The observation is this: the first formula "moves clockwise", while the second formula "moves counterclockwise"!
This led me back to definitions.
Suppose we have a Lie group $G$ and we want to define the Lie algebra of $G$. We can define the Lie algebra using the adjoint representation, with $X_e,Y_e\in T_eG$ and $$[X_e,Y_e]=\mathrm{ad}_{X_e}(Y_e)=\lim_{t\rightarrow 0}\frac{1}{t}(L_{\exp(tX_e)*}R_{\exp(-tX_e)*}Y_e-Y_e),$$ or with left-invariant vector fields $X$ and $Y$ with $$[X,Y]_e=\mathcal{L}_X(Y)_e=\lim_{t\rightarrow 0}(\exp(-tX)_*Y_{\exp(tX)e}-Y_e).$$
I believe that I am much less proficient in the latter than the former, so perhaps this stems from lack of practice, but it looks like these are not the same Lie algebra, in the sense that the identity map on $T_eG$ is not a Lie algebra isomorphism. In fact, it looks like the identity map would be an anti-isomorphism, so that $[X_e,Y_e]=-[X,Y]_e$. Of course, anti-isomorphic Lie algebras are isomorphic, but they are not "the same" in the sense above.
Today, I tested this on the matrix Lie group $\left\{\begin{bmatrix}a & b \\ 0 & 1\end{bmatrix}\!:a,b\in\mathbb{R}\text{ and } a>0\right\}$, but they came out to the same algebra.
Why do these two definitions "move in opposite directions" in the sense above, and how do they determine the same Lie algebra?
I am looking for a detailed explanation, preferably including
- whether or not these two definitions are anti-isomorphic
- relationships between the one-parameter subgroups $\exp(tX)$ and the flow curves $\exp(tX)e$
- a little bit of geometric intuition
If one takes the Lie bracket of two vector fields defined as the commutator (thinking of vector fields as derivations on the commutative algebra of smooth functions), this is exactly minus the infinitesimal counterpart of the adjoint action of diffeomorphisms (change of coordinates).
In complicated language. Let $M$ be a manifold (smooth, Hausdorff, paracompact, and connected... even if not all of this is necessary), denote the diffeomorphism group of $M$ by $\mathrm{Diff}(M)$ and the $C^\infty(M)$-module of vector fields by $\mathfrak{X}(M;\mathbb{R})$.
The adjoint action by a diffeomorphism $\phi\in\mathrm{Diff}(M)$ on a vector field $X\in\mathfrak{X}(M;\mathbb{R})$ is the mapping $\mathrm{Ad}_\phi:\mathfrak{X}(M;\mathbb{R})\to\mathfrak{X}(M;\mathbb{R})$ given by $\mathrm{Ad}_\phi(X)(f):=X(f\circ\phi)\circ\phi^{-1}$ for every $f\in C^\infty(M;\mathbb{R})$.
The infinitesimal counterpart of the adjoint action, $\mathrm{ad}:\mathfrak{X}(M;\mathbb{R})\to\mathrm{End}_{\mathbb{R}}(\mathfrak{X}(M;\mathbb{R}))$, defined by $(\mathrm{ad}_X(Y)(f))(p):=\frac{\mathrm{d}}{\mathrm{d}t}(\mathrm{Ad}_{\exp(tX)}(Y)(f))(p)\Big{|}_{t=0}$, for every $X,Y\in\mathfrak{X}(M,\mathbb{R})$, $f\in C^\infty(M;\mathbb{R})$, and $p\in M$, endows $\mathfrak{X}(M,\mathbb{R})$ with a Lie algebra structure.
The claim is that $\mathrm{ad}_X(Y)(f)=-[X,Y](f)=Y\circ X(f)-X\circ Y(f)$. One can prove this by applying Taylor's theorem to the function $\mathbb{R}\ni t\mapsto f\circ\exp(tX)(q)\in\mathbb{R}$, where $q\in M$ and $\exp(tX)\in\mathrm{Diff}(M)$ denotes the flow of $X\in\mathfrak{X}(M;\mathbb{R})$ at time $t\in\mathbb{R}$.
Now, if $M$ is a Lie group and one takes the commutator of two left invariant vector fields evaluated at the identity to be the Lie algebra structure on the tangent space at the identity, one has the definition $[X,Y]_e$ described by the Original Poster.
The conjugation action of a Lie group $(G,\cdot)$ is the action $\mathrm{A}:G\to\mathrm{Diff}(G)$ that associates to each $g\in G$ a diffeomorphism $A_g\in\mathrm{Diff}(G)$ given by $A_{g}(g_1):=g\cdot g_1\cdot g^{-1}$, for all $g_1\in G$. And for each $g\in G$ the pushforward at the identity of $\mathrm{A}_g\in\mathrm{Diff}(G)$, $\mathrm{Ad}_g:={\mathrm{A}_g}_{*_e}:\mathfrak{g}\to\mathfrak{g}$, defines the adjoint action $\mathrm{Ad}:G\to\mathrm{Aut}_{\mathbb{R}}(\mathfrak{g})$. Here I am using the (usual) notation $\mathfrak{g}:=T_eG$, and $\mathrm{ad}:=\mathrm{Ad}_{*_e}:\mathfrak{g}\to\mathrm{End}_{\mathbb{R}}(\mathfrak{g})$, endows $\mathfrak{g}$ with a Lie algebra structure: which not only agrees with the definition $[X_e,Y_e]$ described by the Original Poster, but agrees with the infinitesimal counterpart of the adjoint action previously defined for any manifold.
This might be the source for the sign disagreement between the two definitions given by the Original Poster. If not, at least it is an explanation of why there are more than one convention for the sign of the Lie bracket between vector fields.
P.S.: The Lie derivative is also known as the fisherman's derivative, which I believe is a perfect nomenclature fitting Qiaochu Yuan's comment.