In the case of a differential equation on the tangent space of a manifold (that is solutions are vector fields) $$ \dot x = A(x) $$ with the Cauchy condition $ x(t)=x_t$, we often denote the solution at time $0$ as $$ x(0;(t,x_t))=e^{-t\, \text{ad}\, A}(x_t). $$
Then I saw the following expansion formula $$ e^{-t\, A}(x_t)=\sum_{k=0}^{+\infty} \frac{(-t)^k}{k!}\,\text{ad}^k A\cdot x_t ? $$
What is the ad operator ?
It seems that in the first expression, it is just a notation but in the second, is it related to Lie brackets ? ...
Could you provide a reference where I could more details ?
To be more precise, here is what I read:
So if you could provide me some books where I could find such ad-formula.
Frankly, your convention is confusing/nonstandard (at least in physics and engineering).
The standard integration of linear diff eqn systems goes through matrix exponentials involving multiplication, not adjoint maps involving Lie commutators.
That is, $$ \dot x= A x \qquad \Longrightarrow x(t)= e^{tA} ~x_0 \qquad \Longrightarrow x(0)=x_0=e^{-tA} x_t, $$ and not an exponent involving commutators.
If you stick x along the diagonal of a square matrix with the dimension of A, call it X, then $$ \operatorname{ad}A ~ X \equiv [A,X], $$ and your expression $$ X(t)= e^{t\operatorname{ad}A} ~ X_0 = X_0+t[A,X_0]+t^2[A,[A,X_0]]/2!+ ... =e^{tA} X_0 e^{-tA} $$ indeed solves the matrix differential equation $$ \dot X= \operatorname{ad}A ~X = [A,X] $$ instead, not quite your original differential equation! (Check it/prove it!).
Now, to be sure, if we operated these matrix equations on a vector in the kernel of A, they may formally be identified with your original one (check this!), but there is something missing or implicit in your layout...