This is just usual composition of functions: for all $f$ in an appropriate domain
$(AB)(f)=A(B(f))=(\partial^2_x+u)(u\partial_x f)=\partial^2_x (u\partial_x f)+u^2\partial_x f=\cdots$ using the chain rule, and similarily for $BA$. Notice that $u$ acts as a multiplication operator. Apart from that, it is totally crucial for composition that you specify both domain and codomain of $A$ and $B$ and in particular what is assumed about $u$ (say domain and codomain is $C^\infty$, and $u$ be also of class $C^\infty$).
Related Questions in ORDINARY-DIFFERENTIAL-EQUATIONS
This is just usual composition of functions: for all $f$ in an appropriate domain $(AB)(f)=A(B(f))=(\partial^2_x+u)(u\partial_x f)=\partial^2_x (u\partial_x f)+u^2\partial_x f=\cdots$ using the chain rule, and similarily for $BA$. Notice that $u$ acts as a multiplication operator. Apart from that, it is totally crucial for composition that you specify both domain and codomain of $A$ and $B$ and in particular what is assumed about $u$ (say domain and codomain is $C^\infty$, and $u$ be also of class $C^\infty$).