The intuition behind the Riemann tensor $$ R(X,Y): \Gamma(TM) \to \Gamma(TM) $$ is that we parallel transport a vector $Z$ along the parallelogram-loop formed by the infinitesimal flows of vector fields $X$ and $Y$. To put this in mathematical language, if we have the parallel transport map $\tau_\alpha: T_{\alpha(0)}M \to T_{\alpha(t)}M$, then we define $$ R(X,Y)Z := \frac{d}{ds} \frac{d}{dt} \tau^{-1}_{sX} \tau^{-1}_{tY} \tau_{sX} \tau_{tY} Z \Big|_{s=t=0}. $$ My question is
How do we recover the standard formula $$ R(X,Y)Z = (\nabla_{X} \nabla_Y - \nabla_Y \nabla_X - \nabla_{[X,Y]})Z $$ from this definition?
I'm aware that $$ \frac{d}{ds} (\tau_{sX} Z)\Big|_{s=0} = \nabla_X Z $$ but I'm still having a difficult time deriving the above identity from first principles. Here's what I have so far but it's definitely riddled with mistakes: $$ \begin{aligned} R(X,Y)Z &= \frac{d}{ds} \frac{d}{dt} \tau^{-1}_{sX} \tau^{-1}_{tY} \tau_{sX} \tau_{tY} Z \Big|_{s=t=0} \\ &= \frac{d}{dt} \frac{d}{ds} \tau^{-1}_{sX} \tau^{-1}_{tY} \tau_{sX} \tau_{tY} Z \Big|_{s=t=0} \\ &= \frac{d}{dt} \Big( \frac{d}{ds} \Big( \tau^{-1}_{sX} \tau^{-1}_{tY} \tau_{sX} \tau_{tY} Z \Big) \Big|_{s=0} \Big) \Big|_{t=0} \\ &= \frac{d}{dt} \Big( \nabla_{-X} (\tau_{tY}^{-1} \tau_{sX} \tau_{tY}Z) \Big|_{s=0} \cdot \tau_{tY}^{-1}\nabla_X (\tau_{tY} Z) \Big) \Big|_{t=0} \\ &= \frac{d}{dt} \Big( \nabla_{-X} Z \cdot \tau_{tY}^{-1}\nabla_X (\tau_{tY} Z) \Big) \Big|_{t=0} \\ &= \nabla_{-X} Z \cdot \frac{d}{dt} \tau_{tY}^{-1}\nabla_X (\tau_{tY} Z) \Big|_{t=0} \\ &= \nabla_{-X} Z (\nabla_{-Y} \nabla_X (\tau_{tY} Z)\Big|_{t=0} \cdot \nabla_X \nabla_Y Z ) \\ &= \nabla_X Z (\nabla_Y \nabla_X Z \cdot \nabla_X \nabla_Y Z) \neq (\nabla_{X} \nabla_Y - \nabla_Y \nabla_X - \nabla_{[X,Y]})Z. \end{aligned} $$ Any pointers would be appreciated.