Taylor expansion of distance along geodesics

Question

On a complete Riemannian manifold $(M,g)$ without boundary, let $x,y\in M$ and $u\in T_xM, v\in T_yM$ be fixed. Let $\gamma_1$ be the geodesic starting at $x$ with initial velocity $u$ and $\gamma_2$ be the geodesic starting at $y$ with initial velocity $v$. I am looking for a Taylor expansion of $d^2(\gamma_0(t),\gamma_1(t))$ around $t=0$, where $d$ is the Riemannian distance. When $x=y$, it is fairly easy, using Jacobi fields that have a simple expression. When $x\neq y$, I am having some trouble. I am mostly curious to see what the first terms should depend on. Thanks!