Problem. I'm working on an exercise from a text in Riemannian geometry, that tells me to do this:
Let $\mathbf{O}(n)=\{A\in\mathbb{R}^{n\times n}:A^t A=I\}$ be equipped with its usual (left-invariant!) Riemannian metric $g_p(X_p,Y_p)=\operatorname{tr}(X_p^t Y_p)$ for $p\in \mathbf{O}(n)$ and $X_p,Y_p\in T_p\mathbf{O}(n)$.
Show that for a $C^2$-curve $\gamma\colon I\to \mathbf{O}(n)$, it holds that $\gamma$ is geodesic if and only if $\gamma^t \ddot{\gamma}=\ddot{\gamma}^t \gamma$.
I think I solved it while trying to explain where I was stuck (it's always surprising how helpful it can be to just try to formulate a question), so I will post my suggested solution as an answer to this post, in case somebody else needs it or somebody else has something to add.
Solution. By definition, $$\text{$\gamma$ is a geodesic $\iff$ $\tilde{\nabla}_\dot{\gamma}\dot{\gamma}=0$}\,,$$ where $\tilde{\nabla}$ is the Levi-Civita connection of $\mathbf{O}(n)$.
Since $\mathbf{O}(n)$ is a submanifold of $\mathbb{R}^{n\times n}$, and $g$ is induced by the usual metric on $\mathbb{R}^{n\times n}$, it holds that $$\tilde{\nabla}_\dot{\gamma}\dot{\gamma}=(\nabla_\dot{\gamma}\dot{\gamma})^\top=(\partial_\dot{\gamma}\dot{\gamma})^\top=\ddot{\gamma}^\top,$$ where $\nabla=\partial$ is the Levi-Civita connection of $\mathbb{R}^{n\times n}$ and $\small \top$ denotes the tangential part. This means that $$\tilde{\nabla}_\dot{\gamma}\dot{\gamma}=0\iff \ddot{\gamma}^\top=0\,.$$ Next, note that $\ddot{\gamma}(s)^\top=0$ is equivalent to $\ddot\gamma(s)\perp T_{\gamma(s)}\mathbf{O}(n)$, which just the same as saying $$\ddot\gamma(s)\in N_{\gamma(s)}\mathbf{O}(n)=\{\gamma(s)Y:Y\in\mathrm{Sym}(\mathbb{R}^n)\}$$ [cf. the end of Ch. 5 here]. Using that $\gamma(s)\in\mathbf{O}(n)$, this can be rewritten as $\gamma(s)^t\ddot\gamma(s)\in \mathrm{Sym}(\mathbb{R}^n)$, from which we finally conclude that $$\ddot\gamma^\top =0 \iff \gamma^t \ddot{\gamma}=(\gamma^t \ddot{\gamma})^t=\ddot{\gamma}^t \gamma.\quad\square$$
Remark. As was pointed out in the comments, we can use this result to show that the geodesics of $\mathbf{O}(n)$ are exactly its one-parameter subgroups composed with left translations, i.e. curves of the form $s\mapsto p\exp(sX)$ for $p\in \mathbf{O}(n)$ and $X\in T_e\mathbf{O}(n)$.
We first show that all such curves are geodesics. Let $p\in \mathbf{O}(n)$ and $X\in T_e\mathbf{O}(n)$ and form $\gamma\colon \mathbb{R}\to \mathbf{O}(n)$ defined by $\gamma(s)=p\exp(sX)$. By using the fact that $p^tp=e$ and $(X^2)^t=(X^t)^2=(-X)^2=X^2$, it's easy to show that $$\gamma(s)^t\ddot\gamma(s)=\exp(sX)^t X^2\exp(sX)= \ddot\gamma(s)^t\ddot\gamma(s)\,,$$ and hence that $\gamma$ is a geodesic.
We now show that all geodesics of $\mathbf{O}(n)$ are (at least locally) of this form. For this end, let $\alpha\colon I\to \mathbf{O}(n)$ be an arbitrary geodesic with $0\in I$ and $\alpha(0)=p$. Since $\alpha$ is a curve in $\mathbf{O}(n)$, we must have $\dot\alpha(0)\in T_p\mathbf{O}(n)$, i.e. $\dot\alpha(0)=pX$ for some $X\in T_e\mathbf{O}(n)$. But by the uniqueness of geodesics, this must mean that $\alpha$ is locally equal to the geodesic $\gamma\colon s\mapsto p\exp(sX)$, since $\gamma(0)=\alpha(0)$ and $\dot\gamma(0)=\dot\alpha(0)$. $\quad\square$