As I understand it, the exponential map allows us to locally compare a manifold $\mathcal{M}$ around a point $p\in\mathcal{M}$ and its tangent space $T_p\mathcal{M}$. This is done by associating to any tangent vector $X_p\in T_p\mathcal{M}$ to a point $q$ in $\mathcal{M}$ obtained by the following method : we know that there is a unique geodesic $\lambda(t)$ such that $\lambda(0)=p$ and $X_p$ is tangent to $\lambda$ in $t=0$. We can then take $q=\lambda(1)$, i.e. $q$ is then point obtained by going "one unit" along $\lambda$ starting from $p$. The parameter $p$ doesn't necessarily take values in all $\mathbb{R}$ so this only well-defined for some subset of tangent vectors around the origin in $T_p\mathcal{M}$.
The exponential map is a very important notion that we use non-stop when working with Lie groups (as they are manifolds and the tangent space to the identity is its corresponding Lie algebra). But the notion of geodesics depends on the connection that we take. In GR, it's clear that we take the Levi-Civita connection, defined from the metric. But in general, when we work with an arbitrary Lie group, wich connection do we take ?
A Lie group is a parallelizable manifold, i.e there are vectors fields $X_1,..,X_n$, such that for every $g\in G$, $(X_1(g),...,X_n(g))$ is a basis of the tangent space $T_gG$. Let ${\cal G}=T_eG$ be the Lie algebra of $G$ and $a_1,...,a_n$ its basis. $a_i$ generated the left invariant vector fiel $X_i(g)=dL_g(a_i)$. It is possible to define a connection on $G$ by setting $\nabla_{X_i}X_j=0$. The geodesics of this connection at the neutral point $e\in G$ are $exp(tu), u\in {\cal G}$.