This should be completely straightforward as every book I have read on the Hopf-Rinow theorem states this is "obvious". But I can find the life of me to justify this.
Geodesically complete means every geodesic $\gamma$ extends to all of time $\mathbb{R}$. How is that related to the velocity $\gamma'$ (which hence relates to exp)? If the curve extends all time, then are they implying the speed also extends all time? Does that even make sense since speed is constant for a geodesic, so why would that be attached to the time parameter?
Let $v \in T_p M$, $\gamma$ a geodesic that has speed $v$ at $\gamma(0)=p$.
The exponential map defined on $p$ takes vectors $v$ from $T_p M$ and considers the geodesic that has that speed at the point $p$. This map does the following, $v \to \gamma(1)$ whenever it is defined the point $\gamma(1)$. Geometrically what the map is doing is taking a geodesic and riding it until it has elapsed a unit of time. So if the surface is geodesically complete there is no reason for it to stop. That means $\gamma(1)$ is always defined and so the domain of the exponential map is all of $T_p M$. Conversely if for all points $p$ and all speed $v \in T_p M$ the exponential map is defined it means the surface is geodesically complete.
This shows why being geodesically complete in the sense that the domain of a geodesic is all $\mathbb R$ is exactly the same that the exponential map is defined on all the points of the surface and for all the speeds in the tangent space to the point.