I am having troubles understanding the following definition of the exponential map of a Reimannian manifold.
The way I see it we define this map via the existence of the geodesics as solutions to some second order non-linear ode. However, when we define $B_{\epsilon_{p}}^{2}$ and use the same $\epsilon_{p}$ it kind of throws me off. How is maximal interval related to the norm of the initial condition? In other words why is $B_{\epsilon_{p}}^{2}$ defined in this particular way?
If I understood your question correctly, the geodesic here should be thought of as a parametrized curve. As you may already know, all geodesics have constant speed which may or may not be 1. The initial condition $\gamma'(0)$ is both specifying the initial direction and the initial speed. If you scale this vector, your maximal interval will scale accordingly. This is why we have to normalize the vectors and consider $T^1_pM$ instead of the whole tangent space.
Edit: Let's start with any non-zero $v\in T_pM$, assume $\gamma_v :(-a_v, b_v) \to M$ is the unique geodeisc with $\gamma_v(0)=p$ and $\gamma'_v(0)=v$. Further assume that $(-a_v, b_v) $ is the maximal interval for that geodesic. Now consider the geodesic for $kv\in T_pM$ for $k\neq 0$. I claim that $r:\left(-\frac{a_v}{k}, \frac{b_v}k\right) \to M$ defined by $r(t) =\gamma_v(kt) $ is the desired geodesic, and the domain is precisely the maximal interval. To see this, clearly we have $r(0)=p$ and $r'(0)=k\gamma'_v(0)=kv$. It is also straightforward to check that $r(t)$ satisfies the geodesic equation. Finally, $\left(-\frac{a_v}k, \frac{b_v} k\right)$ is maximal, if not then we have $r:(-a, b) \to M$ on a larger interval $(-a, b)$. Then by the same argument, we can extend the domain of $\gamma_v$ to a larger interval, still being a geodesic. This is a contradiction to the maximality of $(-a_v, b_v)$.