I've a question about the following. Take an affine space $(E,V)$ and consider the tangent space $T_aE$ at a point $a$.
From John Lee book "Introduction to smooth manifolds" chapter 3 there exist a canonical isomorphism between the "geometric tangent space" $\mathbb R_a^n$ at point $a$ and the vector space of derivations at $a$, namely:
$$D_v|_a f = D_vf(a) = \left . \frac {d} {dt} \right |_{t=0} f(a + tv)$$
Now the vector $v \in \mathbb R_a^n$ can be understood as an element of translation vector space $V$. Therefore I believe there is a canonical/natural isomorphism between $V$ and $T_pE$.
Let me begin by a remark concerning the geometric tangent space $\mathbb R^n_a$ of $\mathbb R^n$ at $a$. Lee defines it as $$\mathbb R^n_a = \{ a \} \times \mathbb R^n $$ with the vector space structure canonically induced from $\mathbb R^n$, i.e. the unique vector space structure making the canonical bijection $\phi_a : \mathbb R^n \to \mathbb R^n_a, \phi_a(v) = (a,v)$, an isomorphism. Alternatively, we can say $\{a\}$ is a vector space (this is trivial, each single point set is a vector space in a unique way) and regard $\mathbb R^n_a$ as the product vector space of $\{a\}$ and $\mathbb R^n$.
Actually $\mathbb R^n_a$ is an affine subspace of $\mathbb R^n \times \mathbb R^n$ and we use $\phi_a$ to identify it with the vector space $\mathbb R^n$.
One could also define $\mathbb R^n_a = \mathbb R^n$, and many authors do so. It is a matter of taste! A benefit of Lee's definition is that $\mathbb R^n_a$ contains the information to which $a \in \mathbb R^n$ it belongs. The $\mathbb R^n_a$ are in fact pairwise disjoint.
Lee associates to each $(a, v) \in \mathbb R^n_a$ the map $D_v \mid_a : C^\infty(\mathbb R^n) \to \mathbb R, D_v \mid_a(f) = \frac {d} {dt} |_{t=0} f(a + tv)$ which is a derivation at $a$ and proves that $(a, v) \mapsto D_v \mid_a$ is vector space isomorphism $\mathbb R^n_a \to T_a \mathbb R^n$.
Changing perspective, one could also say that for each $a \in \mathbb R^n$ one gets a vector space isomorphism $\iota_a : \mathbb R^n \to T_a \mathbb R^n$ via $v \mapsto D_v \mid_a$. The only difference to Lee's approach is that we have to specify separately the point $a$, whereas Lee trivially knows $a$ for the elements of $\mathbb R^n_a$.
Let us come to your affine space $(E,V)$. For each $p \in E$ there is a bijection $\beta_p : V \to E, \beta_p(v) = p + v$. A smooth structure on $E$ is generated by a single chart of the form $\psi \circ \beta_p^{-1} : E \to \mathbb R^n$, where $\psi : V \to \mathbb R^n$ is an isomorphism. It is the unique smoth structure making $\psi \circ \beta_p^{-1}$ a diffeomorphism. It is easy to see that resulting smooth structure does not depend on the choices of $p$ and $\psi$.
Note that also $V$ has a canonical smooth structure making $\psi$ a diffeomorphism; again it does not depend on the choice of $\psi$.
What should be the "geometric tangent space" $E_p$ of $E$ at $p \in E$? I think one should take $$E_p = \{p \} \times V$$ which is identfied with $V$ via $\phi_p : V \to E_p, \phi_p(v) = (p,v)$. Alternatively one can take $$E_p = V$$ as above for $\mathbb R^n$. It seems that you prefer the second variant and want to get a canonical isomorphism $V = E_p \to T_pE$. Anyway it does not make much difference to work with $V$ or with $\{p \} \times V$ here.
Let $\lambda_0 : V \to T_0V$ and $\kappa_p : V \to T_pE$ be the unique isomorphisms making the following diagram commutative:
$\require{AMScd}$ \begin{CD} V @>{\kappa_p}>> T_pE \\ @V{id_V}VV @VV{d(\beta_p^{-1})_p}V \\ V @>{\lambda_0}>> T_0V \\ @V{\psi}VV @VV{d\psi_0}V \\ \mathbb R^n @>>{\iota_0}> T_0 \mathbb R^n \end{CD}
It is easy to see that $\lambda_0$ does not depend on the choice of $\psi$, thus it is a canonical isomorphism.
Also $d(\beta_p)_0 : T_0V \to T_pE$ may be regarded as the canonical isomorphism, simply because the smooth structure on $E$ was defined via $\beta_p$.
This shows that $\kappa_p$ has every right to be regarded as the canonical isomorphism.