Let $A$ be an affine space with translation space $V$. We may think of it as a nonempty set $A$ endowed with a regular action of $V \times A \to A :(v,p)\mapsto p+v$ (the affine action). The symmetry group of $A$ is the collection of affine automorphisms of $A$, i.e. $V$-equivariant invertible maps $$\operatorname{Aut}(A) = \{f :A \to A\ |\ \exists d\!f \in \operatorname{End}(V)\ :\ f(p+v)=f(p)+d\!f(v)\quad \forall p,v,\quad f \text{ bijective}\}. $$ These contain the set of purely translational automorphisms $\operatorname{Tr}(A)=\{f \in \operatorname{Aut}(A)\ |\ d\!f=\operatorname{id}_V\}$ as a normal subgroup, and the set of purely linear automorphisms preserving a given pole $p_0\in A$, $\operatorname{Aut}(A)_{p_0}$, as a non-normal subgroup. Since the intersection between these two subgroups is trivial, $\operatorname{Aut}(A)$ decomposes as an internal semidirect product $$\operatorname{Aut}(A) = \operatorname {Tr}(A) \rtimes \operatorname{Aut}(A)_{p_0}. \tag1$$ Thus every map $f \in \operatorname{Aut}(A)$ has uniquely determined translational component $f_t \in \operatorname{Tr}(A)$ and linear component $f_\ell\in\operatorname{Aut}(A)_{p_0}$ such that $f=f_t\circ f_\ell$; the product of two automorphisms $g,f$ decomposes as $$g\circ f = (g_t\circ g_\ell) \circ (f_t\circ f_\ell) = \underbrace{g_t \circ (g_\ell \circ f_t \circ g_\ell^{-1})}_{\in\operatorname{Tr}(A)} \circ \underbrace{g_\ell \circ f_\ell}_{\in\operatorname{Aut}(A)_{p_0}}, \tag2$$ where we recognise the adjoint action of $\operatorname{Aut}(A)_{p_0}$ on $\operatorname{Tr}(A)$.
On the other hand, this Wikipedia page seems to suggest that the symmetry group of $A$ should be constructed from scratch, i.e. as an external semidirect product $$\operatorname{Aut}'(A) := V \rtimes_1 \operatorname{Aut}(V),\tag3$$ where $1$ is the identity homomorphism $\operatorname{Aut}(V) \to \operatorname{Aut}(V)$ encoding the fundamental action $\operatorname{Aut}(V)\times V \to V : (L,v)\mapsto Lv$. Observe that $\operatorname{Tr}(A)\simeq V$ and $\operatorname{Aut}(A)_{p_0}\simeq \operatorname{Aut}(V)$, so the two factors in $(3)$ "contain the same information" as those in $(1)$; however, because of $(2)$, the internal product $(1)$ is naturally isomorphic to the external semidirect product $V \rtimes_\alpha \operatorname{Aut}(V)$, where $\alpha : \operatorname{Aut}(V) \to \operatorname{Aut}(V) : v \mapsto LvL^{-1}$ now encodes the adjoint action!
This suggests that $\operatorname{Aut}(A)$ and $\operatorname{Aut}'(A)$ should be different groups. So, which one is the correct symmetry group of $A$? Are they secretly isomorphic, or do they describe different classes of structure-preserving maps of $A$?
If you make the isomorphisms $V \to Tr(A)$ and $Aut(V) \to Aut_{p_0}(V)$ explicit,
translations are of the form $f_{[t]}(x) = x + t$ for some $t \in V$, and automorphisms fixing $p_0$ are of the form $f_{(l)}(p_0+v) = p_0+ lv$ for some $l \in Aut(V)$.
Then for any $t \in V, l \in Aut(V)$ and any point $p_0+v \in A$,
$(f_{(l)} \circ f_{[t]} \circ {f_{(l)}}^{-1}) (p_0+v) \\ = (f_{(l)} \circ f_{[t]} \circ {f_{(l^{-1})}}) (p_0+v) \\ = (f_{(l)} \circ f_{[t]}) (p_0 + l^{-1}v) \\ = f_{(l)} (p_0 + l^{-1}(v) + t) \\ = p_0 + l(l^{-1}(v)) + l(t) \\ = p_0 + v + l(t) = f_{[l(t)]}(p_0+v)$.
And so, your $g_{(l)} \circ f_{[t]} \circ g_{(l)}^{-1}$ in (2) is equal to $f_{[l(t)]}$,
This means that your action of $Aut_{p_0}$ on $Tr(A)$ given by $(g_l,f_t) \mapsto g_{(l)}\circ f_{[t]}\circ g_{(l)}^{-1} = f_{[l(t)]}$, after translating through the isomorphisms, corresponds to the action $(l,t) \mapsto l(t)$ of $Aut(V)$ on $V$