Find the tangent space of $\mathrm{Aff}(n)$.
see Proof: Tangent space of the general linear group is the set of all squared matrices
$\mathrm{Aff}(n)$ is the set of all matrices of the form $$ \begin{pmatrix} A & \ b \\ 0 & 1 \\ \end{pmatrix}$$ where A $\in GL_n(\mathbb{R})$, b in $\mathbb{R}^n$, and $0$ is $1 \times n$ zero vector
Let $$ 1 = \begin{pmatrix} I & \ 0 \\ 0 & 1 \\ \end{pmatrix}$$
where I is identity matrix and 0 is a 0 column vector.
So my path is Q(t) $$ \begin{pmatrix} A(t) & \ bt \\ 0 & 1 \\ \end{pmatrix}$$ where $A(t)$ is a path in $ GL_n(\mathbb{R})$ s.t $A(0) = I$.
Continuing from where you were:
$$\forall A:(-\epsilon,\epsilon) \to \operatorname{GL}_n(\mathbb R), A(0) = \operatorname{id}_n$$ $$\forall b \in \mathbb{R}^n$$ Define $Q :(-\epsilon,\epsilon) \to \operatorname{Aff}_n(\mathbb R)$ $$Q(t) =\begin{pmatrix} A(t) & \ bt \\ 0 & 1 \\ \end{pmatrix}$$
Finding the tangent space at the identity means finding the tangent vectors to all possible curves at the identity. Thus, we differentiate and evaluate at $0$. $$\dot Q(t) =\begin{pmatrix} \dot A(t) & \ b \\ 0 & 0 \\ \end{pmatrix}$$ $$\dot Q(0) =\begin{pmatrix} \dot A(0) & \ b \\ 0 & 0 \\ \end{pmatrix} \in \mathfrak{aff}(n) $$ $$\implies \mathfrak{aff}(n) = \left\{ \begin{pmatrix} M & \ b \\ 0 & 0 \\ \end{pmatrix} \ \middle|\ M \in \operatorname{Mat}_n(\mathbb R),b \in \mathbb{R}^n \right\}$$
Thus we find that $b$ is arbitrary, and varying $A$ over the general linear group gives us all possible matrices in $\mathfrak{gl}_n(\mathbb R) = \operatorname{Mat}_n(\mathbb R)$.
Then as vector spaces, $\mathfrak{aff}_n(\mathbb R) = \mathfrak{gl}_n(\mathbb R) \oplus \mathbb R ^n $. As Lie algebras, you have to take the semi-direct product of the two terms, which in this case is just a way of saying the direct sum of the two as vector spaces has the bracket operation defined on it in the natural way.