This question has been asked several times, and virtually every introductory textbook on Lie groups and Lie algebras will cover this. But every single explanation goes beyond me.
Let $G$ be a Lie group. Let $\Psi : G \to \operatorname{Aut}(G)$ be the map sending $g \in G$ to the conjugation map $h \mapsto g h g^{-1}$. For any $g \in G$, the conjugation map $\Psi(g) : G \to G$ admits a total derivative $d\Psi(g)_e : T_e G \to T_e G$. Varying $g$, this may be expressed as a representation $\operatorname{Ad} : G \to \operatorname{Aut}(T_e G)$. Then, we define $\operatorname{ad} : T_e G \to \operatorname{End}(T_e G)$ to be the total derivative $d \operatorname{Ad}_e$.
Question. What is $\operatorname{ad}$ when $G = \operatorname{GL}_n(\mathbb{R})$?
What I know. Suppose $A$ is a matrix in $\operatorname{GL}_n(\mathbb{R})$. Then $\Psi(A)$ is the map sending a matrix $M$ to $AMA^{-1}$. We can take the derivative of $\Psi(A)$, which is a map $d \Psi(A)_e : T_e \operatorname{GL}_n(\mathbb{R}) \to T_e \operatorname{GL}_n(\mathbb{R})$. As $\operatorname{GL}_n(\mathbb{R})$ is an open submanifold of $\operatorname{Mat}_n(\mathbb{R})$, I can identify the tangent space of $\operatorname{GL}_n(\mathbb{R})$ with $\operatorname{Mat}_n(\mathbb{R})$. Moreover, for every matrix $X$, I have a particularly simple choice of a path $\gamma : I \to \operatorname{GL}_n(\mathbb{R})$ passing through $I_n$ with derivative $X$ at $t = 0$, namely, $\gamma(t) = I_n + tX$.
So to find out what $\operatorname{Ad}(A)$ is, I can write $d\Psi(A)_e(X)$, for any matrix $A$ in $\operatorname{GL}_n(\mathbb{R})$ and for any element $X \in T_e \operatorname{GL}_n(\mathbb{R}) = \operatorname{Mat}_n(\mathbb{R})$, as $$d \Psi(A)_e(X) = \frac{d}{dt} \Psi(A)(I + tX) = \frac{d}{dt} A(I + tX)A^{-1} = \frac{d}{dt} (I + AtX A^{-1}) = AXA^{-1}.$$ So, $\operatorname{Ad}(A)$ is $d\Psi(A)_e$, which is the map $X \mapsto AXA^{-1}$.
But then you have to take a further derivative. And here the limit of my capacity is reached. The complexity of the objects and maps involved simply impede me from seeing what it even means to take a derivative anymore, let alone being able to calculate it.
I would have guessed this was a duplicate, but the closest match I could find was: Definition of differential of Adjoint representation of Lie Group.
In any case you're almost there:
Hint As is common in this context, denote $\mathfrak{gl}(n, \Bbb R) := T_I \operatorname{GL}(n, \Bbb R) \cong \operatorname{Mat}_n(\Bbb R)$; since $\operatorname{GL}(n, \Bbb R)$ is a matrix group, I've used the symbol $I$ for its identity element instead of the generic symbol $e$.
As you write, under the identifications you made, the adjoint representation of $\operatorname{GL}(n, \Bbb R)$ is $$\operatorname{Ad} : \operatorname{GL}(n, \Bbb R) \to \operatorname{Aut}(\mathfrak{gl}(n, \Bbb R)), \qquad \operatorname{Ad}(A)(X) = AXA^{-1} .$$ Then, by definition, to compute the adjoint representation $$\operatorname{ad}: \mathfrak{gl}(n, \Bbb R) \to \operatorname{End}(\mathfrak{gl}(n, \Bbb R)) $$ of $\mathfrak{gl}(n, \Bbb R)$ we just differentiate $\operatorname{Ad}(A)(X)$ with respect to $A$.
So, fix a tangent vector $B \in \mathfrak{gl}(n, \Bbb R) \cong \operatorname{Mat}_n(\Bbb R)$, substitute the expression $I + t B$ for $A$ (which defines a path $t \mapsto I + t B$ in $\operatorname{GL}(n, \Bbb R)$ with tangent vector $B$ at $t = 0$) in $\operatorname{Ad}(A)(X) = AXA^{-1}$, and differentiate with respect to $t$ at $t = 0$.