Let $G = \mathrm{SL}(d, \mathbb{R})$ where $d \geqslant 3$. For $g \in G$, define the operator norm of $g$ as
$$ \Vert g \Vert_\mathrm{op} := \max \big\{ \Vert gXg^{-1} \Vert_\rho : X \in \mathfrak{sl}(d, \mathbb{R}), \, \Vert X \Vert_\rho = 1 \big\} $$
where the norm $\Vert \cdot \Vert_\rho$ on $\mathfrak{sl}(d, \mathbb{R})$ is induced by an inner product that gives rise to a right-invariant Riemannian metric $\rho$ on $G$. The metric $\rho$ induces a right-invariant distance function $d_G$ on $G$.
Question: Are there positive constants $C_1$ and $C_2$ such that for any $g, x \in G$, one has $$ d_G ( g x g^{-1}, e_G ) \geqslant C_1 \Vert g \Vert_{\mathrm{op}}^{-C_2} d_G (x, e_G )? $$ (Here $e_G$ denotes the identity element of $G$.)
To provide some context, if $d_X$ is the distance function induced from $d_G$ on the quotient $X = G / \Gamma = \mathrm{SL}(d, \mathbb{R}) / \mathrm{SL}(d, \mathbb{Z})$, then (in relation to a research project) I am trying to show that the Lipschitz norm $$ \Vert f \Vert_\mathrm{Lip} := \sup \Bigg\{ \frac{|f(g \Gamma)-f(h \Gamma)|}{d_X(g \Gamma,h \Gamma)} : g, h \in G, \, g \not \equiv h \text{ (mod $\Gamma$)} \Bigg\} $$
on the space $C_c^\infty (X)$ of compactly supported, smooth functions on $X$ satisfies
$$ \Vert f \circ g^{-1} \Vert_\mathrm{Lip} \ll \Vert g \Vert_\mathrm{op}^\sigma \Vert f \Vert_\mathrm{Lip} $$
for all $f \in C_c^\infty (X)$, $g \in G$, where the implied constant and $\sigma$ ($\sigma > 0$) are independent of $f$ and $g$.