It is known that there are groups $G$ containing an element $g$ which is not mapped to $g^{-1}$ by an automorphism of $G$, but I find this counterintuitive; when I visualize the symmetries of some object in $\mathbb{R}^2$ or $\mathbb{R}^3$, it seems evident to me that doing a rotation one way vs its inverse are performing the same "role" in the group, and hence should be mapped to each other by an automorphism. I'm also aware that every (finite) group can be viewed as the symmetries of some object in $\mathbb{R^n}$
Is there an example (ideally geometric) where my intuition can accept that this can happen? Doing a computation in some semi-direct product isn't satisfying to me.
Below is an arguably geometric example, it may not be what you're looking for.
For a vector space $V$ over a field $F$, let $A(V)$ be the group of affine maps on $V$ which are compositions of translations and scaling maps, i.e. the affine maps of the form $x \mapsto ax + v$, for $a \in F \setminus \{0\}$ and $v \in V$. Let $T(V)$ be the subgroup of $A(V)$ consisting of translations, i.e. the maps $x \mapsto x + v$. Note that $T(V) \cong V$ as groups. We include the translations because they encode additional structure of the action of scaling maps on $V$; if we only considered the group of scaling maps, this would just be $F \setminus \{0\}$ under multiplication, which is abelian, and thus has automorphism $g \mapsto g^{-1}$.
We will now prove that when $F = \mathbb{Q}$ or $\mathbb{F}_p$, every automorphism $\phi$ of $A(V)$ preserves scaling factor, meaning if $f \in A(V)$ is of the form $x \mapsto ax + v$, then $\phi(f)$ is of the form $x \mapsto ax + v'$. In particular, this will mean that every element of $A(V)$ with scaling factor $a \neq \pm 1$ is not sent to its inverse by any automorphism of $A(V)$.
Lemma 1: Let $m, n$ be integers which are not zero in $F$. Then for $f \in A(V)$, $f$ has the property that $fg^n = g^m f$ for all $g \in T(V)$ if and only if $f$ has scaling factor $m/n$, meaning $f$ is of the form $x \mapsto (m/n)x + v$.
Proof: If $f$ is of the given form, then for $g(x) = x + u$, clearly $$f(g^n(x)) = (m/n)(x + nu) + v = (m/n)x + v + mu = g^m(f(x))$$ so $fg^n = g^m f$. In the other direction, if $f$ satisfies $fg^n = g^m f$ for all $g \in T(V)$, then writing $f(x) = ax + v$, and taking $g(x) = x + u$ for some $u \neq 0$, we have $a(x + nu) + v = ax + v + mu$, giving $anu = mu$, hence $a = m/n$ as desired.
Lemma 2: If $F = \mathbb{Q}$ or $\mathbb{F}_p$, then every automorphism $\phi$ of $A(V)$ preserves $T(V)$, meaning $\phi(T(V)) = T(V)$.
Proof: In the case $F = \mathbb{Q}$, we can identify the subgroup $T(V)$ of $A(V)$ as the set of "divisible" elements, namely those elements $g \in A(V)$ for which, for any positive integer $n$, there is an element $h \in A(V)$ with $h^n = g$. Elements $g \in A(V)$ of the form $x \mapsto ax + v$ for $a \neq 1$ do not have this property, since there can be an $h$ with $h^n = g$ only if $a$ is an $n$-th power in $\mathbb{Q}$, and the only nonzero element of $\mathbb{Q}$ which is an $n$-th power for each $n$ is $a = 1$. It is clear that any automorphism maps divisible elements to divisible elements, and non-divisible elements to non-divisible elements, so any $\phi$ has $\phi(T(V)) = T(V)$.
In the case $F = \mathbb{F}_p$, we can identify $T(V)$ as the set of elements of order $p$ (together with the identity). For an element $g$ of the form $x \mapsto ax + v$ for $a \neq 1$, $g$ is conjugate to the map $x \mapsto ax$, and thus $g$ has order dividing $p-1$, since this latter map has order dividing $p-1$. Automorphisms preserve order, so again any $\phi$ has $\phi(T(V)) = T(V)$. [end proof]
Now, let $\phi$ be an automorphism of $A(V)$, and let $f \in A(V)$, so $f(x) = ax + v$ for some $a, v$. Since $F = \mathbb{Q}$ or $\mathbb{F}_p$, $a$ is of the form $m/n$ for some integers $m, n$ which are not zero in $F$, and thus by Lemma 1, $fg^n = g^m f$ for any $g \in T(V)$. By Lemma 2, $\phi$ preserves $T(V)$, so $\phi(f)$ also has the property that $\phi(f) g^n = g^m \phi(f)$ for all $g \in T(V)$. Then, again by Lemma 1, this means that $\phi(f)$ has scaling factor $m/n = a$, so $\phi$ preserves the scaling factor of $f$.