Condition on real matrix $A$ such that the set $\{S^{-1}AS: S \in GL_n(\mathbb R), (SA-AS)e_1 = 0 \}$ is connected

Question

This is closed related to the question I asked here which concerning the number of connected components in $E := \{S^{-1}AS: S \in GL_n(\mathbb R), (SA-AS)e_1 = 0\}$. Although it has not received an answer, but I suspect the answer will be dependent on $A$. Here I will put some condition on $A$ and be interested in sufficient conditions guaranteeing $E$ is connected.

Let $A \in M_n(\mathbb R)$ be such $Ae_1 = e_2$ where $e_1, e_2$ are standard basis in $\mathbb R^n$. Let $E := \{S^{-1}AS: S \in GL_n(\mathbb R), (SA-AS)e_1 = 0\}$, i.e., the conjugacy class of $A$ but with restriction that first column of $S^{-1}AS$ to be $e_2$. Now I am interested in sufficient conditions on $A$ such that $E$ has only $1$ connected components.

Here are some related questions:

Connectedness of matrix conjugacy class,

connectedness of matrix conjugacy classes of a fixed real $A$ but with the first column of $A$ invariant.

Both questions haven't received answers at this moment, but some of the comments might be useful.