How does $\mathrm{SL}_{3}(\mathbb{R})$ act on some space in the way that $\mathrm{SL}_{2}(\mathbb{R})$ acts on the upper half plane in $\mathbb{C}$?

Question

I have yet to figure this out. All I've done is look at some generators and try to see where the connection is to some other space.

Answer 1

For any group $G$ and a subgroup $H$ there is a natural action of $G$ on $G/H$ at the most abstract level. For Lie groups choose $H$ to be a closed subgroup. Or even a special kind of subgroup such as fixed point subgroup of an involution (e.g., orthogonal group will lead to action on the space of symmetric matrices.). You can take #H$ to be the subgroup of upper trianglular matrices then the action is on a projective variety (compact space) of flags etc.

Answer 2

To cut the long story short, the correct generalization of the upper half-plane to the case of the Lie group $G=SL(n, R)$ is the symmetric space $P_n$ of $SL(n, R)$. One way to describe it is as the space of projective classes of positive definite quadratic forms in $n$ variables on which $G$ acts via the change of variables. Here two forms $q, q'$ are projectively equivalent iff they are scalar multiples of each other. Equivalently, $P_n$ can be identified with the space of positive definite symmetric $n\times n$ matrices of determinant $1$. The action of $G$ on this space is given by $$ (g, M)\mapsto g^T M g, $$ $g\in G$, $M\in P_n$.

The action of $G$ on $P_n$ is transitive with the stabilizer of $I_n\in P_n$ equal to the orthogonal group $O(n)$. Now, you see the similarity with the action of $PSL(2,R)$ on the open unit disk in the complex plane. There is also a (unique up to scale) $G$-invariant Riemannian metric on $P_n$, at the identity matrix it is given by $\langle A, B\rangle=tr(AB)$, where $A, B$ are symmetric $n\times n$ matrices. This is an analogue of the hyperbolic metric on the unit disk (or the upper half-plane if you prefer). Notice however that the hyperbolic metric has constant negative curvature, while each $G$-invariant metric on $P_n$ has sectional curvature $\le 0$, with some directions of negative curvature and some directions of zero curvature, if $n\ge 3$.

You can read much more for instance in Helgasson's book on symmetric spaces. However, start here to get an idea of what this staff is about.