Let $\textrm{SO}_2(\textbf{R})$ act on $\textrm{GL}_2(\textbf{R})$ by conjugation. Does the quotient $\textrm{GL}_2(\textbf{R})/\textrm{SO}_2(\textbf{R})$ exist as a manifold?
I asked the same question in MO and got a negative answer for the quotient $\textrm{GL}_2(\textbf{R})/\textrm{SL}_2(\textbf{R})$. Thus I really would like to know if the quotient $\textrm{GL}_2(\textbf{R})/\textrm{SO}_2(\textbf{R})$ exist as a manifold?
Let $X$ be a manifold and $R$ be an equivalence relation of $X$. There is a criterion for the quotient $X/R$ to exist in Serre's book "Lie algebras and Lie groups", which says the following. Let $Gr\subset X\times X$ be the graph of $R$. Then the quotient $X/R$ exists as a manifold if and only if
(1) $Gr\subset X\times X$ is a closed sub manifold (i.e., the inclusion $Gr\rightarrow X\times X$ is a closed embedding), and
(2) the projection map $pr_1: Gr\rightarrow X$ is a submersion.
But I do not know how to check the corresponding maps are immersion and submersion or not in this case.
Anybody can help? Any comment, suggestion will be appreciated.
The question with $\mathrm{SL}_2$ is clearly negative since orbits are not all closed.
For $\mathrm{SO}(2)$, since the acting group is compact, the quotient is Hausdorff so this is more reasonable. Actually, you get a manifold with boundary, which can be completely described:
Every matrix in $M_2(\mathbf{R})$ decomposes as: $$M=\begin{pmatrix} a & b \\ c & d\end{pmatrix}=\frac{a+d}2\begin{pmatrix} 1 & 0 \\ 0 & 1\end{pmatrix}+\frac{b-c}2\begin{pmatrix} 0 & 1 \\ -1 & 0\end{pmatrix}+\frac12\begin{pmatrix} a-d & b+c \\ b+c & d-a\end{pmatrix}.$$
The conjugation by the matrix $R_t=\begin{pmatrix} \cos t& -\sin t \\ \sin t & \cos t\end{pmatrix}$ acts trivially on the first two coordinates and acts as the matrix $R_{2t}$ on the pair $(a-d,b+c)$. As a consequence, the map $$\phi: M\mapsto (a+d,b-c,(a-d)^2+(b+c)^2)$$ is a faithful invariant, i.e. induces an injection from the quotient $M_2(\mathbf{R})/\mathrm{SO}(2)_{\mathrm{conj}}$ into $\mathbf{R}^3$, and more precisely into $\mathbf{R}^2\times\mathbf{R}_+$, where $\mathbf{R}_+$ denotes non-negative reals. This map is actually surjective, by an immediate verification (use the coordinates $(a+d,a-d,b+c,b-c)$).
If $\phi(M)=(x,y,z)$, we see that $\det(M)=-(z-x^2-y^2)/4$. In particular, $\phi(\mathrm{GL}_2(\mathbf{R})/\mathrm{SO}(2)_{\mathrm{conj}})$ is the open subset of $\mathbf{R}^2\times\mathbf{R}_+$ consisting of those $(x,y,z)$ with $x^2+y^2\neq z$. It has two components: $\{x^2+y^2<z\}$ which is a manifold with no boundary and corresponds to $\phi(\mathrm{GL}_2(\mathbf{R})^-/\mathrm{SO}(2)_{\mathrm{conj}})$, and $\{x^2+y^2>z\ge 0\}$, which corresponds to $\phi(\mathrm{GL}_2(\mathbf{R})^+/\mathrm{SO}(2)_{\mathrm{conj}})$, and contains the boundary $\{z=0\}$, which itself is the (injective) image of $\mathbf{R}_{>0}\mathrm{SO}(2)$ (the group of positive homotheties) in the quotient.
Added: we also see that the quotient $\phi(\mathrm{SL}_2(\mathbf{R})/\mathrm{SO}(2)_{\mathrm{conj}})$ is the surface with boundary $\{(x,y,z):z=x^2+y^2-4,z\ge 0\}$.