I am confused by expressions like $\mathbb{R}/\mathbb{Z}$, especially if I see something of the sort $SL(2,\mathbb{R})/SO(2,\mathbb{R})$. I don't know what to make of it. I understand what a quotient of a group for an equivalence looks like, but I can't get my head around expressions like above.
Is a quotient like this always a group and what do elements in the set $SL(2,\mathbb{R})/SO(2,\mathbb{R})$ look like?
They are a lot of questions in one post. I will do my best to answer them.
First, let me recall the notion of group action:
This definition is equivalent to the following:
If $\cdot\colon G\times X\rightarrow X$ is a group action in the sense of definition $1$, then $g\mapsto \{x\mapsto g\cdot x\}$ is a group action in the sense of definition $2$. Conversely, if $\varphi\colon G\rightarrow\mathfrak{S}(X)$ is a group homorphism, then $g\cdot x=\varphi(g)(x)$ is a group action in the sense of definition $1$.
Example 1. If $H$ is a subgroup of $G$, then $H$ acts on $G$ by left translation i.e. $(h,g)\mapsto hg$ is a group action.
Now, let us define the notion of quotient set in this general setup:
The set $G/X$ is a collection of subsets of $X$. In all generality, this is not a group, for example, $X$ may not be a group itself.
Example $2$. If $H$ is a subgroup of $G$, then $G/H:=\{gH;g\in G\}$.
Let us investigate the structure of $G/H$ when $G$ is a group and $H$ is a subgroup acting by left translation.
Proof. Let define the operation on $G/H$ to be $gH\cdot g'H=gg'H$ or equivalently : $\pi(g)\pi(g')=\pi(gg')$. The key point is that this definition does not depend on the choice of $g$ and $g'$, namely: $$gH=xH,g'H=x'H\Rightarrow gg'H=xx'H.$$ Indeed, there exists $h,h'\in H$ such that $x=gh$ and $x'=g'h'$, therefore, one has: $$xx'H=ghg'\underbrace{h'}_{\in H}H=ghg'H=gg'\underbrace{g'^{-1}hg'}_{\in H}H=gg'H.$$ Now that the operation on $G/H$ is well-defined, I let you check the remaining properties (associativity, existence of identity element, existence of inverse). Whence the result. $\Box$
Conversely, if $G/H$ is a group such that $\pi$ is a group morphism, then $H$ is normal as the kernel of $\pi$.
Maybe I'll close this answer mentioning a useful result:
Proof. Consider $gG_x\mapsto g\cdot x$, in particular, check the well-definedness of this map. $\Box$
Remark 1. The set $G_x$ is a subgroup of $G$, but it may not be normal.
Example 3. $\mathbb{S}^n\cong SO(n+1)/SO(n)$ as sets, but even as smooth manifolds with a much-refined theorem.
Here, the action of $SO(n+1)$ on $\mathbb{S}^n$ is given by $(M,x)\mapsto Mx$ which is transitive, which means that each orbit under $SO(n+1)$ is the whole $\mathbb{S}^n$. If $x=(1,0,\ldots,0)$, then the stabilizer is given by: $$SO(n+1)_x=\left\{\begin{pmatrix}1 & 0\\0& A\end{pmatrix};A\in SO(n)\right\}\underset{\textrm{abusive}}{=}SO(n).$$ Indeed, the first column must be $x$ and by orthogonality the first entry of each other column must be $0$. Whence the result by the theorem.
Coming back to your main question (and guessing what the action you are working with is):
Let $\mathbb{H}$ be the upper-half plane of $\mathbb{C}$, namely the set of points in $\mathbb{C}$ having a strictly positive imaginary part. Let $SL(2,\mathbb{R})$ acts on $\mathbb{H}$ by homography: $$\begin{pmatrix}a&b\\c&d\end{pmatrix}\cdot z=\frac{az+b}{cz+d}.$$ Then this action is transitive and the stabilizer of $i$ is given by: $$SL(2,\mathbb{R})_i=\left\{\begin{pmatrix}a & b\\-b&a\end{pmatrix};a,b\in\mathbb{R}\right\}=SO(2).$$ Whence using the theorem, $SL(2,\mathbb{R})/SO(2)\cong\mathbb{H}$ as sets and again even as smooth manifolds.