Endow $SL_2(\mathbb R)$ with the usual $L^2$ topology of $\mathbb R^4$. I wonder if $SL_2(\mathbb Z)$ a maximal discrete subgroup in $SL_2(\mathbb R)$.
I am working through some concrete examples. I tried $SL_2(\mathbb Z[\sqrt 2])$, but it seems that it is no longer discrete due to the fact that $m+n\sqrt 2$ is dense in $\mathbb R$. I also tried the group generated by
$$ A=\left(\begin{matrix}1&\frac{1}{2}\\0&1\end{matrix}\right), B=\left(\begin{matrix}0&-1\\1&0\end{matrix}\right) $$
in matlab, but still I can't find a limit point or prove that is it actually discrete.
It is indeed maximal. There is a grunt proof using the theory of hyperbolic 2-orbifolds and their areas. In brief, for discrete subgroups $\Gamma < SL_2(\mathbb R)$ with quotient $\mathbb H^2 / \Gamma$ of finite area, the set of areas is a discrete set of positive numbers. The area of $\mathbb H^2 / SL_2(\mathbb Z)$ is pretty near the bottom of the list, and you can check by hand that $SL_2(\mathbb Z)$ is not a subgroup of any $\Gamma$ for which $\mathbb H^2 / \Gamma$ has smaller area.
But here is a proof using Bass-Serre theory and orbifold Euler characteristics that I can actually describe in full. (One could translate this into an area proof as indicated in the previous paragraph, using the Gauss-Bonnet theorem to carry out the translation; but I won't do that).
Step 1: $SL(2,\mathbb Z)$ is a nonuniform lattice in $SL(2,\mathbb R)$, meaning a discrete subgroup $\Gamma < SL(2,\mathbb R)$ whose action on $\mathbb H^2$ has a closed fundamental domain $D$ with the following properties:
It follows that any discrete subgroup $\Gamma < SL(2,\mathbb R)$ that contains $SL(2,\mathbb Z)$ is also a nonuniform lattice.
Step 2: The quotient graph of groups. For every nonuniform lattice $\Gamma < SL(2,\mathbb R)$ there exists a tree $T \subset \mathbb H^2$ which is invariant under $\Gamma$ such that the quotient graph $G=T/\Gamma$ is finite, the $\Gamma$-stabilizer of each edge of $T$ is trivial, and the $\Gamma$-stabilizer of each vertex of $T$ is a finite cyclic group (perhaps trivial). Thus $G$ has the structure of a graph of groups, with a finite cyclic group labelling each vertex, and the trivial group labelling each edge. The tree $T$ is the Bass-Serre tree of $G$.
Step 3: The Euler chacteristic of $\Gamma$. Let $\chi_{top}(G)$ be the topological Euler characteristic of the graph $G$, so if $G$ is a tree then $\chi_{top}(G) = 1$, if $G$ deformation retracts to a circle then $\chi_{top}(G)=0$, and otherwise $\chi_{top}(G) \le -1$. Let $v_1,...,v_K$ be the list of vertices of $G$ labelled with a nontrivial finite cyclic group, and let $n_1,...,n_K$ be the respective orders of those cyclic group. Define the Euler characteristic of $\Gamma$ to be the rational number $$\chi(\Gamma) = \chi_{top}(G) - \sum_{i=1}^K \left(1 - \frac{1}{n_i} \right) $$
Step 4: $\chi(\Gamma) < 0$. To prove this, assume that $\chi(\Gamma) \ge 0$. A simple case analysis shows that there is a very short list of possibilities, each of which leads to a contradiction:
Step 5: Given an inclusion of two nonuniform lattices $\Gamma_1 < \Gamma_2 < SL(2,\mathbb R)$, the index $[\Gamma_2 : \Gamma_1]$ is finite, and $$\chi(\Gamma_2) \cdot [\Gamma_2 : \Gamma_1] = \chi(\Gamma_1) $$ This follows from a piece of Bass-Serre theory, namely Bass' theory of edge indexed graphs and their Euler characteristics.
Step 6: We have $\chi(SL(2,\mathbb Z)) = 1 - (1 - 1/2) - (1 - 1/3) = -1/6$. So to prove that $SL(2,\mathbb Z)$ is not properly contained in any other nonuniform lattice $\Gamma$, it suffices by Step 5 to prove that $\chi(\Gamma) \ge -1/6$.
If $\chi_{top}(\Gamma) \le -1$ then $\chi(\Gamma) \le -1 < -1/6$.
If $\chi_{top}(\Gamma) = 0$ then $K \ge 1$ and so $\chi(\Gamma) \le 0 - (1 - 1/2) \le -1/2 < -1/6$.
We are reduced to considering the case that $\chi_{top}(\Gamma)=1$. Since $\chi(\Gamma) < 0$ it follows that $K \ge 2$, because $1-1/n < 1$ for all $n \ge 1$.
If $K \ge 3$ then $$\chi(\Gamma) \le 1 - 3 \times (1-1/2) = -1/2 < -1/6 $$ If $K = 2$ then as seen above we cannot have $n_1=n_2=2$, and therefore at least one of $n_1,n_2$ must be $\ge 3$, and so $$\chi(\Gamma) \le 1 - (1 - 1/2) - (1 - 1/3) = -1/6 $$
Final remark: This proof shows a bit more, namely that $-1/6$ is the maximal Euler characteristic of nonuniform lattices in $SL(2,\mathbb R)$. Furthermore, the maximum is achieved if and only if the lattice is isomorphic to $SL(2,\mathbb Z) \approx \mathbb Z/2\mathbb Z * \mathbb Z/3\mathbb Z$ (from which it follows furthermore that the lattice is conjugate to $SL(2,\mathbb Z)$ in $SL(2,\mathbb R)$).