Let's say I have subgroup of $SL_2(\mathbb{Z})$ generated by two elements, I wanted to compute the limit set. So I wrote down two matrices:
$$ A = \left( \begin{array}{cc} 1 & 1 \\ 0 & 1 \end{array} \right) ,\; B = \left( \begin{array}{cc} 1 & 0 \\ 1 & 1 \end{array} \right) $$ These can be written as two maps of fractional linear transformations or Möbius transformations: $$ z \mapsto z + 1\;\text{ and }z\mapsto \frac{z}{1+z}$$ The plots of the orbits of these sets was surprising. I represented a fraction $\frac{a}{b} = [a:b] \in \mathbb{R}P^2$ as a line in the Euclidean plane.
What are the basic properties of this fractal? E.g. it's dimension or that it's not connected.
a closer look:
an even closer look:
In fact, the set you've found is actually all of $SL_2(\mathbb{Z})$, not just a subgroup, because the two matrices $A$ and $B$ generate the group; the points you've plotted will eventually include all of the points on the circle that are on lines of rational slope.
The reason why you find a fractal-like structure is because of an interesting semi-universal property where simple solutions of an equation, simple points in a structure, etc. 'repel' other points. In this case, it manifests as gaps around all of the points that are on rational lines $ax+by=0$ with particularly small values of $a$ and $b$; thought of another way, there are wider 'gaps' around rational numbers $\frac ab$ (in a Farey sequence) with small $a$ and $b$ than there are around more complicated numbers.
To see an example of this in action, look at https://en.wikipedia.org/wiki/Farey_sequence#/media/File:Farey_diagram_horizontal_arc_9.svg which demonstrates all the points in the Farey sequence of order 9 (that is, all rational numbers $\frac ab$ with $0\leq\frac ab\leq 1$ and $b\leq 9$). Note that the nearest fraction to $0$ is $\frac19$, at distance $\frac19$, and a nearest fraction to $\frac12$ is $\frac49$, at distance $\frac1{18}$; by contrast, the nearest fraction to $\frac45$, for instance, is $\frac79$, which is only distance $\frac1{45}$ away from it!
The distribution of the Farey sequence is, arguably, understood both well and poorly at the same time. It's known that the order-$N$ Farey sequence $F_N$ has $\theta(N^2)$ points in it (in fact, approximately $\frac6{\pi^2} N^2$, where the $\frac6{\pi^2}$ comes from $\frac1{\zeta(2)}$) and that it approaches a uniform distribution as $N\to\infty$; it's also known that the gaps between adjacent numbers range from $\theta(\frac1N)$ to $\theta(\frac1{N^2})$ in size. (For instance, the gap next to $\frac12$ is always approximately $\frac1{2N}$.) But the question of how quickly it approaches a uniform distribution is much trickier:
Suppose that $f_n$ is the $n$th fraction in the order-$N$ sequence $F_N$, and consider the distance between $f_n$ and the point $\frac n{|F_N|}$ that corresponds to a uniform distribution with as many points as the Farey sequence. Now, we can look at the 'mean square' size of these distances over all the sequence, $\displaystyle \Delta_N=\sum_{n=1}^{|F_N|}(f_n-\frac n{|F_N|})^2$; this is a measure of the total discrepancy of the sequence. It's known that $\Delta_N\in o(1)$ as $N\to\infty$; this is one way of formalizing the idea that the Farey sequence approaches a uniform distribution. But how does $\Delta_N$ go to zero as $N\to\infty$? Well, the statement that $\Delta_N\in O(N^{-r})$ for all $r\lt 1$ would be nice; this says that $\Delta_N$ is 'essentially $1/N$', which is as good as we could hope for. But this statement is equivalent to the Riemann Hypothesis! (See the Wikipedia page on Farey fractions and this MathOverflow question for more on this)