What is $7-\cfrac{12}{7-\cfrac{12}{7-\cfrac{12}{7-\cdots}}}$?
Conventional methods of solving these types of problems tell me to do as follows:
Let $$x =7-\cfrac{12}{7-\cfrac{12}{7-\cfrac{12}{7-\cdots}}}$$
Then $x = 7-\frac{12}{x}$. Then the solution is $x=3, 4$. How do I know which one to pick?
On
The practical answer is to compute a few terms of the series and see which root it is converging to. Make a column in a spreadsheet starting with $7$ and then below $=7-12/above$ and copy down. It is clearly converging to $4$
If you do the above, which is a good exploration even if you want a rigorous solution, you should note that the iterations are always greater than $4$ and decreasing. You can show that if one iteration is greater than $4$, the next is less but still greater than $4$. Now you have a sequence that is monotonic and bounded below by $4$, so it must converge. You have shown that if it converges it is to either $3$ or $4$, so you are done.
(EDITED)
If $f(x) = 7 - 12/x$, then $f'(3) = 4/3$ while $f'(4) = 3/4$. Since $|f'(3)| > 1$, $3$ is a repelling fixed point of the iteration $x_{n+1} = f(x_n)$, while since $|f'(4)|< 1$, $4$ is an attracting fixed point. Thus if you don't start exactly at $3$, the iterations won't converge to $3$, but they will converge to $4$ at least if you eventually get close enough to $4$.
The boundary points of the immediate basin of attraction of an attracting fixed point can be $\pm \infty$, a singular point (here $0$), a repelling fixed point, a point mapped to one of those, or a $2$-cycle. Here there are no $2$-cycles, and the basin of attraction is $(3, \infty)$. Thus if you start, or ever get, anywhere $> 3$, you end up approaching the limit $4$.