When given the mobius transformation and three points $f(z)=\frac{az+b}{cz+d}$ for $z,a,b,c,d\in \mathbb{C},ad-bc\neq0, f(z_1)=f1,f(z_2)=f_2,f(z_3)=f_3$,
what is the approach to determine the unique transformation? Because during the substitutions I always end up with a quadratic equation involving one of the variables. How do I then definitively disprove the other solution?
A good idea is to use $(0,\infty,1)$ as a "way station" (midpoint) between two triples.
To go from $(z_1,z_2,z_3)$ to $(0,\infty,1)$, we may put $z-z_1$ in a numerator to ensure $f(z_1)=0$, we may put $z-z_2$ in a denominator to ensure $f(z_2)=\infty$, and then normalize to ensure $f(z_3)=1$:
$$ f(z)=\frac{z-z_1}{z-z_2}\cdot\frac{z_3-z_2}{z_3-z_1}, $$
the "cross-ratio." The corresponding matrix for this transformation is the product
$$ \begin{bmatrix} z_3-z_2 & 0 \\ 0 & z_3-z_1 \end{bmatrix} \begin{bmatrix} 1 & -z_1 \\ 1 & -z_2 \end{bmatrix}. $$
To go from $(z_1,z_2,z_3)$ to $(f_1,f_2,f_3)$, then, we compose with the inverse of the transformation that goes from $(f_1,f_2,f_3)$ to $(0,\infty,1)$, which (using $(AB)^{-1}=B^{-1}A^{-1}$) would yield
$$ \begin{bmatrix} 1 & -f_1 \\ 1 & -f_2 \end{bmatrix}^{-1} \begin{bmatrix} f_3-f_2 & 0 \\ 0 & f_3-f_1 \end{bmatrix}^{-1} \begin{bmatrix} z_3-z_2 & 0 \\ 0 & z_3-z_1 \end{bmatrix} \begin{bmatrix} 1 & -z_1 \\ 1 & -z_2 \end{bmatrix}. $$
The title suggests you also want to understand why there is a unique transformation between triples. Suppose there were two transformations between a pair of triples. We may compose both with a map that turns the second triple into $(0,\infty,1)$, so we end up with two transformations that turn a triple into $(0,\infty,1)$. But then we may precompose with a map that turns $(0,\infty,1)$ into the first triple, and we end up with two transformations from $(0,\infty,1)$ to itself. But it is an easier exercise to show there is only one such transformation.