$$|\frac{z-4}{z-8}| =1$$ and $$|\frac{z-12}{z-8i}| =\frac{5}{3}$$
This is what I thought to do: $$(z-4)^2/(z-8)^2=1$$ $$-z+2=-2z+8$$ $$-x-yi+2=-2(x+yi)+8$$ $$x=6, y=0.$$ The answer in my book is $6+8i$ or $6+17i$. I don't know how to solve these two equationstogether.
You do not have $|z|^2=z^2$ for complex $z$. Instead $|z|^2=x^2+y^2$ where $z=x+yi$, with $x$ and $y$ being the real and imaginary parts of $z$.
Thus $|z-4|=|z-8|$ implies
$(x-4)^2+y^2=(x-8)^2+y^2$
$x^2-8x+16+y^2=x^2-16x+64+y^2$
The quadratic terms cancel leaving just a linear equation for $x$ which you then solve. Thus $x=6$.
Now put this into the second equation
$|z-12|=(5/3)|z-8i|$
$(x-12)^2+y^2=(25/9)(x^2+(y-8)^2)$
Put $x=6$ into that and solve for $y$ getting the quoted answers from the book.