I was puzzling with circle inversions and was wondering about a converse of this:
Given 3 points A, B and M
Is there only one point R so that $d(R,A) * d(R,B) = d(R,M)^2$ or are there more of them?
(No there are many points R, figured out that much)
And if there a more of these points what curve do they form?
On
There are infinitely many.
The locus of points such that the product of the distances from those points to two fixed points is constant, is called a Cassini oval. Thus, for two fixed points $q_1$ and $q_2$, a point $p$ on the oval fulfills $dist(q_1,p) \cdot dist(q_2,p) = k$. In the link there is a figure showing an example of the ovals.
On page $9$ in this paper the equation of the ovals is found in the following form: $$y=\pm \sqrt{-(x^2+a^2) + \sqrt{k^2+4a^2x^2}}$$ where $k$ is the constant and the two fixed points are $(-a,0)$ and $(a,0)$.
In the figure below I've plotted ovals of the two points $(-2,0)$ and $(2,0)$ for constant values $k=1$ (blue), $k=2$ (red) and $k=3$ (green). I've also introduced a third point at $(1,1)$ and plotted circles centered at that point with radii $\sqrt 1$, $\sqrt 2$, and $\sqrt 3$.
We see that for $k=1$, there is no intersection of the blue circle with the blue ovals. But for $k=2$ (red) and $k=3$ (green), there are intersections.
Thus, if a circle centered on the point $q_3$ with radius $\sqrt k$ intersects a Cassini oval with constant $k$, the point of intersection $p$ will fulfill your requirement, i.e. $dist(q_1,p) \cdot dist(q_2,p) = dist(q_3,p)^2$. It should be clear that this will always be possible for large enough $k$.
EDIT
It seems that the set of points which fulfill your requirement are in fact hyperbolas. At least, I can show this is true when the third point is placed at origo. For the oval we have: $$y=\pm \sqrt{-(x^2+a^2) + \sqrt{k^2+4a^2x^2}}\tag{1}$$ and for the circle we have $$x^2+y^2 = k$$ or $$y=\pm \sqrt{k-x^2}\tag{2}$$ Setting equations 1 and 2 equal we find that $$x^2=\frac{2k+a^2}{4}$$ or $$k=\frac{4x^2-a^2}{2}\tag{3}$$ Inserting equation 3 into equation 2 we find, for $k \ge \frac{a^2}{2}$, that $$x^2-y^2=\frac{a^2}{2}$$ which is the equation of a hyperbola.
Assuming that we're talking in the plane with the standard (Euclidean) distance, let's go to coordinates: \begin{align} R&=(r_1,r_2)\\ A&=(a_1,a_2)\\ B&=(b_1,b_2)\\ M&=(m_1,m_2) \end{align} In this case, \begin{align} d(R,A)&=\sqrt{(r_1-a_1)^2+(r_2-a_2)^2}\\ d(R,B)&=\sqrt{(r_1-b_1)^2+(r_2-b_2)^2}\\ d(R,M)&=\sqrt{(r_1-m_1)^2+(r_2-m_2)^2} \end{align} Therefore, if we square both sides of your given equation, you're asking for solutions to $$ \left((r_1-a_1)^2+(r_2-a_2)^2\right)\left((r_1-b_1)^2+(r_2-b_2)^2\right)=\left((r_1-m_1)^2+(r_2-m_2)^2\right)^2. $$ If you multiply this out, the degree $4$ terms ($r_1^4$, $r_1^2r_2^2$, and $r_2^4$) cancel out, leaving a cubic bivariate polynomial whose solutions are the $R$'s of interest. There are lots of cubic plane curves, so it is unlikely that this curve has a name.
However, this formulation allows us to see that there are multiple solutions to the given equation (in many cases). Suppose that you have chosen a value for $r_1$, then you are left with a cubic in $r_2$. Since cubics always have roots real, you can find a value for $r_2$. Since you can do this for every $r_1$ (except for, perhaps, finitely many), there are infinitely many $R$'s satisfying the equation (unless there is some cancellation due to a special structure leading to cancellation in the cubic between $A$, $B$, and $M$.