Edit to better explain the problem:
I have 3 sets of points for which I know they are same but rotated about a point (picture a door rotating around a hinge). I emphasized the word "same" because those points are empirical measurements. Meaning, 1st point of set1 doesn't match 1st point in set2 or set3 (and most likely doesn't have a perfect match in all the other points).
I calculated a best fit line for each 3, so now I have 3 line equations. Using those I want to calculate the center of rotation (I'm aware that it will just be an estimation and not 100% accurate calculation since the lines themselves are an estimation).
$$$$
I first tried to find the center of rotation with only 2 lines, but it seems that all the points equidistant to both lines would be a possible answer. I then tried find the rotation point using 3 line equations (In the form $y=ax+b$).
From what I found searching, by calculating the angle bisector line equation between 2 lines and and again for another pair of lines, and then finding the intersection of those 2 angle bisector lines I would find my rotation point.
1st method:
If we have 2 lines $Ax+By+C=0$ and $Dx+Ey+F=0$
I can find the angle bisector line equation with this:
$\frac{Ax+By+C}{\sqrt{A^{2}+B^{2}}}=\pm\frac{Dx+Ey+F}{\sqrt{D^{2}+E^{2}}}$
This gives 2 equations for the angle bisectors between 2 lines:
$\begin{cases} \left(\sqrt{D^{2}+E^{2}}A-\sqrt{A^{2}+B^{2}}D\right)x+\left(\sqrt{D^{2}+E^{2}}B-\sqrt{A^{2}+B^{2}}E\right)Y+\left(\sqrt{D^{2}+E^{2}}C-\sqrt{A^{2}+B^{2}}F\right)= & 0\\ \left(\sqrt{D^{2}+E^{2}}A+\sqrt{A^{2}+B^{2}}D\right)x+\left(\sqrt{D^{2}+E^{2}}B+\sqrt{A^{2}+B^{2}}E\right)Y+\left(\sqrt{D^{2}+E^{2}}C+\sqrt{A^{2}+B^{2}}F\right)= & 0 \end{cases}$
Since my line equations are in the form $y=ax+c$ and $y=dx+f$ then $A=a;B=-1;C=c;|D=d;E=-1;F=f;$
I tried with some test numbers but the resulted line equation is incorrect:
(The red continuous line is one of the bisectors and red X is the center/point of rotation that I'm want to find, the grey discontinuous lines are best fit lines going through some points,the small red and blue segments under the gey lines are the test points I set)
Then I tried a 2nd method:
First find the angle bisector slopes, then solve with point-slope form with the intersection of the 2 lines.
$\frac{m-m_{1}}{1+mm_{1}}=\pm\frac{m-m_{2}}{1+mm_{2}}$ where $m_1$ and $m_2$ are known and m is what we are looking for.
However I think I did something wrong (tried 3 times same result), but the 2 equations I found for m are:
$\begin{cases} m= & \sqrt{\frac{m_{1}-m_{2}}{m_{2}-m1}}\\ 0= & m^{2}(m_{2}+m_{1})+m(2-2m_{1}m_{2})-(m_{1}-m_{2}) \end{cases}$
But that means in the first line, $m=-1$ (not even -1 because of the square root, $0+1i$ and I have not idea how to deal with a complex number) always and in the second line it's a 2nd degree equation, which gives 2 solutions, so I stopped there.
Clearly I'm doing something wrong, but what is it?
P.S.: By playing around in geogebra, it looked like the angle bisectors that pass through the center of rotation are the obtuse line bisectors, is that correct?