Let $(x_1,y_1), (x_2,y_2), (x_3,y_3)$ be variables and $(a,b),det_1, dot_1,det_2,dot_2,det_3,dot_3$ be given constants. How can I solve the following system of nonlinear equations based on constant values?
\begin{cases} x_1x_2-a(x_1+x_2)+a^2+y_1y_2-b(y_1+y_2)+b^2 = dot_1\\ x_1y_2-x_2y_1+b(x_2-x_1)+a(y_1-y_2) = det_1\\ x_2x_3-a(x_2+x_3)+a^2+y_2y_3-b(y_2+y_3)+b^2 = dot_2\\ x_2y_3-x_3y_2+b(x_3-x_2)+a(y_2-y_3) = det_2\\ x_3x_1-a(x_3+x_1)+a^2+y_3y_1-b(y_3+y_1)+b^2 = dot_3\\ x_3y_1-x_1y_3+b(x_1-x_3)+a(y_3-y_1) = det_3\\ \end{cases}
As I know it has an unlimited number of answers but I don't know how to solve it. I'm looking for answers in integer and real numbers.
I will just give a sketch for now how to solve it, I can fill in more details if necessary.
First define vectors $v_i = (x_i-a,y_i-b),\, i = 1,2,3$. Then we can rewrite the equations as
\begin{align} \langle v_1, v_2\rangle = a_{12},\ \langle v_2, v_3\rangle = a_{23},\ \langle v_3, v_1\rangle = a_{31},\\ \det[v_1^t\ v_2^t] = b_{12},\ \det[v_2^t\ v_3^t] = b_{23},\ \det[v_3^t\ v_1^t] = b_{31},\\ \end{align} where I changed dot's and det's to a's and b's.
Denote by $\vartheta_{ij}$ the (oriented) angle between vectors $v_i$ and $v_j$. Remember that dot product in $\mathbb R^2$ is given by formula $\|v\|\|w\|\cos\vartheta$ and that determinants above measure (signed) area of parallelogram formed by the vectors, which can also be expressed as $\|v\|\|w\|\sin\vartheta$, and thus the system becomes
\begin{align} \|v_1\|\|v_2\|\cos\vartheta_{12} = a_{12},\ \|v_2\|\|v_3\|\cos\vartheta_{23} = a_{23},\ \|v_3\|\|v_1\|\cos\vartheta_{31} = a_{31},\\ \|v_1\|\|v_2\|\sin\vartheta_{12} = b_{12},\ \|v_2\|\|v_3\|\sin\vartheta_{23} = b_{23},\ \|v_3\|\|v_1\|\sin\vartheta_{31} = b_{31}.\\ \end{align}
Then, you can find the lengths $\|v_i\|$ by looking at the system $$\|v_1\|\|v_2\| = c_{12},\ \|v_2\|\|v_3\| = c_{23},\ \|v_3\|\|v_1\| = c_{31},$$ where $c_{ij} = \sqrt{a_{ij}^2+b_{ij}^2}$ obtained by squaring the above equations and using $\sin^2t+\cos^2t = 1$. Solving it gives you $$\|v_1\|=\sqrt{\frac{c_{12}c_{31}}{c_{23}}},\ \|v_2\|=\sqrt{\frac{c_{12}c_{23}}{c_{31}}},\ \|v_3\|=\sqrt{\frac{c_{23}c_{31}}{c_{12}}}.$$
The whole thing is, from geometric perspective, obviously rotationally invariant (rotating all of the vectors won't change the angles between them or the areas), so fix some angle $\vartheta$. Then, the solution can be represented as complex numbers as
$$v_1 = \|v_1\|e^{i\vartheta},\ v_2 = \|v_2\|e^{i(\vartheta +\vartheta_{12})},\ v_3 = \|v_3\|e^{i(\vartheta-\vartheta_{31})}.$$
More explicitly, $e^{i\vartheta_{12}} = \cos\vartheta_{12} + i\sin\vartheta_{12} = \frac{a_{12}}{c_{12}}+\frac{b_{12}}{c_{12}}i$ and $e^{i\vartheta_{31}} = \cos\vartheta_{31} + i\sin\vartheta_{31} = \frac{a_{31}}{c_{31}}+\frac{b_{31}}{c_{31}}i$, so we have
\begin{align}v_1 &= \sqrt{\frac{c_{12}c_{31}}{c_{23}}}(\cos\vartheta + i\sin\vartheta),\\ v_2 &= \sqrt{\frac{c_{12}c_{23}}{c_{31}}}(\cos\vartheta + i\sin\vartheta)(\frac{a_{12}}{c_{12}}+i\frac{b_{12}}{c_{12}}),\\ v_3 &= \sqrt{\frac{c_{23}c_{31}}{c_{12}}}(\cos\vartheta + i\sin\vartheta)(\frac{a_{31}}{c_{31}}-i\frac{b_{31}}{c_{31}}),\ \vartheta\in\mathbb R.\end{align}
All you have to do now is expand and $x$'s will be the real parts, while $y$'s the imaginary parts of the above complex number representation.
Note, however, that the system is overdetermined since knowing the angles between $v_1$ and $v_2$ and $v_2$ and $v_3$ will also give you the angle between $v_1$ and $v_3$.