There is a triangle with points P1(x1,y1),P2(x2,y2),P3(x3,y3) on an XY plane.
The final position after transformation are known to us, P1'(x,y) and P2'(x,y)
How can I find the third point?
Using slope (or distance) formula gives two solutions (one is mirror image of another). Assuming the transformation is a combination of translation and rotation, how do I get the new coordinates of the final point P3' ?
Let \begin{align} u &= (a, b) = (x_2 - x_1, y_2 - y_1)\\ v &= (c, d) = (x_2' - x_1', y_2' - y_1'). \end{align} be the vectors from $P_1$ to $P_2$ and from $P_1'$ to $P_2'$, respectively.
Then $u$ and $v$ should have the same length; let that length be $$ s = \sqrt{u_x^2 + u_y ^2 }. $$
Also define \begin{align} \newcommand{\up}{u^\perp} \newcommand{\vp}{v^\perp} \up &= (-b, a)\\ \vp &= (-d, c) \end{align} Finally, let $$ w = (x_3 - x_1, y_3 - y_1) $$ be the vector from $P_1$ to $P_3$.
Then compute $$ p = w \cdot u \\ q = w \cdot \up $$ and build the vector $$ w' = \frac{1}{s^2} (p v + q \vp) $$ Then $$ P_3' = P_1' + w'. $$
If the "dot product" used above doesn't isn't familiar to you, it just means this: the dot product of two vectors $(e, f)$ and $(g, h)$ is just the number $eg+fh$.
Actually, let me just write out the answer as best I can, trying to avoid typos. The $x$ and $y$ coordinates of $P_3'$ are computed like this:
\begin{align} p &= (x_3 - x_1)(x_2 - x_1) + (y_3 - y_1)(y_2 - y_1) \\ q &= -(x_3 - x_1)(y_2 - y_1) + (y_3 - y_1)(x_2 - x_1) \\ x_3' &= x_1' + \frac{1}{(x_2 - x_1)^2 + (y_2 - y_1)^2} \left( p(x_2' - x_1') - q (y_2' - y_1') \right)\\ y_3' &= y_1' + \frac{1}{(x_2 - x_1)^2 + (y_2 - y_1)^2} \left( p(y_2' - y_1') + q (x_2' - x_1') \right) \end{align} Notice the negative sign in front of the first term in the formula for $q$, and the difference in the last two parts of the formulas for $x_3'$ and $y_3'$ (the first has a minus; the second has a plus).