Given two references points $p_{1i}$ and $p_{2i}$ that lie on plane with normal $n_i = (0,1,0)$ and their corresponding points $p_{1f}$ and $p_{2f}$ on plane with normal $n_f$, how do I find the rotation for any point on that surface?
Currently, I calculate the quaternion $q_1$ that rotates the vector $v_i = |p_{2i} - p_{1i}|$ to $v_f = |p_{2f} - p_{1f}|$. Then using this quaternion to rotate the rest of the points relative to $p_{1i}$, i.e. $p_{rot} = q_1 * | p - p_{1i}|$.
This however results in a somewhat random rotation along the $p_{2f} - p_{1f}$ axis. So, I find the new normal and find the angle between that rotation and the correct normal. and rotate, using a quaternion, the points to the correct orientation.
This approach works most of the time, however, for one quadrant the image is inverted. I am wondering if there is something I am missing? I can share the code I wrote if need be.