I have a monitor and can light up any pixel on it. I have a camera looking at the monitor.
I want to figure out in which manner pixels on monitor are displayed on a matrix. For this, I light up three pixels on a monitor and capture it on a camera. This leads me to the following situation:
Points on screen and how matrix sees them
What is happened: I showed up some point $S_0$ on a screen, then I showed two points - shifted only horizontally ($S_U$) and only vertically ($S_V$). Then I captured these points with a camera and on camera's matrix these points appeared in $M_0$, $M_U$ and $M_V$ accordingly. This helps me understand that matrix and monitor are also inclined towards each other in some way.
Now I can calculate in which ratio points on matrix are moved according to screen point movement (let's add function $u(Point)$ that returns horizontal $U$ coordinate of a point and function $v(Point)$): $$\Delta S_U=u(S_U)-u(S_0)$$ $$\Delta S_V=v(S_V)-v(S_0)$$ $$\Delta M_U=u(M_U)-u(M_0)$$ $$\Delta M_V=v(M_V)-v(M_0)$$ $$U_{ratio}=\Delta M_U/\Delta S_U$$ $$V_{ratio}=\Delta M_V/\Delta S_V$$ As we can see, there is some deflection for each plane. E.g. when we lifted up dot by $4$ pixels on a screen, on a matrix dot also shifted to the right on $0.5$ pixels. Let's call this $UFromVDeflection$ - $U$ coordinate of a matrix point is changing when $V$ coordinate on a screen is changing
Now same but vice versa: when I shift screen dot horizontally, point on a matrix is also shifting vertically. E.g. if I shift screen point to the right on $4$ pixels, point on matrix also will shift vertically to $-0.5$ pixels. Let's call it $VFromUDeflection$.
I can also calculate these deflections: $$UFromVDeflection = \frac{u(M_V)-u(M_0)}{\Delta S_V}$$ $$VFromUDeflection = \frac{v(M_U)-v(M_0)}{\Delta S_U}$$
Now having all these coefficients I can easy calculate, where point on matrix will appear for any point on a screen. For example:
Calculated point on a matrix accroding to random selected point on a screen
Now I can select any point $S_T$ with coordinated $U_T$ and $V_T$ on a screen and interpolate where point $M_R$ with coordinates $U_R$ and $V_R$ will appear on a matrix: $$\Delta U_T=U_T-u(S_0)$$ $$\Delta V_T=V_T-v(S_0)$$ $$U_R=u(M_0)+\Delta U_T*U_{ratio}+\Delta V_T*UFromVDeflection$$ $$V_R=v(M_0)+\Delta V_T*V_{ratio}+\Delta U_T*VFromUDeflection$$
Now to the question: How to reverse this operation? How to find point on a screen from any point on a matrix?. Look at the image above. There is point on a matrix $M_R(13.5, 8.5)$ for point $S_T(6, 6)$ on a screen. How can I calculate $S_T$ from $M_R$ only having points $S_0$, $S_U$, $S_V$ and correspoinding points on a matrix? I spent a lot of time figuring out how to do complete backwards interpolation and all my results have some error. Is it even possible to find exact point on a screen from point on a matrix in this situation? I am bad at math and can't figure it out by myself.