I have the paper with 3 points on it. I have also a photo of this paper. How can I determine where is the paper on the photo, if I know just the positions of these points? And are 3 points enough? It doesn't look like a linear transform: the paper turns into trapeze on the photo. But it should be able to be written mathematically. I was unsuccessfully looking for some function $\mathbb{R}^2\rightarrow\mathbb{R}^2$, which turns the coordinates of the points on the default paper to their coordinates on photo, so I could just apply this function to the paper corners, but I have no idea, what kind of transform it should be.
Any help is appreciated :)
I believe a projective transformation is exactly right. But with a projective transformation you can take any four points in general position (no three collinear) to any other four. So you'd better add a fourth point to reconstruct the transformation.
In terms of the linear algebra, you want to construct a $3\times 3$ matrix $A$, up to scalar multiples. You think of a point in $\mathbb R^2$ as a vector $X=\begin{bmatrix}1\\x\\y\end{bmatrix}\in\mathbb R^3$ and its image will be the vector $AX$ rescaled so that its first coordinate is likewise $1$. So see that you can determine $A$ from knowing where four points map, think of it this way. A linear map from $\mathbb R^3$ to $\mathbb R^3$ is determined by where it sends a basis. Given our three points in general position, they correspond to vectors $P$, $Q$, $R\in\mathbb R^3$ up to scaling. But now, given our fourth point in general position, we can rescale those vectors so that the fourth point is given by $P+Q+R$. Doing the corresponding construction with the four image points will now allow us to specify precisely where $A$ sends our basis vectors, up to one common scalar multiplication.