I'm struggling with finding formula of affine transformation where we have:
$(1,3) \to (3,5)$
$(2,1) \to (0,6)$
$(4,0) \to (0,11)$
I know in affine transformation we have $f(p)=q_o+\psi(\vec{p_0p})$ but have no idea how to solve this, would be greatful for hints or links to similiar problems
I'm not familiar with your notation for an affine transformation, but here's my take.
Associated with the affine transformation $T:\mathbb{R}^2\rightarrow \mathbb{R}^2$ is a $2\times2$ matrix and a vector, i.e.,
$$T(x,y) = \begin{bmatrix} a & b \\ c & d \\ \end{bmatrix}\begin{bmatrix} x \\ y \\ \end{bmatrix}+\begin{bmatrix} e \\ f \\ \end{bmatrix}$$
If you multiply it out, you have $T(x,y)=(ax+by+e,cx+dy+f)$. You are given 6 inputs and 6 outputs, so you should be able to solve for the 6 unknowns $a,b,..,f$ from there by substituting the various $(x,y)$ pairs.
You can check your answer with Wolfram Alpha's solution.