I am supposed to find the affine matrix, that transform the large square $P$ into $Q$
It is given, that the transformation consists of a scaling, rotation, and translation matrix. We also know 3 points: $(0.09,1.15)$ $(1,1)$ and $(1,1.67)$
I have come to this point.
$$ A= \left(\begin{matrix}a*cos(v)&-b*sin(v)\\a*sin(v)&b*cos(v)\end{matrix}\right) \left(\begin{matrix}x\\y\end{matrix}\right) +\left(\begin{matrix}0.09\\1.15\end{matrix}\right) $$
I just don't know how to determine $a$ and $b$ which are the coefficients of our scaling matrix.
Anyone know how I should continue?

You know one vertex of the transformed rectangle, $(x_0,y_0)=(0.09,1.15)$, and that its sides are rotated 45° relative to the coordinate axes. With this information you can use the point-normal form of equation of a line $n_xx+n_yy=n_xx_0+n_yy_0$ to develop equations for the extensions of the two rectangle sides that meet at this vertex. You also have a point on each of the sides opposite to these two, so use the point-line distance formula ${\lvert ax+by+c\rvert\over\sqrt{a^2+b^2}}$ to determine the width and height of the transformed rectangle, and from that, the appropriate scale factors.