I have a matrix for transforming an image into a target frame. The matrix is a function of a scale, $s$ rotation angle, $\theta$, and a translation that is applied after rotating, $tx, ty$. The affine transform t is calculated by multiplying scale, then rotation then translation:
$$ t= \begin{bmatrix} s & 0 & 0 \\ 0 & s & 0 \\ 0 & 0 & 1 \\ \end{bmatrix} . \begin{bmatrix} cos(\theta) & sin(\theta) & 0 \\ -sin(\theta) & cos(\theta) & 0 \\ 0 & 0 & 1 \\ \end{bmatrix} . \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ tx & ty & 1 \\ \end{bmatrix} $$
This works well, I can adjust the scale, angle or translation to adjust the output image.
However the rotation and scale happens about the anchor point {0, 0}. I would like to be able to scale about an arbitrary anchor point {rX, rY} (in the co-ordinate space of the target frame).
What translation {tx, ty} should be applied (after scale and translation) so that the image rotates about {rX, rY} (instead of {0,0})?
I have looked at similar questions but I can not make it work for this problem.
The standard scaling and rotation matrices pivot point is $(0,0)$. To adjust it, compose your transformation with appropriate translation before and after the main matrix.
Suppose you want to rotate your picture with pivot at $(p_x,p_y)$, then first translate $(-p_x,-p_y)$, then rotate and finally translate back $(p_x,p_y)$
$$ \begin{bmatrix}x'\\y'\\1\end{bmatrix}= \begin{bmatrix} 1 & 0 & p_x \\ 0 & 1 & p_y \\ 0 & 0 & 1 \\ \end{bmatrix} \cdot \begin{bmatrix} cos(\theta) & -sin(\theta) & 0 \\ sin(\theta) & cos(\theta) & 0 \\ 0 & 0 & 1 \\ \end{bmatrix} \cdot \begin{bmatrix} 1 & 0 & -p_y \\ 0 & 1 & -p_x \\ 0 & 0 & 1 \\ \end{bmatrix} \cdot \begin{bmatrix}x\\y\\1\end{bmatrix} $$
or for row-vector representation of points (which seems you are using)
$$ \begin{bmatrix}x,y,1\end{bmatrix} \cdot \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ -p_x & -p_y & 1 \\ \end{bmatrix} \cdot \begin{bmatrix} cos(\theta) & sin(\theta) & 0 \\ -sin(\theta) & cos(\theta) & 0 \\ 0 & 0 & 1 \\ \end{bmatrix} \cdot \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ p_x & p_y & 1 \\ \end{bmatrix} = \begin{bmatrix}x',y',1\end{bmatrix} $$
The same will work with scaling. Also, if both your scaling and rotation use the same pivot point, then you can get rid of the inner translation as these cancel out, i.e.
$$ \begin{bmatrix} 1 & 0 & -p_y \\ 0 & 1 & -p_x \\ 0 & 0 & 1 \\ \end{bmatrix} \cdot \begin{bmatrix} 1 & 0 & p_y \\ 0 & 1 & p_x \\ 0 & 0 & 1 \\ \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{bmatrix} $$
I hope this helps $\ddot\smile$