I was trying to calculate the new coordinates of the 4 corners of the rectangle in the image, but I think I only formed the rotation matrix. what is the correct transformation matrix for this operation ?
this is the triangle and the points names in order
and here is my calculations in Excel , attention that coefficient is only used to determine the sign of X and Y displacement of the origin, I tried both ±1. a=10 and b=5
using this transformation matrix
I don't get the expected results :
On
I can't really comment due to reputation but, can you please include the details: what exactly you're trying to do and include a grid of reference to denote the transformation you're trying to achieve(or atleast the position of origin) and label the vertices of the image in the rectangle. I'll try to help you further then. e.g. an explanation like rotating the rectangle and translating via xyz.
TL;DR: You need to use the inverse of the transformation matrix in your question.
You’ve made a fairly common error here. The transformation matrix that you’ve shown maps the old coordinate axes onto the new ones. However, to get the coordinates of a point relative to these new axes, you have to invert that transformation. Why this is so is covered elsewhere on this SE and on the Internet, but I’ll briefly explain why here.
Let $\mathbf U$ and $\mathbf V$ be the positive unit direction vectors of an arbitrary pair of coordinate axes, and $O$ the origin point of this coordinate system. We can express any point as $u\mathbf U+v\mathbf V+O$; the coefficients $u$ and $v$ are the coordinates of this point in this coordinate system. The $x$- and $y$- coordinates in the standard coordinate system can be understood in the same way by taking $\mathbf X=(1,0)$ and $\mathbf Y=(0,1)$, so that we have $x\mathbf X+y\mathbf Y$. We can write $\mathbf U=a\mathbf X+b\mathbf Y$ and $\mathbf V=c\mathbf X+d\mathbf Y$, and if we express $O$ as coordinates in the standard coordinate system, we get the following identity: $$x\mathbf X+y\mathbf Y = u(a\mathbf X+b\mathbf Y)+v(c\mathbf X+d\mathbf Y)+O.$$ Using homogeneous coordinates, we can express this identity in matrix form as $$\begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}a&c&O_x\\b&d&O_y\\0&0&1\end{bmatrix} \begin{bmatrix}u\\v\\1\end{bmatrix}.$$ Note that the $3\times3$ matrix in this identity maps the standard unit coordinate vectors onto the new ones and sends the origin to $O$. We want to solve this equation for $u$ and $v$, which we can do by multiplying both sides by the inverse of the $3\times3$ matrix.
In your case, $\mathbf X$ gets sent to $-\mathbf Y=(0,-1,0)$, $\mathbf Y$ gets sent to $\mathbf X=(1,0,0)$, and the origin gets sent to $(a/2,b/2)$. The correct matrix for computing the new coordinates of a point is therefore $$\begin{bmatrix}0&1&\frac a2\\-1&0&\frac b2\\0&0&1\end{bmatrix}^{-1} = \begin{bmatrix}0&-1&\frac b2\\1&0&-\frac a2\\0&0&1\end{bmatrix}.$$ More generally, if the new coordinate axes are obtained by rotating and translating the old ones, the coordinate transformation matrix will be $$\begin{bmatrix}R&\mathbf t\\\mathbf 0&1\end{bmatrix}^{-1} = \begin{bmatrix}R^T&-R^T\mathbf t\\\mathbf 0&1\end{bmatrix} = \begin{bmatrix}\cos\theta&\sin\theta&-(b_1\cos\theta+b_2\sin\theta)\\-\sin\theta&\cos\theta&-(b_2\cos\theta-b_1\sin\theta)\\0&0&1\end{bmatrix}.$$ Here I used the fact that the inverse of a rotation matrix is its transpose.