I have to find the corresponding matrix for the following rotation:
$R_ \frac{π}{4}$ (1,1)
My general matrix formula is:
$$\begin{pmatrix}1&0&h\\ 0&1&k\\ 0&0&1 \end{pmatrix}\begin{pmatrix}cosα&-sinα&0\\ sinα&cosα&0\\ 0&0&1 \end{pmatrix}\begin{pmatrix}1&0&-h\\ 0&1&-k\\ 0&0&1 \end{pmatrix} $$
I have said that $h$=1 and $k$=1 therefore my matrix is
$$\begin{pmatrix}1&0&1\\ 0&1&1\\ 0&0&1 \end{pmatrix}\begin{pmatrix}cosα&-sinα&0\\ sinα&cosα&0\\ 0&0&1 \end{pmatrix}\begin{pmatrix}1&0&-1\\ 0&1&-1\\ 0&0&1 \end{pmatrix}$$
$$=\begin{pmatrix}(1)(1)+(0)(0)+(1)(0) & (1)(0)+(0)(1)+(1)(0) & (1)(-1)+(0)(-1)+(1)(1)\\(0)(1)+(1)(0)+(0)(0) & (0)(0)+(1)(1)+(1)(0) & (0)(-1)+(1)(-1)+(1)(1) \\ (0)(1)+(0)(0)+(1)(0) & (0)(0)+(0)(1)+(1)(0) & (0)(-1)+(0)(-1)+(1)(1) \end{pmatrix} \begin{pmatrix}cosα&-sinα&0\\ sinα&cosα&0\\ 0&0&1 \end{pmatrix}$$
$$=\begin{pmatrix}1 & 0 & 0\\0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}\begin{pmatrix}cosα&-sinα&0\\ sinα&cosα&0\\ 0&0&1 \end{pmatrix}$$
$$=\begin{pmatrix} (1)(cosα)+(0)(sinα)+(0)(0) & (1)(-sinα)+(0)(cosα)+(0)(0) & (1)(0)+(0)(0)+(0)(1)\\ (0)(cosα)+(1)(sinα)+(0)(0) & (0)(-sinα)+(1)(cosα)+(0)(0) & (0)(-1)+(1)(-1)+(0)(1)\\ (0)(cosα)+(0)(sinα)+(1)(0) & (0)(-sinα)+(0)(cosα)+(1)(0) & (0)(-1)+(0)(-1)+(1)(1) \end{pmatrix}$$
$$=\begin{pmatrix} cosα&-sinα& 0\\sinα&cosα&-1 \\ 0 & 0 & 1 \end{pmatrix}$$
Does this matrix look right? My notes has something different for the third column.
You seem to have correctly set up your three matrices for rigid motions in the (coordinatized) Euclidean plane. We can write them $ABA^{-1}$. Then you made your first mistake, it would seem, by multiplying the outer two to each other, to get the correct answer (given your erroneous setup) for $AA^{-1}B$ That you did this right can be seen in the fact that the lefthand matrix after the second equals sign is the identity matrix. Then you seem to have made a mistake in multiplying the identity matrix times your rotation matrix $B$.
In any event, you made a serious mistake in multiplying the two outer matrices together.