OK so the problem says find a linear transformation $U,T: F \to F$ such that $UT=0$ but $TU \neq 0$. Then use your answer to find matrices $A$ and $B$ such that $AB = 0$ but $BA \neq 0$.
I found $U: (a,b) \mapsto (a-a,b-b)$ and $T: (a,b) \mapsto (a+3, b+3)$
now $UT(a,b)=(0,0)$ while $TU(a,b)=(3,3)$. So it's okay for the linear transformation.
But when I compute their matrix with regard to the standard basis $\beta ={(1,0), (0,1)}$,
I get $[U]\beta$ = $\begin{bmatrix}0 & 0 \\0 & 0\end{bmatrix}$ and $[T]\beta = \begin{bmatrix}4 & 3 \\ 3 & 4\end{bmatrix}$.
so now i take the corresponding matrix why do i always get the zero matrix no matter what?
here $[U]\beta*[T]\beta = \begin{bmatrix}0 & 0\\ 0 & 0\end{bmatrix}\begin{bmatrix} 4 & 3\\ 3 & 4 \end{bmatrix} = 0$ and $[T]\beta*[T]\beta = \begin{bmatrix}4& 3\\ 3 & 4 \end{bmatrix} \begin{bmatrix}0& 0 \\ 0 &0\end{bmatrix} = \begin{bmatrix}0 & 0 \\ 0 & 0\end{bmatrix}$, which is not consistent with the linear transformation? What am I doing wrong here?
Sorry for the format I really don't know how to type symbols here.
Your $[U]_{\beta}=0$ because $a-a=0$ and $b-b=0$. Furthermore, your $T$ isn't linear, as $T(0,0)=(3,3)\not=0$...
You'll need to try something different. For instance, $U(a,b)=(a,a)$ and $T(a,b)=(a-b,0)$. (Check me on this.)