I need your help with the question below :
We define inner product on V = R(2x2) by <A,B> = tr(B*A) ( * = Transpose)
U is a subspace of V , U = {A ∈ V | tr(A) = 0}
How do I find an orthonormal basis for U relative to the given inner product?
My attempt:
a ∈ U should be element like this : [x y]
[z -x]
Then I can say that this is the regular base of U :
x[1 0] y[0 1] z[0 0]
[0 -1] [0 0] [1 0]
If this is the right way , how should I move on from here?
Recall that the trace of $B^*A$ is the usual dot product if you regard the matrices $A$ and $B$ as $4$-dimensional vectors $\vec a$ and $\vec b$ in $\mathbb{R}^4$, written as column vectors.
To avoid confusion, we use the notation $$\langle A,B\rangle_{\text{tr}}=\mathrm{trace}(B^*A)$$ and we'll use $\langle \vec a,\vec b\rangle$ for the dot product. Then $$\langle A,B\rangle_\text{tr}=\langle \vec a,\vec b\rangle.$$
I'll give you an example. Let $$A=\begin{pmatrix}1&-3\\4&5\end{pmatrix}\text{ and } B=\begin{pmatrix}-5&3\\44&15\end{pmatrix}.$$ Now the trace of $B^*A$ is the usual dot product of the vectors $$\vec a=\begin{pmatrix}1\\-3\\4\\5\end{pmatrix}\text{ and } \vec b=\begin{pmatrix}-5\\3\\44\\15\end{pmatrix},$$ that is, $$\langle A,B\rangle_\text{tr}=\langle\vec a,\vec b\rangle= 1\cdot(-5)+(-3)\cdot3+4\cdot44+5\cdot15.$$
Furthermore the norm $\|A\|_\text{tr}$ of $A$ is given by $$\|A\|^2_\text{tr}=\langle A,A\rangle_\text{tr}=\langle\vec a,\vec a\rangle,$$ as usual.
Now to your question. Notice that $x$, $y$, and $z$ are mutually perpendicular. For example, $$\langle x,y\rangle_\text{tr}=\langle\begin{pmatrix}1\\0\\0\\-1\end{pmatrix}, \begin{pmatrix}0\\1\\0\\0\end{pmatrix}\rangle=0.$$
Moreover, $y$ and $z$ have norm $1$ whereas obviously the norm of $x$ is $\sqrt{2}$. So to achieve a orthonormal basis you only have to divide $x$ by its length, which is $\sqrt2$.