find an orthogonal basis and signature for each of following inner products:
$g(A,B)=tr(A^TUB)$
$h(A,B)=tr(AUB)$
$U=\begin{bmatrix}2 & 1 \\ 1 & 1 \end{bmatrix}$
firstly tried with g and i wanted to get the associated matrix (gram matrix) for this inner product the problem is if i try it with the standard basis i get just 0 everywhere (because of the trace) am i doing something wrong? i can't use standard basis?
Hints
As noted in the comments, the statement of the problem uses non-standard terminology. Since $\ g\ $ is a symmetric bilinear form on the space of $\ 2\times2\ $ (presumably) real matrices, the terms "orthogonal basis" and "signature" have standard meanings, which we can reasonably assume to be those intended by the author of the exercise.
Since $\ h\ $ is not symmetric, we can't be sure what the author of the exercise means by "orthogonal basis" and "signature" in that case. My guess is that they're referring to an orthogonal basis with respect to, and the signature of, the symmetrisation $\ h_s\ $ of $\ h\ $, given by $$ h_s(A,B)=\frac{h(A,B)+h(B,A)}{2}\ . $$
I can see at least two ways of approaching the problems of finding an orthogonal bas1s for, and signature of a symmetric bilinear form on $\ 2\times2\ $ real matrices, both starting from the standard basis: $$ E_1=\pmatrix{1&0\\0&1},\ E_2=\pmatrix{0&1\\0&0},\ E_3=\pmatrix{0&0\\1&0},\ E_4=\pmatrix{0&0\\0&1}\ . $$ I'll use $\ g\ $ to illustrate them (mainly the first)
Use coordinates with respect to the basis $\ E_1,E_2,E_3,E_4\ $:\begin{align}\text{tr}\left(\pmatrix{a_1&a_3\\a_2&a_4}\right.&\left.U\pmatrix{b_ 1&b_2\\b_3&b_4}\right)\\&=\begin{matrix}\pmatrix{a_ 1&a_2&a_3&a_4}\\\\\\\end{matrix} \pmatrix{2&0&1&0\\0&2&0&1\\1&0&1&0\\0&1&0&1}\pmatrix{b_1\\b_2\\b_3\\b_4}\end{align}
Thus, the matrix of $\ g\ $ with respect to the basis $\ E_1,E_2,E_3,E_4\ $ is $$M_g=\pmatrix{2&0&1&0\\0&2&0&1\\1&0&1&0\\0&1&0&1}$$ and the signature of $\ M_g\ $ will be the signature of $\ g\ $.
If $\ \Omega=\big(\omega_{ij}\big)\ $ is the orthogonal matrix such that $\ \Omega^TM_g\Omega=$$\,\text{diag}\big(\lambda_1, \lambda_2,\lambda_3,\lambda_4\big)\ ,$ where $\ \lambda_1, \lambda_2,\lambda_3,\lambda_4\ $ are the eigenvalues of $\ M_g\ $ (whereby the columns of $\ \Omega\ $ will be the corresponding normalised eigenvectors), then the matrices $$B_i=\omega_{1i}E_1+\omega_{2i}E_2+ \omega_{3i}E_3+\omega_{4i}E_4 $$ for $\ i=1,2,3,4\ $ will be an orthogonal basis with respect to $\ g\ $.
You could also apply Gram-Schmidt orthogonalisation (with respect to $\ g\ $) to the basis $\ E_1,E_2,E_3,E_4\ $, to obtain an orthogonal basis. As it happens, $\ g\big(E_i,E_i\big)>0\ $ for all $\ i^\color{red}{\,\dagger}\ $, so you can also apply the normalisation steps of the Gram-Schmidt procedure for that case. In the more general case, where the bilinear form might be degenerate or negative (semi-) definite, some of the normalisation steps might not be possible. You will still obtain an orthogonal basis, however, if you simply omit those steps. If $\ b\ $ is a general bilinear form, and $\ u_1,u_2,\dots,u_n\ $ an orthogonal basis with respect to it, then its signature will be given by the numbers of indices $\ i\ $ for which $\ b(u_i,u_i)\ $ is respectively positive, negative. or zero.
${}^\color{red}{\dagger}$ I obtained $\ g\big(E_1,E_1)=g\big(E_2,E_2)=2,$$\, g\big(E_3,E_3)=$$\,g\big(E_4,E_4)=$$\,1\ $, so I don't understand your comment "I just get $0$ everywhere".