I am given a matrix $$A=\left(\begin{matrix}1&1&0&1\\1&2&1&0\\0&1&3&-2\\1&0&-2&4\end{matrix}\right)$$ which defines a scalar product in $\mathbb{R}^4$ as $\langle x,y\rangle=x^TAy$. I have to find an orthonormal system. I try to orthonorm canonical basis using Gram orthonormal procedure, so:
$$v_1=e_1\\v_2=\frac{e_2-\langle e_2,v_1\rangle v_1}{||e_2-\langle e_2,v_1\rangle v_1||}\\v_3=\frac{e_3-\langle e_3,v_2\rangle v_2-\langle e_3,v_1\rangle v_1}{||e_3-\langle e_3,v_2\rangle v_2-\langle e_3,v_1\rangle v_1||}$$ and so on. But when I do like this I get $\langle v_2,v_3\rangle\neq0$, so this method must be wrong for arbitrary scalar products and I have to do some adjustments, but I can't see them. Hope you can help me, thanks in advance!
You should get \begin{align*} v_1 &= e_1,\\ v_2 &= (-1,1,0,0)^T,\\ v_3 &= \frac1{\sqrt{2}}(1,-1,1,0)^T. \end{align*} Note that, when you normalise a vector $w$, you should not use the Euclidean 2-norm, but calculate the norm as $\|w\|=\sqrt{\langle w,w\rangle}=\sqrt{w^TAw}$.