Let's say I have $V=\mathbb{R}$ and the scalar product is $<X,Y> = X^t*A*Y$ and $A = \begin{bmatrix} 2 & 1 \\ 1 & 2\\ \end{bmatrix}$
Now I want to compute an orthonormal basis. Can I use the Gram-Schmidt process? And how would I start? Because If I just choose the first vector in the Gram-Schmidt process to be the standard $v = (1,0)^t$ this won't be normalized. To normalize a vector can I always divide it by the square root of its scalar product, no matter how the scalar product is defined?
At some step of the GS process, you will need to form linear combinations such as
$$X-\lambda U-\mu V$$ where $U,V$ are previously built orthogonal vectors, and make them orthogonal to $U$ and to $V$.
You write $$U^TA(X-\lambda U-\mu V)=U^TAX-\lambda U^TAU-\mu U^TAV=U^TAX-\lambda U^TAU=0$$ giving
$$\lambda=\frac{U^TAX}{U^TAU}.$$
And similarly
$$\mu=\frac{V^TAX}{V^TAV}.$$
You can indeed normalize by the square root of the scalar square.