Problem: Let $\alpha = \left\{(1,2,0), (1,0,1), (2,3,1)\right\}$ be a basis vor $\mathbb{R}^3$. Apply the Gram-Schmidt orthogonalisation process to turn $\alpha$ into an orthonormal basis for $\mathbb{R}^3$ with respect to the standard innerproduct.
Attempt at solution: I change this set to an orthogonal set first, and normalize afterwards.
Define: \begin{align*} v_1 &= (1,2,0) \\ v_2 &= (1,0,1) - \frac{ \langle (1,0,1), (1,2,0) \rangle}{\langle (1,2,0), (1,2,0) \rangle} (1,2,0) \\ v_3 &= (2,3,1) - \frac{ \langle (2,3,1), (1,0,1) \rangle}{ \langle (1,0,1), (1,0,1) \rangle} (1,0,1) - \frac{ \langle (2,3,1), (1,2,0) \rangle}{\langle (1,2,0), (1,2,0) \rangle} (1,2,0) \end{align*}
Then this gives \begin{align*} v_2 = (1,0,1) - \frac{1}{1+4} (1,2,0) = (1,0,1) - (\frac{1}{5}, \frac{2}{5}, 0) = (\frac{4}{5}, - \frac{2}{5}, 1) \end{align*} and \begin{align*} v_3 &= (2,3,1) - \frac{(2+1)}{(1+1)} (1,0,1) - \frac{(2+6)}{(1+4)}(1,2,0) \\ &= (2,3,1) - (\frac{3}{2}, 0, \frac{3}{2}) - (\frac{8}{5}, \frac{16}{5}, 0) \\ &= (- \frac{11}{10}, - \frac{1}{5}, - \frac{1}{2}). \end{align*} Now I wanted to check my work. So I compute the innerproduct of $v_2$ with $v_3$, i.e. \begin{align*} \langle (\frac{4}{5}, - \frac{2}{5}, 1), (- \frac{11}{10}, - \frac{1}{5}, - \frac{1}{2}) \rangle \end{align*} but I don't get zero? When I compute $\langle v_1, v_2 \rangle$, however, I do get zero. So something must be wrong with my $v_3$. I triple checked my work and I can't find the mistake. Please help!
The problem is that you projected two non-orthogonal vectors, $v_1$ and $v_2$, out of $v_3$. Each of those orthogonalizations kills the other. In orthogonalizing $v_3$, you need to use the result of orthogonalizing $v_2$ against $v_1$, not $v_2$ itself.