I am given a vector:
\begin{bmatrix}1\\-1\\3\end{bmatrix}
The inner product is defined:
$$ \langle \, f, \, g \, \rangle =a^\top\!* \begin{bmatrix}2&1&0\\1&2&-1\\0&-1&2\end{bmatrix} * b $$
How can I compute the length of the vector.
$ ||a||$ = $\sqrt{\langle a,a\rangle}$ = $\sqrt{ 2a(1)^2 +a(1)*a(2) +a(2)*a(1) +2a(2)^2 -a(3)*a(2)-a(3)*a(2) +a(3)^2}$ = $\sqrt{ 2 -1 -1+2+3+3+9}$
As the result, I got square root of 20, which is incorrect. Could anyone help me to find a mistake in my calculations?
I see no way to get $20$ from what you wrote under the radical. That expression evaluates to $17$.
Also, the algebraic expression under your radical has a bug: it should be "$2 a(3)^2$". (The diagonal of your matrix is $(2,2,2)$, so all the squared terms should have coefficient $2$.) This would give an extra $9$ to your expression under the radical.
Let's try this computation a different way, which has an extra step, but is usually easier in each step. Call your matrix $M$. Then \begin{align*} M.a &= (1, -4,7) \\ a^\mathrm{T}.(M.a) &= \begin{pmatrix} 1 \\ -1 \\ 3 \end{pmatrix} \cdot (1, -4, 7) \\ &= 26 \text{,} \end{align*} which should be the value under the radical.