Can somebody tell me where I made a mistake? How would you approach such an exercise? Was my way too complicated?
$ B:\mathbb{R}^3\times \mathbb{R}^3\to \mathbb{R},\quad B(x,y)=\sum \limits_{i\neq j}^{}x_{i}y_{j}+2\sum \limits_{i=1}^{3}x_{i}y_{i}=x_{1}\cdot (2y_{1}+y_{2}+y_{3})+x_{2}\cdot (y_{1}+2y_{2}+y_{3}) +x_{3}\cdot (y_{1}+y_{2}+2y_{3}) $
The exercise was to first determine an orthonormal basis for $(1,1,1)^{\perp }$ ( the orthogonal complement of (1,1,1)) regarding $B$.
So I guessed $v:=(-1,1,0)^T$ as one orthogonal vector of $(1,1,1)$, which is true because $B(\begin{pmatrix} 1\\1\\1 \end{pmatrix}, \begin{pmatrix} -1\\1\\0 \end{pmatrix}) =0 $
Then I did the following:
(i) $B(\begin{pmatrix} 1\\1\\1 \end{pmatrix}, \begin{pmatrix} y_{1}\\y_{2}\\y_{3} \end{pmatrix}) = 4y_{1}+4y_{2}+4y_{3} =0 $
(ii) $B(\begin{pmatrix} -1\\1\\0 \end{pmatrix}, \begin{pmatrix} y_{1}\\y_{2}\\y_{3} \end{pmatrix})= -y_{1}+y_{2} =0 \leftrightarrow y_{1}=y_{2}$
So I defined $y_{1}=y_{2}=\mu $ with $\mu\in \mathbb{R}$. If you put that value in (i) you get $y_{3}=-2\mu $, $w:= \mu \begin{pmatrix} 1\\1\\-2 \end{pmatrix})$
Now I calculated $B(\begin{pmatrix} 1\\1\\1 \end{pmatrix}, \begin{pmatrix} 1\\1\\1 \end{pmatrix}) =12, B(\begin{pmatrix} -1\\1\\0\end{pmatrix}, \begin{pmatrix} -1\\1\\0 \end{pmatrix}) =2, B(\begin{pmatrix} 1\\1\\-2 \end{pmatrix}, \begin{pmatrix} 1\\1\\-2 \end{pmatrix}) =6\\ \rightarrow\Vert (1,1,1)^T \| =\sqrt{12}, \quad \Vert (-1,1,0)^T \| =\sqrt{2},\quad \Vert (1,1,-2)^T \| =\sqrt{6}$
So $C=(\frac{(-1,1,0)^T}{\sqrt{2}}, \frac{(1,1,-2)^T}{\sqrt{6}})$ is an orthonormal basis for $(1,1,1)^{\perp }$ regarding B.
The next exercise was to determine a orthonormal basis of $\mathbb{R}^3$ regarding $B$.
Since $D=(\frac{(1,1,1)^T}{\sqrt{12}},\frac{(-1,1,0)^T}{\sqrt{2}}, \frac{(1,1,-2)^T}{\sqrt{6}}) $are linear independent, D should be a orthonormal basis of $\mathbb{R}^3$ regarding $B$.
But then $Q$ with the vectors of $D$ as rows should be an orthogonal matrix, so that $Q\cdot Q^{T}=E_{3}$, with $E_{3}$ being the 3x3 identity matrix.
$\begin{pmatrix} \frac{\sqrt{12}}{12} & \frac{-\sqrt{2}}{2} & \frac{\sqrt{6}}{6} \\ \frac{\sqrt{12}}{12} &\frac{\sqrt{2}}{2} &\frac{\sqrt{6}}{6} \\\frac{\sqrt{12}}{12}&0&\frac{-\sqrt{6}}{3} \end{pmatrix}\cdot \begin{pmatrix} \frac{\sqrt{12}}{12} &\frac{\sqrt{12}}{12} &\frac{\sqrt{12}}{12} \\ \frac{-\sqrt{2}}{2} &\frac{\sqrt{2}}{2} &0 \\\ \frac{\sqrt{6}}{6}& \frac{\sqrt{6}}{6}&\frac{-\sqrt{6}}{3} \end{pmatrix} \neq E_{3}$
That is the reason why I think that I made a mistake.
PS: I'm not used to writing about math in English; please ask if something doesn't make sense to you.
You're so close. When you write $$ Q^t Q = I, $$ that means that the dot-product of the $i$th column of q with the $j$th column of $Q$ is $\delta_{ij}$, the $ij$-element of the identity. But the "dot product" here is the ordinary dot product. You want to make this claim using YOUR inner product. For that, you aim to get $$ Q^t B Q = I $$ where $B$ is the matrix representing your inner product. If you need to do it with rows instead of columnss, then you're hoping for $$ Q B Q^t = I. $$
Post-comment additions You've got the matrix $Q$ transposed. You said that you wanted the basis vectors as rows of $Q$, but your last equation has them as columns. This isn't very important, however.
The matrix $Q$ (with your vectors as rows) has the property that $$ Q B Q^t = I $$ You can see that here numerically, from a matlab program I wrote. (In matlab, multiplying means "matrix multiply", and
A'means "the transpose of matrixA". A semicolon at the end of a line suppresses output.)The output of this program is just what you'd expect (except that I've added comments after
//, and deleted some blanks lines):You've raised the concern that "det Q" is not 1, which it should be if $Q$ is orthogonal...and that's correct, if it's orthogonal with respect to the usual inner product. But it's actually orthogonal with respect to the inner product defined by $B$, which gives an altogether different result, as you discovered.
In short: aside from using columns instead of rows when you built $Q$, everything you did is just fine. Your matrix $Q$ really is orthogonal (in the $B$-metric).