Let $\mathbb{R^3}$ be equipped with the inner product $<,>$ defined by setting
$$<\mathbf{u},\mathbf{v}>=2u_1v_1+5u_2v_2+3u_3v_3$$ for any pair of vectors $\mathbf{u}=(u_1,u_2,u_3)$ and $\mathbf{v}=(v_1,v_2,v_3)$. Which of the following subsets of $\mathbb{R}^3$ are bases of the orthogonal complement of the line $L=span\left\{1,-1,1\right\}$ in $\mathbb{R^3}$ relative to $<,>$?
i) $\left\{(1,1,0),(0,1,1)\right\}$
ii) $\left\{(-2,4,8),(3,-6,-12)\right\}$
iii) $\left\{(3,6,-12),(1,2,-4)\right\}$
iv) $\left\{(-2,4,8),(1,1,1)\right\}$
v) $\left\{(-2,4,8),(0,0,0)\right\}$
$(A)$ Just i.
$(B)$ Just iii.
$(C)$ Just iv.
$(D)$ ii and iv.
$(E)$ ii, iv and v.
My idea was to do the inner product with some arbitrary $x$ and $y$ such that:
$\mathbf{x}=(x_1,x_2,x_3)$
$\mathbf{y}=(y_1,y_2,y_3)$
Thus, $<\mathbf{q},\mathbf{x}>=0$ and $<\mathbf{q},\mathbf{y}>=0$ where
$\mathbf{q}=(1,-1,1)$
Thus:
$<\mathbf{q},\mathbf{x}>=2x_1-5x_2+3x_3=0$
$<\mathbf{q},\mathbf{y}>=2y_1-5y_2+3y_3=0$
Then we check each point in the answer choices and see which pair of $x$ and $y$ points work:
For instance in choice $(i)$, $x=(1,1,0)$ and $y=(0,1,1)$. Therefore:
$2(1)-5(1)+3(0)=-3$ (For $x$)
$2(0)-5(1)+3(0)=-2$ (For $y$)
Because this is not zero, this is not an orthogonal complement of the line $L$.
For choice $(ii)$:
$2(-2)-5(4)+3(8)=0$ (For $x$)
$2(3)-5(-6)+3(-12)=0$ (For $y$)
Because this is zero, this is an orthogonal complement of the line $L$.
Similar calculations for the other points yield that only $ii,iv,v$ work and thus the answer is $(E)$. However, it says the correct answer is $(C)$. Why is that?
The basis of $L^\perp$ consists of $(2)$ linearly independent vectors. The vectors in $(ii),(v)$ are linearly dependent, and can't form the basis of $L^\perp$.
As an additional comment on your method of solving, you don't need to take two vectors $\bf x,y$. Just take a general vector $\mathbf x\in L^\perp$, and the condition you get must be satisfied by all vectors in $L^\perp$. To then find the basis of $L^\perp$, you need to see which option contains the required number of linearly independent vectors that satisfy the condition.