Now I really feel that this problem should be straightforward, but I just don't feel right about what am I doing. I am given:
$A = \begin{pmatrix} 1 & -1 & -2 \\ -1 & 0 & -3 \\ -2 & -3 & 7 \end{pmatrix}$ as the matrix of the bilinear form, and the goal is to find find the orthogonal complement of $\text{span}(a_1,a_2)$ where $a_1 = (1,2,3), a_2 = (3,4,5)$.
This is all seems well enough, I consider essentially that I have $\alpha : V \times V \to K$ (field $K$, I think I could just say $\mathbb{R}$, but this is unspecified) as my bilinear map associated with $A$. I consider $W\subset V$ as $W =$ $\text{span}(a_1,a_2)$, then by definition $W^\perp = \{ y\in V : \alpha(x,y) = 0, \forall x \in W\}$ (this being my goal to find).
I can express my linear span $W = \text{span}(a_1,a_2) = \{(x_1+3x_2, 2x_1+4x_2,3x_1+5x_2, x_i \in K\}$. Then using the definition of what it means for a vector $y \in V$ to be in the orthogonal complement I say:
$$ \begin{pmatrix} x_1 + 3x_2 \\ 2x_1+4x_2 \\ 3x_1+5x_2 \end{pmatrix}^T \begin{pmatrix} 1 & -1 & -2 \\ -1 & 0 & -3 \\ -2 & -3 & 7 \end{pmatrix} \begin{pmatrix} y_1 \\ y_2 \\ y_3 \end{pmatrix}=0$$ which gives me $−11x_2y_1−18x_2y_2+17x_2y_3+18x_1=0$. But I'm just not certain about this, I feel it makes sense at every step, but somehow I'm left feeling I could do more?
So essentially my question is have I gone about this correctly? Because the math seems fine by my intuition isn't very satisfied. Thanks very much for any help.
Your space $W$ is generated by the two vectors $a_1$ and $a_2$, hence $b\in W^\perp$ is equivalent to the system: \begin{align*} \langle b,a_1 \rangle_A &=0 \\ \langle b,a_2 \rangle_A &=0 \\ \end{align*} where you scalar product is defined by $\langle v,w \rangle_A = v^t.A.w$
With your $a_1=(1,2,3)$ and $a_2=(3,4,5)$ and $b=(x,y,z)$ you get: \begin{align*} \langle b,a_1 \rangle_A &= -7x-10y+13z &=0 \\ \langle b,a_2 \rangle_A &=-11x-18y+17z &=0 \\ \end{align*} The solution is $$ b=(x,-\frac{3}{8}x,\frac{1}{4}x) $$ hence $W^\perp$ is a one dimensional space generated by the $(1,-\frac{3}{8},\frac{1}{4})$ vector.