I am confused on how to find the orthogonal compliment for $M=\operatorname{sp}(1,-1,1)$ where $M \subset \Bbb{R} ^3$. My current working is letting $\mathbf x=(x,y,z)$ and then taking $0=x-y+z$ such that $y=x+z$.
From here note that $M^\perp=\operatorname{sp}(1,2,1)$ but I am unsure whether this is correct and what it is telling me. I am then tasked to find $(M^{\perp})^{\perp}$ which I have no idea how to find. Also if anyone could recommend any books on spectral theory that would be great!
Note that $M^{\perp}$ should be $2$-dimensional since $M\subset\mathbb{R}^{3}$ is one-dimensional. You correctly noted that $$M^{\perp}=\{(x,y,z)\in\mathbb{R}^{3}:y=x+z\},$$ from which it follows that $$M^{\perp}=\text{Span}\{(1,1,0),(0,1,1)\}.$$
Next, it is true in general that $(M^{\perp})^{\perp}=M$.