I'm stuck with this problem. I pulled apart x and y so x = \begin{bmatrix}1\\0\\1\end{bmatrix} and y = \begin{bmatrix}0\\1\\1\end{bmatrix} and set them up for addition. I did the process for finding the orthogonal components, then I added them back together and I got a solution of \begin{bmatrix}1/2\\-1/2\\1\end{bmatrix} and the answer is incorrect. How is this solved?
Hint: As you (sort of) said, $W$ is the space spanned by the vectors $$ v_1 = \pmatrix{1\\0\\1}, \quad v_2 = \pmatrix{0\\1\\1} $$ in particular, $W$ is the set of all vectors of the form $xv_1 + yv_2$. Now, if a vector $u = (u_1,u_2,u_3)$ is in $W^\perp$, the orthogonal complement of $W$, then $u$ must be orthogonal to both $v_1$ and $v_2$. In particular, we must have $v_1^Tu = 0$ and $v_2^Tu = 0$ ($v^Tu$ is the dot-product of $v$ and $u$). Writing out the equations, we have $$ (1)u_1 + (0)u_2 + (1)u_3 = 0\\ (0)u_1 + (1)u_2 + (1)u_3 = 0 $$ which is to say that $$ \pmatrix{1&0&1\\0&1&1} \pmatrix{u_1\\u_2\\u_3} = \pmatrix{0\\0} $$ in other words, the orthogonal complement $W^\perp$ is exactly the same as the nullspace of the marix $$ \pmatrix{1&0&1\\0&1&1} $$ so, what you need to do is find a basis for the nullspace of this matrix (which is already in RREF).