Let $S[(x_1,y_1,z_1)(x_2,y_2,z_2)]=3x_1x_2+x_2y_1+x_1y_2+y_1y_2+z_1z_2$ be an inner product,
$W=${$(x,y,z)\in\mathbb{R}^3|x+z=0$} a subspace and $\vec u=(1,0,0)$
Find $w_1\in W,$ $w_2\in W^\perp$ (orthogonal in respect of $S$) such that $\vec u=\vec w_1+\vec w_2$
Here's what I've done so far:
An orthogonal basis of W in respect of the inner product $S$ is $B=${$(1,0,-1),(-\frac{1}{4},1,\frac{1}{4}$)}
If $w=(x,y,z)\in W^\perp$ then $S[(x,y,z),(1,0,-1)]=0$ and $S[(x,y,z),(-\frac{1}{4},1,\frac{1}{4})]=0$
So $3x+y-z=0$ and $\frac{1}{4}x+\frac{3}{4}y+\frac{1}{4}z=0\Rightarrow x+3y+z=0$
Solving the above system we get that the basis for $W^\perp$ in respect of $S$ is $B'=\left \{(1,-1,0),(0,0,1) \right \}$
I don't know how to proceed from here..
Also, are the $\vec w_1$, $\vec w_2:\vec u=\vec w_1+\vec w_2$ unique?
You have found the basis for $W^\perp$ wrongly as those elements do not satisfy $3x+y-z=0$. To compute a solution for
$$3x+y-z=0$$
$$x+3y+z=0$$
We can simply compute the cross produdct:
$$\begin{bmatrix} 3 \\ 1 \\ -1\end{bmatrix} \times \begin{bmatrix} 1 \\ 3 \\ 1\end{bmatrix}=\begin{bmatrix}4 \\ -4 \\ 8 \end{bmatrix}=4 \begin{bmatrix} 1 \\ -1 \\ 2\end{bmatrix}$$
Hence a basis of $W^\perp = \left\{ ( 1 , -1 ,2)\right\}$
Let's project $u$ onto $W^\perp$. $$S[(x_1,y_1,z_1)(x_2,y_2,z_2)]=3x_1x_2+x_2y_1+x_1y_2+y_1y_2+z_1z_2$$ $$S[(1,-1,2),(1,-1,2)]=3(1)(1)+(1)(-1)+(1)(-1)+(-1)(-1)+(2)(2)=6$$ $$S[(1,0,0),(1,-1,2)]=3(1)(1)+(1)(0)+(1)(-1)+(0)(-1)+(0)(2)=2$$ $$w_2=\frac{2}{6}(1,-1,2)=(\frac13,-\frac13,\frac23)$$
$$w_1=w-w_2=(1,0,0)-(\frac13,-\frac13,\frac23)=(\frac23,\frac13,-\frac23)$$
Yes, the decomposition is unique.