Let $V = R^3$ with the standard inner product $u = (2,1,3)$ and $W = \{(x,y,z) : x + 3y - 2z = 0\}$
I came up with the basis $\{(-3,1,0), (2,0,1)\}$ but these are not orthogonal to each other. I'm not exactly sure how to approach this question, any help would be appreciated. Thanks
There are many ways how to find an orthogonal projection.
You seem to want to use an orthogonal (or an orthonormal) basis of $W$ in some way.
If you already have a basis of $W$, you can get an orthogonal basis from it using Gram-Schmidt process.
Another way to do this. Let us choose $\vec b_1=(2,0,1)$ at the first vector basis. Now you want a find another vector which belongs to $W$ (i.e., it satisfies $x+3y-z=0$) and which is orthogonal to $\vec b_1$ (i.e., it satisfies $2x+z=0$). Can you find solution of these two equations? Can you use it to get an orthogonal basis of $W$?
Solution using a linear system. Here is another way to find an orthogonal projection. We are given a vector $\vec u=(2,1,3)$. And we want to express it as $\vec u=\vec u_1+\vec u_2$, where $\vec u_1 \in W$ and $\vec u_2=W^\bot$. We know bases of $W=[(-3,1,0),(2,0,1)]$ and of $W^\bot=[(1,3,-2)]$.
So we simply express the vector $\vec u$ as a linear combination $\underset{\in W}{\underbrace{c_1(-3,1,0)+c_2(2,0,1)}}+\underset{\in W^\bot}{\underbrace{c_3(1,3,-2)}}$.
To find $c_{1,2,3}$ it suffices to solve the system of equations $$ \left(\begin{array}{ccc|c} -3 & 2 & 1 & 2 \\ 1 & 0 & 3 & 1 \\ 0 & 1 &-2 & 3 \end{array}\right) $$ If you do so, you will find that the only solution is $c_1=\frac{17}{14}$, $c_2=\frac{20}7$, $c_3=-\frac1{14}$.
This gives you $\vec u_1=\underline{\underline{\frac1{14}(29,17,40)}}$ and $\vec u_2=\frac1{14}(-1,-3,2)$.
Projection to $W^\bot$. As mentioned in a comment, since $W^\bot$ is one-dimensional, it is easy to find projection to $W^\bot$. The vector $\vec a=\frac1{\sqrt{14}}(1,3,-2)$ is unit vector which spans $W^\bot$. The projection can be found as $$\vec u_2 = \vec u \vec a^T \vec a =\frac1{14} (2,1,3)\begin{pmatrix}1\\3\\-2\end{pmatrix}(1,3,-2)=-\frac1{14}(1,3,2).$$
Then the projection to the subspace $\vec u_1$ can be computed as $\vec u_1=\vec u-\vec u_2$.