The question says to determine if W is a subspace if:
$V = \mathbb{R}^4$, and $W = \{(x_1, x_2, x_3, x_4) \textrm{ in }\mathbb{R}^4 \mid \textrm{det}(\begin{bmatrix}x_1&x_2&x_3&x_4\\1&1&0&-1\\1&0&1&1\\0&1&0&2 \end{bmatrix}) = 0\} $
I understand that these conditions must hold in order for W to be a subspace:
a) W is nonempty, W ≠ 0
b) If x∈W and y∈W, then x+y∈W.
c) If c∈R and x∈W, then cx∈W.
But the determinant is just throwing me off. How do I approach this question? Do they want me to find the determinant first and then check those conditions?
You can actually compute the determinant out as a function of four variables $x_1, x_2, x_3,$ and $x_4$, using the usual cofactor expansion through the first row. Basically, you will get something like $$a_1x_1 - a_2x_2 + a_3x_3 - a_4x_4$$ where $a_1, a_2, a_3, a_4$ are the determinants of the minors $M_{1, 1}, M_{1, 2}, M_{1, 3}, M_{1, 4}$.
Then your set becomes $W = \{(x_1, x_2, x_3, x_4) \mid a_1x_1 - a_2x_2 + a_3x_3 - a_4x_4 = 0\}$. From here, I think you can check the three conditions in a straightforward way.
Intuitively, the answer is yes because you can define a function $f(x_1, x_2, x_3, x_4) = a_1x_1 - a_2x_2 + a_3x_3 - a_4x_4$ and then show that $f$ is a linear tranformation from $\mathbb{R}^4$ to $\mathbb{R}$. Then $W$ is the kernel of $f$, which is necessarily a subspace of the domain of $f$.