Let $U\subset\mathbb{R}^3$ be the space spanned by the vectors $v_1=[1,-1,0]^T$ and $v_2=[1,0,-1]^T$. Determine a basis for $U^\perp.$
In the solution they say that:
$U^{\perp}$ is equal to the nullspace of the matrix
$$A^T=\begin{bmatrix}1 & -1 & 0 \\ 1 & 0 & -1\end{bmatrix},$$
where its rows are $v_1^T$ and $v_2^T$. By rowreduction we obtain $U^{\perp}=\text{span}(u)$ where $u=[1,1,1]^T.$
Questions:
1) Does this mean that $U$ is not a space but a plane since two vectors in space can only span a plane?
2) Why is it equal to the nulllspace of $A^T?$
3) Why is it not equal to the nullspace of $A$ instead?
4) What is the answer? They have not given a bssis for $U^{\perp}?$
1) Does this mean that $U$ is not a space but a plane since two vectors in space can only span a plane?
Any plane through the origin is a vector space. So it can be both.
2) Why is it equal to the nulllspace of $A^T?$
For real values matrices, it's because the dot product is defined by the transpose: $u \cdot v= u^Tv$. So since you're looking for all the vectors perpendicular to the column space of $A$ this means finding all $x$ such that $a \cdot x=a^Tx=0$ for all a in $A$. In particular it's enough to check that this works for every vector that is in a basis for $A$ (or a span of the columns). Write out $$ A^Tx=\begin{bmatrix}v_1^Tx\\ v_2^Tx\end{bmatrix}=\begin{bmatrix}v_1\cdot x\\ v_2\cdot x\end{bmatrix} $$ So you can see that settin $A^Tx=0$ gives a solution to all vector perpendictular to these vectors.
3) Why is it not equal to the nullspace of $A$ instead?
The null space of $A$ is the solutions to $Ax=0$ this is looking for solutions to $A^Tx=0$. The nullspace of $A$ is a subspace of $\mathbb{R}^3$ in your example the nullspace of $A^T$ is a subspace of $\mathbb{R}^2$ so they can't even be the same.
4) What is the answer? They have not given a bssis for $U^{\perp}?$
First note they have given you a basis for $U$ then they find the basis using something like the following
Row reduce
$$A^T=\left[\begin{array}{rrr}1 & -1 & 0 \\ 1 & 0 & -1\end{array}\right] \sim \left[\begin{array}{rrr}1 & 0 & -1 \\ 0 & 1 & -1\end{array}\right] $$
Note when you solve $Ax=0$ the 3rd coordinate of $x_3$ variable is free so $x_3=t$. Then multiplying out by $x$ you get equations $$ \begin{align} x_1-x_3&=0\\ x_2-x_3&=0 \end{align} $$ So solving gives $x_1=x_2=x_3=t$ so solutions have the form $x=t\begin{bmatrix}1\\1\\1\end{bmatrix}$. So a basis for $U^\perp$ is $\begin{bmatrix}1\\1\\1\end{bmatrix}$