Here's the entire question: Let $A$ be an 8 $\times$ 5 matrix of rank 3, and let $b$ be a nonzero vector in $N(A^T)$.
a) Show that the system $Ax = b$ must be inconsistent. Gonna take a wild stab at this one... If the rank is 3, that means the dimension of the column space is 3. But $A$ has 5 columns, so they are not all linearly independent and therefore $Ax = b$ is inconsistent.
b) How many least squares solutions will the system $Ax = b$ have? Explain.
On previous problems, I found the best least squares linear fit, where the approximation of $x$ was a vector that contained sometimes regular numbers, and sometimes variables. Does this mean that there must be either 1 linear solution or infinite (because you can always find an approximation)? In the example that apparently had an infinite number of least squares solutions, it appeared that one row of $A^TA$ was a constant multiple of another row, leading to a row of zeros in reduced row echelon form. From this problem I know that $A^TA$ is a 5x5 matrix, but I don't think I can prove that any rows are a scalar multiple of other rows, so I'm guessing I have to use some other means of figuring this out.
Sorry if I sound like I have no idea what I'm talking about. Just wanted to try out the problem to my best ability before asking about it.
a
The system is inconsistent. Example: $$ \left[ \begin{array}{cc} 1 & 0 \\ 0 & 0 \end{array} \right] % \left[ \begin{array}{cc} x_{1} \\ x_{2} \end{array} \right] = \left[ \begin{array}{c} 0 \\ 1 \end{array} \right] $$
b
No, the system is not unique. The solution is an affine space: infinite solutions. In linear algebra the number of solution is either 0 (no existence), 1 (existence and uniqueness), or infinite (existence, no uniqueness).
A pencil and paper exercise: Unique least square solutions
Theoretical treatment: Is a least squares solution to Ax=bAx=b necessarily unique
To see a diagram Is the unique least norm solution to Ax=b the orthogonal projection of b onto R(A)?
How the SVD produces the Moore-Penrose pseudoinverse: How does the SVD solve the least squares problem?