The matrix $A$ is given by $$\left(\begin{array}{ccc} 1 & 2 & 3 & 4\\ 3 & 8 & 11 & 8\\ 1 & 3 & 4 & \lambda\\ \lambda & 5 & 7 & 6\end{array} \right)$$
Given that $\lambda$=$2$, $B$=$\left(\begin{array}{ccc} 2 \\ 4 \\ \mu \\ 3 \end{array} \right)$ and $X$=$\left(\begin{array}{ccc} x \\ y \\ z \\ t \end{array} \right)$
Find the value of $\mu$ for which the equations defined by $AX=B$ are consistent and solve the equations in this case. State the rank of A.
So I began by reducing matrix $A$ to reduced row echelon form (kind of like taking the null space, except I'm dealing with $Ax=B$ instead of $Ax=0$) but since I have 5 variables and only 4 equations, I'm not sure how to continue onward.
First, note that $$ \det A=2\,(\lambda-2)^2 $$ so $A$ is invertible if and only if $\lambda\neq 2$. In this case we can use Cramer's rule to solve for $X$. For example, we have $$ x= \frac{ \begin{vmatrix} 2 & 2 & 3 & 4 \\ 4 & 8 & 11 & 8 \\ \mu & 3 & 4 & \lambda \\ 3 & 5 & 7 & 6 \end{vmatrix} } { \begin{vmatrix} 1 & 2 & 3 & 4\\ 3 & 8 & 11 & 8\\ 1 & 3 & 4 & \lambda\\ \lambda & 5 & 7 & 6 \end{vmatrix} } = \frac{0}{2\,(\lambda-2)^2} = 0 $$ and solving for $y$, $z$, and $t$ is similar.
Now, if $\lambda=2$, then row-reducing shows that $$ \DeclareMathOperator{rref}{rref}\rref \begin{bmatrix} 2 & 2 & 3 & 4 & 2 \\ 3 & 8 & 11 & 8 & 4 \\ 1 & 3 & 4 & 2 & \mu \\ 2 & 5 & 7 & 6 & 3 \end{bmatrix} = \begin{bmatrix} 1 & 0 & 1 & 8 & 0 \\ 0 & 1 & 1 & -2 & 0 \\ 0 & 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 & 0 \end{bmatrix} $$ which implies the system $AX=B$ is inconsistent (do you see why?).
Finally, the above also shows that $$ \rref A = \begin{bmatrix} 1 & 0 & 1 & 8 \\ 0 & 1 & 1 & -2 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{bmatrix} $$ What does this say about the rank of $A$?