I need to tell if the following system of equations has either:
- A unique solution, or
- Infinite solutions, or
- No solutions at all.
(for all possible $R$eal values of the given parameter, $m$)
To solve this question I decided to use the well-known Rouché-Capelli theorem. The system of equations under study is: $$\begin{cases} x + my + z = 0 \\ mx – y + 3z = 0\\ x - 3y + mz =0\end{cases}$$
Let's study the rank of the augmented matrix, which I will call $A^*$: $$A^*=\left(\begin{array}{ccc|c} 1 & m & 1 & 0\\ m & -1 & 3 & 0\\ 1 & -3 & m & 0\end{array}\right)\sim \ldots \sim \left(\begin{array}{ccc|c} 1 & m & 1 & 0\\ 0 & (1+m^2)(3+m) & m^2-9 & 0\\ 0 & 0 & 10-m-m^3 & 0\end{array}\right)$$
I obtained the last equivalent matrix after several operations I performed on the original augmented matrix. I double-checked that it is correct, which I'm 100% confident it is.
The thing is that, for $m=2$, both the augmented matrix ($A^*$) and the coefficient matrix ($A$) have rank equal to 2, so there would be infinite solutions in this case.
Now my question
If you give the previous matrix a second look, you will notice that, for $m=-3$, the rank of both $A^*$ and $A$ would be 2 again: $$A^*(m=-3)=\left(\begin{array}{ccc|c} 1 & -3 & 1 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 40 & 0\end{array}\right)\sim \left(\begin{array}{ccc|c} 1 & -3 & 1 & 0\\ 0 & 0 & 40 & 0\\ 0 & 0 & 0 & 0\end{array}\right)$$
And the solutions would be given by: $$\left(\begin{array} \\x \\ y \\ z\end{array}\right)=\lambda\,\left(\begin{array} \\3\\ 1\\ 0\end{array}\right)$$
However this is incorrect. Following my book's solution, $m=-3$ is no special case. But why? We can clearly see that the rank of the system would still be 2, and in theory there would be infinite solutions (rather than just the trivial solution, $(x,y,z)=(0,0,0)$)
Where did I go wrong? Why do I get $m=-3$ as a special case?
EDIT
Let me add the operations I performed on $A^*$ to put it in its Row echelon form:
In your computation of the rank of $A^*$ to go from the third matrix to the fourth you multiply the second row by $(3+m)$. But this is a rank preserving operation only if $3+m \neq 0$.