Let
$$\left( {\matrix{ 1 & 0 & { - 3} \cr 0 & 2 & {\lambda + 3} \cr 0 & 0 & {5 - {\lambda ^2}/2 - 3\lambda /2} \cr } \left| {\matrix{ { - 3} \cr 2 \cr {4 - \lambda } \cr } } \right.} \right)$$
I found out that this system doesn't have a solution when $\lambda=-5,2$
Assuming $\lambda\ne-5, 2$, I wish to keep investigate the linear system.
The "problem" is the calculation from this point is getting very messy. Is there some sort of trick to make it easier to solve? Or I just need to "eat that frog"?
Thanks.
I don't think you can do too much: the solution is ugly, so sooner or later it'll get messy.
When $\lambda \not\in \{-5,2\}$, we can continue the row operations: \begin{align*} &\left(\begin{array}{ccc|c} 1 & 0 & -3 & -3 \\ 0 & 2 & \lambda+3 & 2 \\ 0 & 0 & 5-\lambda^2/2-3\lambda/2 & 4-\lambda \\ \end{array}\right) \\ & \xrightarrow{R_3 \gets R_3/(5-\lambda^2/2-3\lambda/2)} \left(\begin{array}{ccc|c} 1 & 0 & -3 & -3 \\ 0 & 2 & \lambda+3 & 2 \\ 0 & 0 & 1 & c \\ \end{array}\right) & \text{where } c=\frac{4-\lambda}{5-\lambda^2/2-3\lambda/2} \\ & \xrightarrow{R_2 \gets R_2-(\lambda+3)R_3} \left(\begin{array}{ccc|c} 1 & 0 & -3 & -3 \\ 0 & 2 & 0 & 2-(\lambda+3)c \\ 0 & 0 & 1 & c \\ \end{array}\right) \\ & \xrightarrow{R_1 \gets R_1+3R_3} \left(\begin{array}{ccc|c} 1 & 0 & 0 & -3+3c \\ 0 & 2 & 0 & 2-(\lambda+3)c \\ 0 & 0 & 1 & c \\ \end{array}\right) \\ & \xrightarrow{R_2 \gets R_2/2} \left(\begin{array}{ccc|c} 1 & 0 & 0 & -3+3c \\ 0 & 1 & 0 & 1-(\lambda+3)c/2 \\ 0 & 0 & 1 & c \\ \end{array}\right). \end{align*} So the reduced row echelon form is $$ \left(\begin{array}{ccc|c} 1 & 0 & 0 & \frac{3\lambda^2+3\lambda-6}{10-\lambda^2-3\lambda} \\ 0 & 1 & 0 & \frac{-4\lambda-2}{10-\lambda^2-3\lambda} \\ 0 & 0 & 1 & \frac{8-2\lambda}{10-\lambda^2-3\lambda} \\ \end{array}\right). $$
It's probably safest to get a computer to do/verify this.