My system of equations is: \begin{cases} x + 5y- 6z = 2 \\ kx + y - z = 3 \\ 5x - ky + 3z = 7 \end{cases}
So the augmented matrix is:
$$ \left[ \begin{array}{ccc|c} 1&5&-6&2\\ k&1&-1&3\\ 5&-k&3&7 \end{array} \right] $$
I reduced it to this (with the shown steps), but I'm not sure if i've done so correctly and i'm not sure if i need to do Gauss-Jordan elimination or just Gauss elimination:
\begin{align} r_2 -> r_2 - 2r_1 \end{align} \begin{align} r_3 -> r+3 - 5r_1 \end{align} \begin{align} r_2 -> r_2 * \frac{1}{-9} \end{align} \begin{align} r_1 -> r_1 - 5r_2 \end{align} \begin{align} r_3 -> r_3 +27r_2 \end{align}
$$ \left[ \begin{array}{ccc|c} 1-5(\frac{k-2}{-9})&0&\frac{1}{9}&\frac{13}{9}\\ \frac{k-2}{-9}&1&\frac{-11}{9}&\frac{1}{9}\\ 27(\frac{k-2}{-9})&-k+2&0&0 \end{array} \right] $$
From this reduced from that i got, i noticed that there is an infinite solution for k=2.
However, I believe that i need to get one of the values in the 4th column of the matrix in terms of k so that i can find all the solutions i need, but i am unsure on how to do this.
Could someone please help me and show full working, including the steps taken to reduce the matrix?
From my comments above, I always find the determinant first which gives us
$$\det \begin{bmatrix} 1&5&-6 \\ k&1&-1 \\ 5&-k&3 \end{bmatrix} = 2 (k-2) (3 k-2)$$
This tells us we may have to account for $$k = 2, k = \dfrac{2}{3}$$
The RREF is given by the steps
Swap $R_1$ and $R_2$
Set $R_2 \longleftarrow R_2 - \dfrac{1}{k} R_1$
Set $R_3 \longleftarrow R_3 - \dfrac{5}{k} R_1$
Swap $R_2$ and $R_3$
Set $R_3 \longleftarrow R_3 - \dfrac{5k-1}{-k^2-5} R_2$
Set $R_3 \longleftarrow - \dfrac{k^2+5}{2(3k^2-8k+4)} R_3$
Set $R_2 \longleftarrow R_2 - \dfrac{3k+5}{k} R_3$
Set $R_1 \longleftarrow R_1 + R_3$
Set $R_2 \longleftarrow \dfrac{k}{-k^2-5} R_2$
Set $R_1 \longleftarrow R_1 - R_2$
Set $R_1 \longleftarrow \dfrac{1}{k} R_1$
This results in the RREF of
$$\left[ \begin{array}{ccc|c} 1 & 0 & 0 & \dfrac{8}{3 k-2} \\ 0 & 1 & 0 & -\dfrac{24}{3 k-2} \\ 0 & 0 & 1 & \dfrac{-k-18}{3 k-2} \\ \end{array} \right]$$
We can see that we have an issue for $k = \dfrac{2}{3}$.
Also, from the determinant, when $k = 2$, we have a RREF of
$$\left[ \begin{array}{ccc|c} 1 & 0 & \dfrac{1}{9} & \dfrac{13}{9} \\ 0 & 1 & -\dfrac{11}{9} & \dfrac{1}{9} \\ 0 & 0 & 0 & 0 \\ \end{array} \right]$$