The matrix in question is A: \begin{bmatrix}6&-1&-3\\6&-1&-3\\2&-5&5\end{bmatrix} and so I get a system of equations from AX=0
6x - y - 3z = 0
6x - y - 3z = 0
2x - 5y + 5z = 0
which I then apply Gauss elimination to:
14y - 18z = 0 (added 3rd equation multiplied by -3)
2x - 5y + 5z = 0
Now it seems I cannot solve it any further and so I put z = t and y = 2t, which gives me x = 5t/2. However, this does not seem to be correct! What am I doing wrong? Can it in fact be solved further?
You made a minor error by taking $y=2t.$ Since $7y=9z.$ Therefore $$2x=5(y-z)=5\left(\frac{9}{7}-1\right)z=\frac{10}{7}z,$$ i.e. $x=\frac{5}{7}z.$ Thus the solution is $$\left\{\begin{bmatrix}5\\9\\7\end{bmatrix}t:t\in \mathbb{R}\right\}$$