My working
I multiplied the matrix with the matrix $(x, y)$ and named the resultant matrix $(x', y')$
I got: $y' = mx' +c$
$y = mx + c$
(since they both lie on the same line)
$x + 2y = x'$
$5y = y'$
(by multipling the $\begin{bmatrix}1 & 2\\6 & 5\end{bmatrix}$ with $\begin{bmatrix}x \\ y\end{bmatrix}$ and comparing the results to $\begin{bmatrix}x' \\ y'\end{bmatrix}$
well I solved both of them and I got $-2m^{2}x + 4mx -2mc +4c = 0$ I wrote this a bunch of different ways and got "($m = 0$ and $c = 0$) or ($x = 0$ and $c =0$)" which I summed up into "$c = 0$ and ($m=0$ or $x=0$)"
which results in $y = 0 + 0$ ($mx = 0$ since we know either $m$ or $x$ is $0$ so $mx$ will always be $0$)
well $y = 0$ isn't the answer so where is my working wrong?
You missed out the term $6x$ when multiplying $\begin{bmatrix}1 & 2\\6 & 5\end{bmatrix}$ with $\begin{bmatrix}x \\ y\end{bmatrix}$. It should be
$$x+2y=x' \quad (1)$$ $$\color{red}{6x}+5y=y' \quad (2)$$
Then you can substitute $y=mx+c$ for both expression, use $y'=mx'+c$ to substitute $(1)$ into $(2)$ and find the required values.