What's wrong with my approach to transforming this plane:
$$2x + y - z =12$$
under this transformation: $\begin{pmatrix}1&2&0\\ 1&0&1\\ 1&1&1\end{pmatrix}$
My answer in the end was horrendously wrong. Here is what I did:
I applied the transformation to the normal, and then to a point in the plane, ie, $\begin{pmatrix}4\\ 1\\ 2\end{pmatrix}$, the new normal and $\begin{pmatrix}0\\ 0\\ -12\end{pmatrix}$, the new point. I then used the formula $r.n = a.n$ and converted that into a cartesian equation for the plane:
$$4x+y+2z = -36$$
The correct answer is $$3x +4y - 5z=12$$
What did I incorrectly assume? If all the points are transformed by the same matrix, surely the transformed normal and point on the plane is all you need to write a new equation?
Your mistaken assumption is that plane normals transform in the same way as do points. Unfortunately, they don’t: plane normals are covariant. If the point transformation is given by $\mathbf x'=M\mathbf x$, then the normal is transformed by $M^{-T}$, the transpose of the inverse of $M$.
To see this, write the equation of the plane as $\mathbf n\cdot(\mathbf x-\mathbf x_0)=0$. Substituting for $\mathbf x$ and $\mathbf x_0$ in this equation produces $$\mathbf n\cdot(M^{-1}\mathbf x'-M^{-1}\mathbf x_0') = \mathbf n\cdot \left(M^{-1}(\mathbf x'-\mathbf x_0')\right) = \left(\mathbf M^{-T}\mathbf n\right)\cdot(\mathbf x'-\mathbf x_0') = \left(\mathbf M^{-T}\mathbf n\right)\cdot(\mathbf x'-M\mathbf x_0) = 0.$$ So, the known point on the plane becomes $M\mathbf x_0$, as you’d expect, but the normal of the transformed plane is $M^{-T}\mathbf n = (3,4,-5)$.
Now, if $M$ happens to be an orthogonal transformation—a rotation or reflection—then $M^{-T}=M$ and normals will appear to transform in the same way as do points.