I've been looking at different sources to see how the transformed unit vectors are displayed in a 3 by 3 matrix, but there seems to be some contradiction.
In my textbook, they give a shear example.
$x' = x$
$y' = y + z$
$z' = z$
$M=\begin{pmatrix}1&0&0\\0&1&1\\0&0&1 \end{pmatrix}$
This is where my confusion lies. If $y' = y + z$, then $y'=\begin{pmatrix}0\\1\\1 \end{pmatrix}$ but they have have written the second column as $\begin{pmatrix}0\\1\\0 \end{pmatrix}$.
No, it's not true that $$y'=\begin{pmatrix}0\\1\\1 \end{pmatrix}.$$ (Obviously, since $y'$ is a scalar and $(0,1,1)^t$ is a vector!)
What's true is that $$y'=\begin{pmatrix}0&1&1 \end{pmatrix} \begin{pmatrix}x\\y\\z \end{pmatrix},$$ which corresponds to the second row in the matrix.