If $A = \begin{bmatrix}1 & -3\\-3 & 5\end{bmatrix}$ and $AB = \begin{bmatrix}-11 & -3 & 7\\13 & 1 & 9\end{bmatrix}$ determine the first and second columns of B.
I'm not quite sure how to do this. My intuition would be to "divide" (sorry, I don't know what the correct term is, I'm just using MATLAB terminology here) AB by A, but that would result in a 3x2 matrix, not a 2x3 matrix...
The program is recommending using an augmented matrix for $A=b_1$ and $A=b_2$, where $b_x$ is a column from B, but I'm not sure how that helps me.
Since $A$ is $2\times 2$ and $AB$ is $2\times 3$, $B$ must have been $2\times 3$ too. The hint is suggesting that you view the entries of $B$ as unknowns, so your second equation is $$ \begin{bmatrix}1&-3\\-3&5\end{bmatrix} \begin{bmatrix}x & z & p\\y & w & q\end{bmatrix} = \begin{bmatrix}-11 & -3 & 7\\13 & 1 & 9\end{bmatrix} $$ When you expand the matrix multiplication on the left, this becomes a system of linear equations: $$ x-3y = -11 \\ -3x+5y=13 \\ z-3w = -3 \\ -3+5w = 1 $$ in addition to some equations involving $p$ and $q$ which are not interesting as long as we're after only the two first columns of $B$.
Solving the two first equations for $x$ and $y$ by the book leads us to doing Gaussian elimination on $$ \left[ \begin{array}{cc|c} 1 & -3 & -11 \\ -3 & 5 & 13 \end{array} \right] $$ Note that the left side of this is just $A$ and the right side is a column of $AB$.
You can solve for $z$ and $w$ similarly -- but actually the row operations you'll do in each case are exactly the same, because they're governed entirely by the values in the $A$ part of the extended matrix. So we can do everything in one go by Gaussian elimination on $$ \left[ \begin{array}{cc|cc} 1 & -3 & -11 & -3 \\ -3 & 5 & 13 & 1 \end{array} \right] $$
It should be clear that we can recover all of $B$ simply by doing elimination on the entire $[A\mid B]$.
These row operations, by the way, are exactly the row operations that make $A$ into $I$, so what they actually do is multiplying from the left by $A^{-1}$ and recovering $B$ as $B=A^{-1}(AB)$. For a case as small as this one it's a bit of a toss-up whether it is easier to Gauss-eliminate the $5\times 2$ matrix, or to compute $A^{-1}$ first and then multiply it by $AB$.