In the book by Serge Lang, titled 'Introduction to Linear Algebra', there is given on pg. #88, as below:
Let $A = (a_{ij}), B = (b_{jk})$ & let $AB =C$ with $C =(c_{ik})$. Let $C^k$ be the $k$-th column of $C$. Express $C^k$ as a linear combination of the columns of $A$. Describe precisely which are the coefficients, coming from the matrix $B$.
Let us take an example with given size of matrices $A,B$, such that the sizes differ for all $3$ matrices; e.g. $A$ is a $2*3$ matrix, $B$ is a $3*2$ matrix; while their product is a $2*2$ one; while the terms are general ones.
I hope this will help understand the general case of matrices: $A$ of size $m*n$, $B$ of size $n*k$, $AB$ of size $m*k$.
$A = \begin{bmatrix} a_{11} & a_{12} & a_{13}\\ a_{21} & a_{22} & a_{23} \\ \end{bmatrix} $, $B = \begin{bmatrix} b_{11} & b_{12}\\ b_{21} & b_{22}\\ b_{31} & b_{32} \end{bmatrix} $, $C = AB = \begin{bmatrix} a_{11}b_{11} + a_{12}b_{21} + a_{13}b_{31} & a_{11}b_{12} + a_{12}b_{22} + a_{13}b_{32}\\ a_{21}b_{11} + a_{22}b_{21} + a_{23}b_{31} & a_{21}b_{12} + a_{22}b_{22} + a_{23}b_{32}\\ \end{bmatrix}$
Let $k=2$ (out of the two possible values for $k=1,2$), for $C^2$ or the $2$-nd column of $C$, and follow the approach given here to express as a linear combination of $A$.
Let us express the matrix $A$ in terms of columns only as : $A = \begin{bmatrix}
A^1 & A^2 & A^3\\
\end{bmatrix} $. So, for $C$ have $\begin{bmatrix}
A^1 & A^2 & A^3\\
\end{bmatrix} \begin{bmatrix}
b_{11} & b_{12}\\
b_{21} & b_{22}\\
b_{31} & b_{32}
\end{bmatrix} = \begin{bmatrix}
C^1 & C^2
\end{bmatrix} = \begin{bmatrix}
A^1 b_{11}+ A^2 b_{21}+ A^3 b_{31} & A^1 b_{12}+A^2 b_{22}+A^3 b_{32}
\end{bmatrix},$
with $C^2$ being the second term .
I feel tempted here (due to lack of clarity due to generalized terms) to consider a practical example of two linearly independent vectors $A,B$, with given dimensions. Taking actual value based example for $A,B$, get:
$A = \begin{bmatrix} 2 & 3 & 4\\ 5 & 6 & 7 \\ \end{bmatrix} $, $B = \begin{bmatrix} 1 & 2\\ 3 & 4\\ 5 & 6 \end{bmatrix} $, $AB = \begin{bmatrix} 2\times 1 + 3\times 3 + 4\times 5 & 2\times 2 + 3\times 4 + 4\times 6\\ 5\times 1 + 6\times 3 + 7\times 5 & 5\times 2 + 6\times 4 + 7\times 6\\ \end{bmatrix}$ $AB = \begin{bmatrix} 2 + 9 + 20=31 & 4 + 12 + 24=40\\ 5 + 18 + 35=58 & 10 + 24 + 42=76\\ \end{bmatrix}$
Want verify the above expression $C^2$, for the given example with $A^1 = \begin{bmatrix} 2 \\ 5 \\ \end{bmatrix}, A^2 = \begin{bmatrix} 3 \\ 6 \\ \end{bmatrix}, A^3 = \begin{bmatrix} 4 \\ 7 \\ \end{bmatrix}$.
So, $C^2 = \begin{bmatrix} A^1b_{12}+A^2b_{22}+A^3b_{32} \end{bmatrix} = \begin{bmatrix} 2 \\ 5 \\ \end{bmatrix}\cdot 2 + \begin{bmatrix} 3 \\ 6 \\ \end{bmatrix}\cdot 4 + \begin{bmatrix} 4 \\ 7 \\ \end{bmatrix}\cdot 6 $.
Although am clueless, yet follow the hint given here at mse, even though my chosen matrix $A$ would not lead to an identity ($I$) matrix (as, the $rref$ form works for rectangular matrices. But, it will 'only' produce leading row entry (left to right, across rows) as $1$).
Still would follow it to find $rref$ form of the augmented matrix $[A|AB]$, given as :
$[A|AB] = \begin{array}{ccc|cc}
a_{11} & a_{12} & a_{13}&a_{11}b_{11} + a_{12}b_{21} + a_{13}b_{31} & a_{11}b_{12} + a_{12}b_{22} + a_{13}b_{32}\\
a_{21} & a_{22} & a_{23} & a_{11}b_{11} + a_{12}b_{21} + a_{13}b_{31} & a_{11}b_{12} + a_{12}b_{22} + a_{13}b_{32}\\
\end{array} $
which for the example at hand is:
$[A|AB] = \begin{array}{ccc|cc}
2 & 3 & 4 & 31 & 40\\
5 & 6 & 7 & 58 & 76\\
\end{array}\implies \begin{array}{ccc|cc}
2 & 3 & 4 & 31 & 40\\
0 & -1.5 & -3 & -19.5 & -24\\
\end{array}\implies \begin{array}{ccc|cc}
2 & 0 & -2 & -7 & -16\\
0 & -1.5 & -3 & -19.5 & -24\\
\end{array}\implies \begin{array}{ccc|cc}
1 & 0 & -1 & -7/2 & -8\\
0 & 1 & 2 & 13 & 36\\
\end{array}$
Please help me with : (i) vetting for any error above,
(ii) please tell if have expressed above $C^2$ correctly as a linear combination of $A$,
(iii) how to find out the coefficients coming from matrix $B$.
Say $A$ has $n$ columns $A^i$, $1\le i\le n$. We can then describe $A$ using the columns of $A$ thus: $A = \begin{bmatrix}A^{1} & A^{2} & \dots & A^{j} \dots & A^{n} \end{bmatrix}$. To multiply $A$ and $B$, $B$ necessarily has $n$ rows, and so the $k$th column of $C=AB$, or $C^k$, is formed by multiplying $A$ with the $k$th column of $B$, or $B^k$, thus $$ \begin{bmatrix} A^{1} & A^{2} & \dots & A^{j} \dots & A^{n} \end{bmatrix} \begin{bmatrix} b_{1k}\\ b_{2k}\\ \vdots\\ b_{jk}\\ \vdots\\ b_{nk}\\ \end{bmatrix}=b_{1k}A^1+b_{2k}A^2+\dots+b_{jk}A^j+\dots+b_{nk}A^n\tag{1}$$
Only square matrices have the possibility of being invertible, so you can't row reduce to isolate $B$ in the non-square case.
Once you have a linear combination as in the left hand side of $(1)$ you can find the $b_{jk}$ entries of the $k$th column of $B$ by comparing with the original columns of $A$, if you know the $A^i$ first, i.e., if you already know $A$. For this consider your example: $A^1b_{12} = \begin{bmatrix}4 \\ 10 \\ \end{bmatrix}$, so $A^1=\frac{1}{b_{12}}\begin{bmatrix}4 \\ 10 \\ \end{bmatrix}$. But you know $A^1 = \begin{bmatrix}2 \\ 5 \\ \end{bmatrix}=\frac{1}{2}\begin{bmatrix}4 \\ 10 \\ \end{bmatrix}$, so $b_{12}=2$ follows.
As to why $b_{ij}A^i=A^ib_{ij}$, $b_{ij}$ is a scalar here, and $A^i$ is a column vector so this is merely commutativity of multiplication of scalars thus: $$A^1b_{12} = \begin{bmatrix}2 \\ 5 \\ \end{bmatrix}\cdot2=\begin{bmatrix}2\cdot2 \\ 5\cdot2 \\ \end{bmatrix}=2\cdot \begin{bmatrix}2 \\ 5 \\ \end{bmatrix}=\begin{bmatrix}2\cdot2 \\ 2\cdot5 \\ \end{bmatrix}=b_{12}A^1$$