I'm a little rusty on my linear algebra, but I would like to explore the solutions for the matrix $M$ that transforms $A$ into $B$.
I generated code to find some solutions for low dimensional spaces, but I didn't find a pattern to the solution set other than the transformation itself.
Is there any property for the possible $M$ that can be identified?
Note that you can look at each row of $M$ independently. For row $i$ you have one equation and $i$ unknowns, so you expect typically for there to be an $i-1$ dimensional space of solutions for that row.
More specifically, let $b_i$ be the $i$th entry of $b$, $[a]_i$ the vector consisting of the first $i$ entries of $a$, and $[m]_i$ the first $i$ entries (in other words, the nonzero part) of the $i$th row of $M$.
You need
$$b_i = [m]_i \cdot [a]_i$$ which has general solution $$[m]_i = b_i [a]_i / \| [a]_i\|^2 + c$$ for $c$ any vector in the orthogonal complement of $[a]_i$; you can compute an explicit basis for this complement if needed using Gram-Schmidt.
Notice there always exists infinitely many solutions ($i-1$ dimensions of them) unless $[a]_i$ is all zeros. In that case there is no solution, unless $b_i$ is also zero, in which case you can set the row $[m]_i$ to whatever you want.