Assuming that matrix $A$ & $B$ are both square and invertible, is there a reliable way to separate out matrix $A$ and $B$ from $AB$ and $BA$? For example, if I want to find matrix $A$ and matrix $B$: $$ AB = \begin{pmatrix} 4 & -4 & -1 \\ 5 & 1 & 1 \\ -5 & -1 & -1 \\ \end{pmatrix} $$ and $$ BA = \begin{pmatrix} 5 & 4 & 1 \\ -4 & 1 & -5 \\ -1 & 1 & -2 \\ \end{pmatrix} $$ How would would I find $A$ and $B$?
(for the record, these specific matrices were arbitrarily made with a calculator and are only here for the sake of example, so:) $$ A = \begin{pmatrix} 1 & 2 & -1 \\ 2 & 1 & 1 \\ -2 & -1 & -1 \\ \end{pmatrix} $$ and $$ B = \begin{pmatrix} 1 & 1 & -1 \\ 2 & -2 & 1 \\ 1 & 1 & 2 \\ \end{pmatrix} $$
$\def\ed{\stackrel{\text{def}}{=}}$ If $\ K_1\ $ and $\ K_2\ $ are two square matrices, then there exist invertible square matrices $\ A\ $ and $\ B\ $ satisfying the equations \begin{align} AB&=K_1\label{e1}\tag{1}\\ BA&=K_2\label{e2}\tag{2} \end{align} if and only if $\ K_1\ $ and $\ K_2\ $ are similar, and both invertible.
If $\ A\ $ and $\ B\ $ are invertible matrices satisfying equations (\ref{e1}) and (\ref{e2}), then $$ A=K_1B^{-1}=B^{-1}K_2\ , $$ from which it follows that $\ K_1=B^{-1}K_2B\ $—that is, $\ K_1\ $ and $\ K_2\ $ are similar—and since $\ B^{-1}A^{-1}K_1=I=A^{-1}B^{-1}K_2\ ,$ then $\ K_1\ $ and $\ K_2\ $ are invertible.
Conversely, if $\ K_1\ $ and $\ K_2\ $ are invertible and similar, then there exists an invertible square matrix $\ Q\ $ such that $$ K_2=QK_1Q^{-1}\ ,\label{e3}\tag{3} $$ and if we put $\ A\ed K_1Q^{-1}\ ,$$\ B\ed Q\ ,$ then $\ A\ $ and $\ B\ $ satisfy equations (\ref{e1}) and (\ref{e2}) and are invertible.
You can determine whether $\ K_1\ $ and $\ K_2\ $ are similar by reducing them both to Jordan normal form . While a matrix doesn't usually have a unique Jordan normal form, all of them must contain the same set of Jordan blocks. If $\ J_1\ $ and $\ J_2\ $ are Jordan normal forms for $\ K_1\ $ and $\ K_2\ ,$ respectively, then $\ K_1\ $ and $\ K_2\ $ are similar if and only if there's a permutation matrix $\ P\ $ such that $$ J_2=PJ_1P^{-1}\label{e4}\tag{4}\ . $$ Whether or not this is the case can always be seen easily by inspection (provided $\ J_1\ $ and $\ J_2\ $ are not too large). The matrices $\ K_1\ $ and $\ K_2\ $ are related to their Jordan forms by equations
\begin{align} J_1&=Q_1^{-1}K_1Q_1\ \ \text{and}\\ J_2&=Q_2^{-1}K_2Q_2\ , \end{align} where $\ Q_1\ $ and $\ Q_2\ $ are matrices whose columns form a complete set of generalised eigenvectors of $\ K_1\ $ and $\ K_2\ $ respectively. If equation (\ref{e4}) holds, then \begin{align} K_2&=Q_2J_2Q_2^{-1}\\ &=Q_2PJ_1P^{-1}Q_2^{-1}\\ &=Q_2PQ_1^{-1}K_1Q_1P^{-1}Q_2^{-1}\\ &=Q_2PQ_1^{-1}K_1\big(Q_2PQ_1^{-1}\big)^{-1}\ , \end{align} so equation (\ref{e3}) holds with $\ Q=Q_2PQ_1^{-1}\ .$ Therefore, a solution of equations (\ref{e1}) and (\ref{e2}) is given by $\ A\ed K_1(Q_2PQ_1^{-1}\big)^{-1}\ ,$$\ B\ed Q_2PQ_1^{-1}\ .$
Example
Since your matrices $\ AB\ $ and $\ BA\ $ aren't invertible it's impossible for $\ A\ $ and $\ B\ $ to be both invertible. Nevertheless, it's still possible to perform the above procedure to find two matrices $\ A_0\ $ and $\ B_0\ $ such that \begin{align} A_0B_0&=AB=\pmatrix{4 & -4 & -1 \\ 5 & 1 & 1 \\ -5 & -1 & -1 }\ \ \text{, and}\label{e5}\tag{5}\\ B_0A_0&=BA=\pmatrix{5 & 4 & 1 \\ -4 & 1 & -5 \\ -1 & 1 & -2}\ .\label{e6}\tag{6} \end{align} As several commenters on your question have pointed out, you can't expect the solution of the above equations to be unique, so if you hadn't told us the values of the $\ A\ $ and $\ B\ $ you used to generate the matrices $\ AB\ $ and $\ BA\ $ there would be no way of telling which of the many solutions $\ A_0,B_0\ $ of equations (\ref{e5}) and (\ref{e6}) would correspond to them. As it happens, both matrices $\ AB\ $ and $\ BA\ $ have the same $3$ distinct eigenvalues $\ 2+i\sqrt{11}, 2-i\sqrt{11}, $ and $\ 0\ ,$ and are therefore diagonalisable with the same Jordan normal form $$ J=\pmatrix{2+i\sqrt{11}&0&0\\0&2-i\sqrt{11}&0\\0&0&0}\ . $$ We can therefore (and, in fact, must) take $\ P\ $ to be the identity in equation (\ref{e4}) above. The eigenvectors of the matrices $\ AB\ $ and $\ BA\ $ give us the matrices $$ Q_1=\pmatrix{1&1&1\\\frac{2}{3}-\frac{i\sqrt{11}}{3}&\frac{2}{3}+\frac{i\sqrt{11}}{3}&3\\ -\frac{2}{3}+\frac{i\sqrt{11}}{3}&-\frac{2}{3}-\frac{i\sqrt{11}}{3}&-8} $$ and $$ Q_2=\pmatrix{1&1&1\\-\frac{22}{31}+\frac{7i\sqrt{11}}{31}&-\frac{22}{31}-\frac{7i\sqrt{11}}{31}&-1\\ -\frac{5}{31}+\frac{3i\sqrt{11}}{31}&-\frac{5}{31}-\frac{3i\sqrt{11}}{31}&-1} $$ for which \begin{align} J&=Q_1^{-1}ABQ_1\\ &=Q_2^{-1}BAQ_2\ , \end{align} and the procedure described above then gives us \begin{align} A_0&=ABQ_1Q_2^{-1}=\pmatrix{5&4&1\\4&-3&7\\-4&3&-7}\\ B_0&=Q_2Q_1^{-1}=\pmatrix{1&0&0\\-\frac{8}{31}&-\frac{29}{31}&-\frac{8}{31}\\ \frac{1}{31}&-\frac{8}{31}&\frac{1}{31}} \end{align} as a solution to equations (\ref{e5}) and (\ref{e6}), which can be verified by direct multiplication: \begin{align} \pmatrix{5&4&1\\4&-3&7\\-4&3&-7}\pmatrix{1&0&0\\-\frac{8}{31}&-\frac{29}{31}&-\frac{8}{31}\\ \frac{1}{31}&-\frac{8}{31}&\frac{1}{31}}&=\pmatrix{4 & -4 & -1 \\ 5 & 1 & 1 \\ -5 & -1 & -1 }\\ \pmatrix{1&0&0\\-\frac{8}{31}&-\frac{29}{31}&-\frac{8}{31}\\ \frac{1}{31}&-\frac{8}{31}&\frac{1}{31}}\pmatrix{5&4&1\\4&-3&7\\-4&3&-7}&=\pmatrix{5 & 4 & 1 \\ -4 & 1 & -5 \\ -1 & 1 & -2}\ . \end{align} Note that while $\ B_0\ $ is invertible, $\ A_0\ $ isn't.