Given two matrices A, B, respectively, and an equation XA=B; how can we obtain X and discuss the system?
\begin{pmatrix} 1 & a\\ a & 1 \\ \end{pmatrix} \begin{pmatrix} a & 1 \\ a^2 & 0 \\ 1 & 1 \\ \end{pmatrix}
I am not familiar with this structure and I can't even picture what the dimensions of X are. Any general matrix I write down looks uncomplete.
$X$ must be a $3 \times 2$ matrix; we may write
$X = \begin{pmatrix} x_{11} & x_{12} \\ x_{21} & x_{22} \\ x_{31} & x_{32} \end{pmatrix}; \tag 1$
we may thus write out the system
$XA = B \tag 2$
as
$\begin{pmatrix} x_{11} & x_{12} \\ x_{21} & x_{22} \\ x_{31} & x_{32} \end{pmatrix} \begin{pmatrix} 1 & a \\ a & 1 \end{pmatrix} = \begin{pmatrix} a & 1 \\ a^2 & 0 \\ 1 & 1 \end{pmatrix}; \tag 3$
here
$A = \begin{pmatrix} 1 & a \\ a & 1 \end{pmatrix}, \tag 4$
and
$B = \begin{pmatrix} a & 1 \\ a^2 & 0 \\ 1 & 1 \end{pmatrix}; \tag 5$
we may write out the system (3) as a set of $6$ linear equations in two variables each:
$x_{11} + ax_{12} = a, \tag 6$
$ax_{11} + x_{12} = 1; \tag 7$
$x_{21} + ax_{22} = a^2, \tag 8$
$ax_{21} + x_{22} = 0, \tag 9$
$x_{31} + ax_{32} = 1, \tag{10}$
$ax_{31} + x_{32} = 1; \tag{11}$
the system of equations (6)-(11) is in fact easy to solve, one pair $(x_{i1}, x_{i2})$ at a time; for example, we multiply (6) by $a$:
$ax_{11} + a^2x_{12} = a^2, \tag{12}$
and subtract (7) from (12):
$(a^2 - 1) x_{12} = a^2 - 1, \tag{13}$
whence
$x_{12} = 1, \tag{14}$
provided of course that
$a \ne \pm 1; \tag{15}$
having $x_{12}$ we find via (6) that
$x_{11} = a(1 - x_{12}) = 0; \tag{16}$
the remaining equations (8)-(11) may be similarly solved.
The system (2), (3) may also be addressed in a somewhat more matrix-oriented manner by setting
$Y = \begin{pmatrix} x_{11} & x_{12} \\ x_{21} & x_{22} \end{pmatrix}, \; \mathbf y = (x_{31}, x_{32}), \; C = \begin{pmatrix} a & 1 \\ a^2 & 0 \end{pmatrix}; \mathbf d = (1, 1); \tag {17}$
then (3) may be written in the from
$\begin{pmatrix} Y \\ \mathbf y \end{pmatrix} A = \begin{pmatrix} C \\ \mathbf d \end{pmatrix}, \tag{18}$
leading to
$YA = C, \; \mathbf y A = \mathbf d, \tag{19}$
whence
$Y = CA^{-1}, \; \mathbf y = \mathbf d A^{-1}; \tag{20}$
where it is easily seen that
$A^{-1} = \dfrac{1}{1 - a^2} \begin{pmatrix} 1 & -a \\ -a & 1 \end{pmatrix}. \tag{21}$
We assume $a = \pm 1$ in deriving (17)-(21); in the event that $a = \pm 1$ we see that (8)-(9) become
$x_{21} \pm x_{22} = 1, \; \pm x_{21} + x_{22} = 0, \tag{22}$
which lead to the contradicion
$\pm 1 = 0; \tag{23}$
thus we see there is no solution to (2) when $a = \pm 1$.