Let $N$,$M$ be integers where $M<N$; Operator $F$ defined on $N$ by $N$ matrices, extracts the upper left $M$ by $M$ submatrix. For example, if $N=3$ and $M=2$, we have:
$$F(\begin{bmatrix} x_{11} & x_{12} & x_{13}\\ x_{21} & x_{22} & x_{23}\\ x_{31} & x_{32} & x_{33} \end{bmatrix})=\begin{bmatrix} x_{11} & x_{12} \\ x_{21} & x_{22} \end{bmatrix}$$
We would like to find $N$ by $N$ matrices $X$ and $Y$ that satisfy the following properties:
$$F[X]=F[XX]\neq \mathbf{0}$$ $$F[Y]=F[YY]\neq \mathbf{0}$$ $$F[XY]=F[YX]=F[XYX]=F[YXX]=F[XXY]=F[YYX]=F[YXY]=F[XYY]=F[XXYY]=F[XYYX]=F[YYXX]=F[XYXY]=F[YXYX]\neq \mathbf{0}$$ and
$F[X]$, $F[Y]$, and $F[XY]$ have different values.
Question: What is the smallest $N$ for which such matrices $X$ and $Y$ exist? In particular does there exist any solution for $N=3$ and $M=1$ or $2$?
In the above equations, $XY$ denotes matrix multiplication (or any similar operation that is distributive and associative) and $\mathbf{0}$ is the $M$ by $M$ zero matrix.
Note this question has another variant posted here.
For any $M$ and $N$ we can take $X = Y = I_{N \times N}$ the identity matrix.
Slightly less trivially, we can let $X$ and $Y$ be block diagonal of the form
$$ \begin{pmatrix} I_{M \times M} & 0 \\ 0 & * \end{pmatrix}$$
So, for instance, when $N = 3$ and $M = 2$, $X = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & x \end{pmatrix}$ and $Y = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & y \end{pmatrix}$ will work for any $x$ or $y$.