According to this, vectorization of a 2D matrix M=$[\begin{array}{cc}a & b \\ c & d\\ \end{array}]$ is Vec(M) = $[\begin{array}{c} a \\ c \\ b \\ d \end{array}]$.
So, what will be the vectorization of a 3D matrix like K=[$[\begin{array}{cc}a & b \\ c & d\\ \end{array}]$, $[\begin{array}{cc}e & f \\ g & h\\ \end{array}]$, $[\begin{array}{cc}i & j \\ k & l\\ \end{array}]$]
Is it Vec(K) = $[\begin{array}{c} a \\ c \\ b \\ d \\e\\g\\f\\h\\i\\k\\j\\l \end{array}]$ or Vec(K) = $[\begin{array}{c} a \\ c \\ e \\ g \\i\\k\\b\\d\\f\\h\\j\\l \end{array}]$
PS: I actually need it for something CS related, but this is a math concept so I am asking it here.
$ \def\o{{\tt1}} \def\R#1{{\mathbb R}^{#1}} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\vecc#1{\op{vec}\LR{#1}} \def\modd#1{\op{mod}\LR{#1}} \def\divv#1{\op{div}\LR{#1}} \def\A{{\cal A}} \def\AA{{\large\mathbb A}} $For $\sf 3D$ and higher matrices, it helps to use ($\o$-based) index notation.
First, consider the $\sf 2D$ case: $\quad A\in\R{m\times n},\;a\in\R{mn}$ $$\eqalign{ A_{ij} &\implies a_q \\ q &\;\;=\;\; i + m\LR{j-\o} \qquad\qquad\qquad \\\\ }$$ For the $\sf 3D$ case: $\quad\A\in\R{m\times n\times p},\;a\in\R{mnp}$
First combine $(i,j)\to q\;$ using the mapping shown above $\;\A_{ijk} \implies A_{qk}$
then combine $(q,k)\to r\;$ with a similar index mapping $$\eqalign{ A_{qk} &\implies a_{r} \\ r &\;\;=\;\; q + mn\LR{k-\o} \qquad\qquad\quad \\ }$$ Or in a single step $$\eqalign{ \A_{ijk} &\implies a_{r} \\ r &\;\;=\;\; i + m\LR{j-\o} + mn\LR{k-\o} \\\\ }$$ Generalizing to higher dimensions: $\quad\AA\in\R{m\times n\times p\times\ldots}$ $$\eqalign{ \AA_{ijk\ell\ldots} &\implies a_{r} \\ r &\;\;=\;\; i + m\LR{j-\o} + mn\LR{k-\o} + mnp\LR{\ell-\o} + \ldots \\ }$$
This is the indexing scheme that numerically focused computer languages like $\sf Fortran{\rm\;and\;}Julia$ use behind the scenes for their multi-dimensional arrays. It is called column-major ordering.
Other (general purpose) languages like $\sf C/C^{++}$ don't have multi-dimensional arrays. They use arrays of arrays to fake it, and end up with row-major ordering.