There are famous Kronecker product trick $$ vec(AXB) = (B^T \otimes A)vec(X) $$ where $vec(X)$ - stacks columns of $X$. It's super useful when we need to take derivatives of expression with respect to vectorized version of $X$
I have a little bit different situation, i need to find such matrix $M$ for which $$ y = vec(X^TB) = Mvec(X) $$ I can use Kronecker product trick to obtain $$ y = vec(IX^TB) = (B^T\otimes I)vec(X^T) $$ But there are still $X^T$. Can anyone explain how i can get rid of transpose? I'm pretty sure it's possible to use row major $vec$ but problem is - my whole computations built around column major $vec$, so it's quite problematic.
Here's a potential solution. Let $E_{ij}$ be the matrix with a $1$ in the $(i,j)$ entry and $0$ elsewhere. Then define $$ U = \sum_{ij} E_{ij} \otimes E_{ji}. $$ Note that $U$ is a unitary matrix. In quantum information it is referred to as the swap operator because it has the follow property $$ U (e_i \otimes e_j) = (e_j \otimes e_i) $$ where $e_i$ are the standard basis vectors. As Ben Grossmann notes, it also goes under the name commutation operator.
Writing $X = \sum_{ij} x_{ij} E_{ij}$ we find that $$ \begin{aligned} \mathrm{vec}(X^T) &= \mathrm{vec}(\sum_{ij} x_{ij} E_{ji}) \\ &= \sum_{ij}x_{ij} e_{j} \otimes e_{i} \\ &= U \sum_{ij} x_{ij} e_{i} \otimes e_{j} \\ &= U \mathrm{vec}(\sum_{ij} x_{ij} E_{ij}) \\ &= U \mathrm{vec}(X) \end{aligned} $$
Thus you have $$ y = (B^T \otimes I) U \mathrm{vec}(X). $$