In reading a paper I came across this expression which I don't quite understand: $$ \frac{\lambda_1}{2N}\operatorname{tr}\left((\mathbf{H}^M\mathbf{H}^M)^T\right) $$ For context, $\lambda_1$ and $N$ are scalars and $\mathbf{H}^M$ ($H$ henceforth) is a $MxN$ matrix.
The paper claims that this value is related to the variance of the column vectors which make up H, but either there is a typo or (quite likely) a linear algebra concept I don't know. The reason I am confused is that trace requires square input, and if H were to be made square then the transpose would have to apply to one of the two $H$s, not both after multiplying them (which itself doesn't make sense to me because only a square matrix can be multiplied by itself in the first place).
The paper is here, the mentioned expression is at the bottom of the third page in equation (5).
It looks like a typo. In the paper that you have linked, the equation(12) on page(4) rewrites
$$ arg.min .J = ... -\frac{\lambda_1}{2}\Bigl(tr\bigl(\frac{1}{N}H^M(H^M)^T\bigr)+\alpha tr(\Sigma_B-\Sigma_W)\Bigr) + ... $$
with the transpose correctly placed.