In these notes by N. Helwig http://users.stat.umn.edu/~helwig/notes/cancor-Notes.pdf covariance between two matrices is defined as follows:
$Cor(U,V) = \frac{Cov(U,V)}{\sqrt{Var(U)}\sqrt{Var(V)} }$
If U and V differ in their size and are, how can we calculate the matrix product in the denominator? How can we calculate the quotient?
It is true that $X$ and $Y$ have different sizes, but note that $U:=a'X$ and $V:=b'Y$ are scalars (i.e., $1\times 1$), so the formula is evaluating the correlation between two random variables; all the entities $\operatorname{Cov}(U,V)$, $\operatorname{Var}(U)$ and $\operatorname{Var}(V)$ are ordinary numbers.