Given two symmetrical square matrices $A,B \in \mathbb R^{n\times n}$ and a rectangular matrix $W\in \mathbb R^{n,k}$. I want to maximize
$$\max_{W} \frac{\det (W^TAW)}{\det(W^TBW)}.$$
I read that the $W$ the maximizes this consists of the generalized eigenvectors that correspond to the largest eigenvalue in $$Aw_i=\lambda_iBw_i.$$
I know that this is true if we want to maximize $\frac{w^TAw}{w^TBw}$, $w \in \mathbb R^n$. However, I am not sure how this corresponds to the first optimization problem where we have a rectangular matrix $W$ and the determinant involved. Is there a way I can see this?
According to matrix calculus the derivative of what you want to maximize with respect to $W$ is
$$ \frac{2 |W^TAW| |W^TBW| \left( AW (W^TAW)^{-1} - BW (W^TBW)^{-1} \right)}{|W^TBW|^2} = 0 $$ where $|\cdot|$ represents the determinant. So you need
$$ AW = BW (W^TBW)^{-1} (W^TAW) \tag{1} $$
It is easy to see that the solution satisfies $AW = BWX$ for some $X \in \mathbb{R}^{k \times k}$ and also
$$ \frac{|W^TAW|}{|W^TBW|} = |X| $$
Since the determinant is the product of eigenvalues you get your result.
However, one needs to take into the sign of the eigenvalues. For example, let the eigenvalues be $-1, -2, 1$. For $k = 1$, the maximizer is $w_3$, while for $k=2$, $W=[w_1 ~~ w_2]$ is the maximizer.
Edit: The specific equation in matrix calculus wiki is in the "Identities: scalar-by-matrix" table one previous from last row: "A is not a function of X, X is non-square, A is symmetric".
If $AW=BWX$ then from equation $(1)$
$$\begin{align} AW &= BW (W^TBW)^{-1} (W^TAW)\\ &= BW (W^TBW)^{-1} W^T(AW)\leftarrow\text{Plug in here}\\ &= BW (W^TBW)^{-1} W^T(BWX)\\ &= BW (W^TBW)^{-1} (W^TBW)X\\ &= BWX \end{align}$$
Let $X = T \tilde{X} T^{-1}$, then $$\begin{align} AW &= BW X\\ AW &= BW T \tilde{X} T^{-1}\\ AWT &= BW T \tilde{X}\\ A \tilde{W} &= B \tilde{W} \tilde{X} \end{align}$$
where $\tilde{W} := WT$. So, without losing generality, we can assume that $X$ is in Jordan normal form. In particular, if $X$ is diagonal, then its elements correspond to the generalized eigenvalues of $(A,B)$. So, maximization of determinant can be done by selecting the eigenvalues that has maximum multiplication value. (If $X$ is upper triangular, then its diagonal elements correspond to the eigenvalues.)
Edit 2. Let $X(W) := (W^TBW)^{-1} (W^TAW)$. We want to solve $$ \max_W |X(W)| $$ Therefore we need, $$ \frac{\partial |X(W)|}{\partial W} = 2 |X(W)| \left( AW (W^TAW)^{-1} - BW (W^TBW)^{-1} \right) = 0$$ and consequently $W$ and $X(W)$ must satisfy $$ AW = BW X(W) $$