Let $A$ be a $m\times n$ matrix and $B$ be a $n\times l$ matrix. For $1 \le i_1 < i_2 < \cdots < i_k \le m$ and $1 \le j_1 < j_2 < \cdots < j_k \le n$, is there a formula to compute the determinant of the $k\times k$ minor matrix $(AB)_{i_1, i_2, \ldots, i_k, j_1, j_2, \ldots, j_k}$ in the product $AB$?
I couldn't find such a formula in linear algebra textbooks and wonder if there is anything from the literature.
On
Yes , for this use the Binet-Cauchy formula. I use the notation $\binom{[i]}{j}$ to denote $j$-elements subsets of $\{1,\dots i\}$. And for $A$ a matrix and $I$ a subset, I denote $A_I$ the extracted matrix with the rows that are indexed by elements of $I$ and $A^I$ the matrix extracted by taking the columns indexed by $I$ (for suitable I).
The Binet Cauchy formula for $A$ of size $m \times n$ and $B$ of size $n \times m$ is $$ \det(AB) = \sum_{S \in \binom{[n]}{m}} \det(A^S) \det(B_S) $$
So by checking that $A^I B_J = (AB)^I_J$, you get $$\det (AB)^I_J = \sum_{S \subset \binom{[n]}{k}} \det (A^I_S) \det(B^S_J)$$
From wikipedia, by Cauchy–Binet formula,
$$\det(AB)_{I,J} = \sum_{K} \det({A})_{I,K} \det(B)_{K,J}$$