My calulations show that for all $n\times m$ matrices $A, B$, it is $det(A^tB)=0$. Where is the mistake?

Question

Let $A, B \in \mathbb{M}^{n\times m}(\mathbb{R})$ be two $n\times m$ real matrices. Then the product $A^t\cdot B$ is a $m\times m$ real matrix, thus, it has a determinant $D=det(A^t B)$.

I calculated $D$ using the Leibniz Formula and my calulations result in $D=0$. I know this is wrong: for example if $A=B$ a square invertible matrix, then $D=det(A^tA)= det(A^t)det(A)=\big(det(A)\big)^2\neq 0$

So, unavoidably, there is a mistake in my calulations, but I cannot find it. I would apreciate any hints, corrections, or any insight. I would deeply apreciate if you could show me the correct calculations.

First, let $A=(a_{i,j})_{i,j}$, hence $A^t=(a_{j,i})_{i,j}$ and $B=(b_{i,j})_{i,j}$ for $i=1,...,n$ and $j=1,...,m$. We calculate the $(i,j)$-th term of the matrix $A^t B$ to be: $$(A^t B)_{ij} = \sum_{k=1}^n a_{k,i}b_{k,j}$$

Now we calculate the determinant of $A^t B$: $$\begin{array}{rlc} D &= \displaystyle \sum_{\sigma \in S(m)} \left(sgn(\sigma)\prod_{i=1}^m (A^t B)_{i, \sigma(i)}\right) & \text{(Leibniz formula for determinant)}\\ &= \displaystyle \sum_{\sigma \in S(m)} \left( sgn(\sigma)\prod_{i=1}^m \sum_{k=1}^n a_{k,i}b_{k,\sigma(i)} \right) & \text{(substitution)}\\ &= \displaystyle \sum_{\sigma \in S(m)} \left( sgn(\sigma)\sum_{k_1=1}^n\sum_{k_2=1}^n...\sum_{k_m=1}^n a_{k_1,1}a_{k_2,2}...a_{k_m,m}b_{k_1,\sigma(1)}b_{k_2,\sigma(2)}...b_{k_m,\sigma(m)} \right) & \text{(distributive property)}\\ & =\displaystyle \sum_{\sigma \in S(m)} \left( sgn(\sigma)\sum_{k_1=1}^n...\sum_{k_m=1}^n a_{k_1,1}...a_{k_m,m}b_{k_1,1}b_{k_2,2}...b_{k_m,m}\right) & (\sigma\text{ is a bijection)}\\ & = \displaystyle \left( \sum_{k_1=1}^n...\sum_{k_m=1}^n a_{k_1,1}...a_{k_m,m}b_{k_1,1}b_{k_2,2}...b_{k_m,m} \right) \left( \sum_{\sigma \in S(m)} sgn(\sigma) \right) & \text{(distributive property)}\\ & = 0 & \left( \sum_{\sigma \in S(m)} sgn(\sigma) = 0 \right) \end{array}$$

Answer 1

In the step where you write all $b_{k_i,\sigma(i)}$ to be $b_{k_i,i}$ in the product inside the summation stating the reason to be that $\sigma$ is a bijection, you're actually imposing $\sigma=\textrm{id}$.

Let us reduce ourselves to say $n=3$; then the body inside the summation is $a_{k_1,1}a_{k_2,2}a_{k_3,3}b_{k_1,\sigma(1)}b_{k_2,\sigma(2)}b_{k_3,\sigma(3)}$, your argument reduces this to $a_{k_1,1}a_{k_2,2}a_{k_3,3}b_{k_1,1}b_{k_2,2}b_{k_3,3}$ for all $\sigma\in S_3$ which disregards all the non-identity permutations.

For example, for say $\sigma=(1 2)\in S_3$, the corresponding term in the sum should be $a_{k_1,1}a_{k_2,2}a_{k_3,3}b_{k_1,2}b_{k_2,1}b_{k_3,3}$ while the term in your argument stays the same $a_{k_1,1}a_{k_2,2}a_{k_3,3}b_{k_1,1}b_{k_2,2}b_{k_3,3}$ which is not necessarily the same.

Answer 2

One good way to find such mistakes is to take some particular example and calculate numbers on every step.

Lets take $A = B = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$.

On $3$rd line you have (given that only terms with $k_1 = 1$, $k_2 = 2$ are non-zero) $\operatorname{sgn}((1)(2)) \cdot (a_{11} a_{22} b_{11}b_{22}) + \operatorname{sgn}((12))\cdot(a_{11}a_{22}b_{12}b_{21}) = 1$, as it should be.

However, on the 4th line you have $\operatorname{sgn((1)(2))}\cdot (a_{11}a_{22}b_{11}b_{22}) + \operatorname{sgn((12))}\cdot (a_{11}a_{22}b_{11}b_{22}) = 0$. So the mistake is in this transition.

The reason is that, while $\sigma$ is a bijection and you have the same terms if you look on $b$ only, different $a$-s are attached to different $b$-s depending on permutation.