When is $A^TBA$ invertible, where $B$ is an invertible, symmetric matrix?

Question

Let $B$ be an $n×n$ invertible matrix such that $B^T=B$, and let $A$ be a $n×2$ matrix with linearly independent columns. When is the product $A^TBA$ invertible?

Answer 1

We'll prove $A^TBA$ is invertible $\iff$ $B\left(\text{Col}(A)\right)\cap \text{Col}(A)^{\perp}=\{0\}$.

Here is the "only if" part. Assume $A^TBA$ is invertible but there is some non$-$zero $y$ in $B\left(\text{Col}(A)\right)\cap \text{Col}(A)^{\perp}$. Then $y=BAx$ for some $x\in \mathbb{R}^2$. Because $\text{Col}(A)^{\perp}=\text{Nul}(A^T)$ we also get $A^Ty=0$. Putting both together implies $A^TBAx=0$ i.e. $x\in \text{Nul}(A^TBA)$. Now $A^TBA$ is assumed to be invertible, so $x$ must be the zero vector, and so is $y$, a contradiction.

Here is the "if" part. Assume $B\left(\text{Col}(A)\right)\cap \text{Col}(A)^{\perp}$ has a trivial intersection and $A^TBAx=0$ for some $x\in \mathbb{R}^2$. Then we have $BAx\in \text{Nul}(A^T)=\text{Col}(A)^{\perp}$. But $BAx$ also belongs to $B\left(\text{Col}(A)\right)$ which means $BAx=0$ from our hypothesis. Hence $Ax=B^{-1}0=0$ and since $A$ has independent columns, $x=0$ and $A^TBA$ is invertible.

Answer 2

A slightly different way to look at this same idea: $A^TBA$ is invertible exactly if it has rank 2. Trivally we have that $BA$ has rank 2. Thus $BA$ can be understood as a map that maps the two canonical unit vectors to $Ba_1$ and $Ba_2$ where $a_i$ are the columns of $A$.

So $A^TBA$ is invertible exactly if $A^TBa_1$ and $A^TBa_2$ are independent. This means $(a_1Ba_1, a_2Ba_1)$, $(a_1Ba_2, a_2Ba_2)$ need to be independent. Note that as $B$ is symmetric this means $(a_1Ba_1, a_1Ba_2)$, $(a_1Ba_2, a_2Ba_2)$ need to be independent. This is equivalent by using the determinant to $(a_1Ba_1)(a_2Ba_2) - (a_1Ba_2)^2 \neq 0$, or equivalently to $$ (a_1Ba_1)(a_2Ba_2) \neq (a_1 B a_2)^2 $$

EDIT: Note that if $B$ is positive definite the above is related to Cauchy-Schwarz. In this case $xBy$ defines a scalar product, so $(a_1Ba_2)^2 \leq (a_1Ba_1)(a_2Ba_2)$ and equality only if $a_1,a_2$ are linearly dependent. So in this case $a_1,a_2$ independent already implies this property.