Let $A \in M_5(\mathbb{R})$. If $A = (a_{ij})$, let $A_{ij}$ denote the cofactor of entry $a_{ij}$, $1 \leq i, j \leq 5$. Let $\hat{A}$ denote the matrix whose $(ij)$th entry is $A_{ij}$, $1 \leq i, j \leq 5$.
- What is the rank of $\hat{A}$ when the rank of $A$ is 5?
- What is the rank of $\hat{A}$ when the rank of $A$ is 3?
It is a standard result that $A\hat{A}^T = (\det A) I$. This follows pretty much immediately from the expansion formula of determinants along rows.
If rank $A = 5$, then $\dim \text{null } A = 0$ by the rank-nullity theorem. So, $\det A \neq 0$ and $\hat{A}^T$ is invertible. As rank $B = \text{ rank } B^T$ for any matrix $B$, $\hat{A}$ is invertible. Thus, rank $\hat{A} = 5$.
As for the second question, I find myself stumped. In this case, as $\det A = 0$,
$$ A\hat{A}^T = 0$$
We can conclude that the range of $\hat{A}^T$ is entirely contained in the nullspace of $A$, which has dimension 2. So, rank $\hat{A}$ = rank $\hat{A}^T$ is at most $2$, but I can't figure out how to say definitely what the answer is.
The matrix of co-factors is identically zero when $A$ has rank $3$.
The thing is that if rank of $A$ is three, then all $5\times 5$ and all $4\times 4$ sub-matrices must have determinant zero. Otherwise the rank would be larger than $3$. In particular, all co-factors, determinants of $4\times4$ sub-matrices are zero.